Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion pkg/ebpf/gadgets/http/capture/capture_test.go
Original file line number Diff line number Diff line change
Expand Up @@ -44,7 +44,7 @@ func harness(t *testing.T) string {
func TestSyscallPayloadChunks(t *testing.T) {
bin := harness(t)
for _, mode := range []string{"w", "r", "wv", "rv"} {
for _, size := range []int{10240, 16384, 16385, 20992, 40960} {
for _, size := range []int{10240, 16383, 16384, 16385, 20992, 40960} {
for _, split := range []int{0, 8192} {
t.Run(fmt.Sprintf("%s/%d/split%d", mode, size, split), func(t *testing.T) {
payload := []byte("POST / HTTP/1.1\r\nContent-Length: 500000\r\n\r\n" + strings.Repeat("b", size))[:size]
Expand Down
5 changes: 5 additions & 0 deletions pkg/ebpf/gadgets/http/capture/testdata/harness.h
Original file line number Diff line number Diff line change
Expand Up @@ -55,6 +55,11 @@ static int bpf_probe_read(void *dst, size_t n, const void *src) {
memcpy(dst, src, n); return 0;
}
static int bpf_probe_read_user(void *dst, size_t n, const void *src) {
if (aggregate_buffer && (uintptr_t)dst >= (uintptr_t)aggregate_buffer &&
(uintptr_t)dst < (uintptr_t)aggregate_buffer + 32768) {
size_t offset = (uintptr_t)dst - (uintptr_t)aggregate_buffer;
assert(n <= 16384 && offset < 16384 && n <= 16384 - offset);
}
if (((reserved && (uintptr_t)dst >= (uintptr_t)reserved && (uintptr_t)dst < (uintptr_t)reserved+20000) || (aggregate_buffer && (uintptr_t)dst >= (uintptr_t)aggregate_buffer && (uintptr_t)dst < (uintptr_t)aggregate_buffer+16384)) && ++copy_calls == fail_copy) return -1;
memcpy(dst, src, n); return 0;
}
Expand Down
13 changes: 10 additions & 3 deletions pkg/ebpf/gadgets/http/program.bpf.c
Original file line number Diff line number Diff line change
Expand Up @@ -608,12 +608,19 @@ static __noinline int emit_iov_step(struct syscall_trace_exit *ctx, struct paylo
capture_failed(msg->fd, args->is_rx, HTTP_LOSS_LIMIT);
return -1;
}
__u32 buffered = args->buffered;
__u64 buffered = args->buffered;
if (buffered >= MAX_DATAEVENT_BUFFER)
return -1;
__u32 size = min_size(args->actual_len - args->offset, MAX_DATAEVENT_BUFFER - buffered);
__u64 capacity = MAX_DATAEVENT_BUFFER - buffered;
__u64 size = min_size(args->actual_len - args->offset, capacity);
size = min_size(size, HTTP_MAX_CHUNKS * MAX_DATAEVENT_BUFFER - captured);
if (size > MAX_DATAEVENT_BUFFER || buffered + size > MAX_DATAEVENT_BUFFER)
// LLVM can otherwise fold away the explicit bounds below because min_size
// already implies them in C. The verifier needs those checks after both
// minima; this register barrier emits no instructions.
asm volatile("" : "+r"(size), "+r"(capacity));
// Check the helper's size itself in 64 bits. Checking a narrowed sum does
// not bound this scalar on older verifiers (notably COS 121 / Linux 6.6).
if (size > MAX_DATAEVENT_BUFFER || size > capacity)
return -1;
__u32 zero = 0;
struct http_aggregate *aggregate = bpf_map_lookup_elem(&aggregate_scratch, &zero);
Expand Down
50 changes: 50 additions & 0 deletions pkg/ebpf/gadgets/http/tests/README.md
Original file line number Diff line number Diff line change
@@ -1,5 +1,34 @@
# Plain HTTP response continuation regression tests

## Kernel verifier and live iovec capture

On a disposable Linux host with root access, Docker, Python 3 and IG `v0.48.1`:

```sh
sudo python3 pkg/ebpf/gadgets/http/tests/verifier_capture.py \
--output-dir /var/tmp/http-verifier-regression
```

The runner builds current HTTP source under a unique local tag, loads and
attaches it with registry pulling disabled, and requires an actual HTTP event
before checking capture. It verifies byte-for-byte payloads and the 16 KiB
event / 256 KiB syscall limits through real TCP `sendmsg`, `recvmsg`, `writev`
and `readv` calls, including empty/split vectors and short receive buffers.
It retains environment information, JSON events, generated ELF files and
verbose load logs. Any load, capture or fixture failure exits nonzero; missing
privileges are not treated as a pass. It traces only the runner's PID.

To compare an already imported release artifact, add `--image IMAGE:TAG` and
use a separate output directory. Run both versions on COS 121 build
`18867.624.2`, kernel `6.6.157+`, and confirm the baseline ELF digest is
`d278a0cf36ffcd8e95dad1b97da80422ba2e9c7d92342da4b9ff1857baa1151d`.
The baseline must reject loading and the patched image must pass. A pass on
another kernel does not establish COS compatibility. Repeat the patched run
to check detach and reattachment. The runner does not deploy node-agent or
establish backend/profile integration.

## Native continuation tests

Run `go test ./pkg/ebpf/gadgets/http/tests` from the repository root. A native C
compiler (`cc`) is required; the test explicitly skips when it is unavailable.
The harness compiles the production HTTP classifier and syscall entry/exit
Expand Down Expand Up @@ -42,3 +71,24 @@ The same per-CPU layout applies to `capture_loss`; its three reason keys are
unchanged. Neither map changes the payload/event ABI, but readers of the old
shared ARRAY layout must adapt to per-CPU values. The native capture harness
checks the production map types and isolates increments on two simulated CPUs.

## Local COS 121 VM evidence

`local_cos121.py` extracts and verifies Google's published `18867.624.2`
full update payload and boots its original kernel and root filesystem under
KVM/QEMU. It requires Linux x86_64, KVM access, QEMU, `mkfs.ext4`, and the
validation fixture disk/initramfs supplied with the evidence bundle. Run
`python3 pkg/ebpf/gadgets/http/tests/local_cos121.py --help` for arguments.

The guest uses a test initramfs and a writable overlay over the verified vendor
root partition. It starts Docker manually with the vfs storage driver and runs
Python/IG in a privileged container sharing the guest PID and network namespaces.
This isolates the affected kernel from the workstation kernel. It does not
reproduce GCE firmware, normal COS systemd startup, or a deployed host-agent.
Keep the payload/partition hashes, guest kernel configuration and command line,
boot log, OCI identities, baseline verifier rejection, both patched capture runs,
and exit statuses together when reporting results.

See [the COS 121 validation report](validation-cos121.md) for the measured
baseline rejection, patched capture/reattachment results, artifact identities,
boot differences, and replay command.
145 changes: 145 additions & 0 deletions pkg/ebpf/gadgets/http/tests/local_cos121.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,145 @@
#!/usr/bin/env python3
"""Prepare and boot the exact COS regression kernel with supplied test fixtures.

Requires Python 3, qemu-system-x86_64, KVM access, and mkfs.ext4. The initrd and
fixture disk come from the accompanying evidence archive; inspect guest.sh there
before running. Guest output is evidence to review, not an automatic test pass.
"""
import argparse
import base64
import bz2
import hashlib
import json
from pathlib import Path
import shutil
import struct
import subprocess
import urllib.request
import zlib
import lzma

URL = 'https://storage.googleapis.com/cos-tools/18867.624.2/cos_full_payload_stable.bin'
EXPECTED = {
'kernel': 'e821cec8c0beb62b2b4ea61bc63afbdce7449b9634e0bd791bd4b28edda4bc14',
'root': '77fb2cb94838dbb12c5585ff2d6e77474541804e772ff69a6456ca193d648e26',
}


def varint(data, offset):
value = shift = 0
while True:
byte = data[offset]
offset += 1
value |= (byte & 127) << shift
shift += 7
if byte < 128:
return value, offset
if shift > 63:
raise ValueError('invalid protobuf varint')


def fields(data):
offset = 0
while offset < len(data):
key, offset = varint(data, offset)
number, wire = key >> 3, key & 7
if wire == 0:
value, offset = varint(data, offset)
elif wire == 2:
length, offset = varint(data, offset)
value = data[offset:offset + length]
offset += length
else:
raise ValueError(f'unsupported protobuf wire type {wire}')
yield number, value


def require(condition, message):
if not condition:
raise ValueError(message)


def prepare(payload, work):
data = payload.read_bytes()
digest = base64.b64encode(hashlib.md5(data).digest()).decode()
require(digest == 'hl91B9GxQQ/T4mZr/MJNGg==', 'unexpected COS payload MD5')
magic, version, length, signature = struct.unpack('>4sQQI', data[:24])
require(magic == b'CrAU' and version == 2, 'unexpected update payload header')
start = 24 + length + signature
manifest = list(fields(data[24:24 + length]))
block = dict(manifest)[3]
for tag, partition in manifest:
if tag != 13:
continue
entries = list(fields(partition))
values = dict(entries)
name = values[1].decode()
require(name in EXPECTED, f'unexpected partition {name}')
info = dict(fields(values[7]))
target = work / (name + '.img')
with target.open('xb') as output:
output.truncate(info[1])
for number, raw in entries:
if number != 8:
continue
operation = list(fields(raw))
op = dict(operation)
content = data[start + op[2]:start + op[2] + op[3]]
if 8 in op:
require(hashlib.sha256(content).digest() == op[8], 'operation checksum mismatch')
content = {0: lambda x: x, 1: bz2.decompress, 8: lzma.decompress}[op[1]](content)
offset = 0
for field, extent in operation:
if field != 6:
continue
extent = dict(fields(extent))
size = extent[2] * block
output.seek(extent[1] * block)
output.write(content[offset:offset + size])
offset += size
require(offset == len(content), 'unexpected operation length')
digest = hashlib.sha256(target.read_bytes()).digest()
require(digest == info[2] and digest.hex() == EXPECTED[name], 'partition checksum mismatch')
kernel = (work / 'kernel.img').read_bytes()
image = zlib.decompress(kernel[66252:], 31)
require(image.startswith(b'\x7fELF'), 'vendor kernel is not ELF')
require(hashlib.sha256(image).hexdigest() == 'b12ae2a84f196c94eda35658a1855813501fbe1b5d220408840bb7ff15395bff', 'kernel ELF checksum mismatch')
(work / 'vmlinux').write_bytes(image)


def main():
parser = argparse.ArgumentParser(description=__doc__)
parser.add_argument('--work-dir', required=True, type=Path, help='new directory; never reuse active VM disks')
parser.add_argument('--payload', type=Path, help='existing payload; otherwise download exact public build')
parser.add_argument('--initrd', required=True, type=Path)
parser.add_argument('--fixtures', required=True, type=Path, help='prepared ext4 fixture image from evidence archive')
parser.add_argument('--prepare-only', action='store_true')
args = parser.parse_args()
work = args.work_dir.resolve()
work.mkdir(parents=True, exist_ok=False)
payload = args.payload
if payload is None:
payload = work / 'payload.bin'
urllib.request.urlretrieve(URL, payload)
prepare(payload, work)
shutil.copyfile(args.initrd, work / 'initrd.cpio')
shutil.copyfile(args.fixtures, work / 'fixtures.img')
with (work / 'state.img').open('xb') as disk:
disk.truncate(24 * 1024**3)
subprocess.run(['mkfs.ext4', '-F', str(work / 'state.img')], check=True)
command = ['qemu-system-x86_64', '-enable-kvm', '-cpu', 'host', '-smp', '4', '-m', '4096',
'-kernel', str(work / 'vmlinux'), '-initrd', str(work / 'initrd.cpio'),
'-append', 'console=ttyS0 rdinit=/init panic=-1',
'-drive', f'file={work}/root.img,format=raw,if=virtio,readonly=on',
'-drive', f'file={work}/state.img,format=raw,if=virtio',
'-drive', f'file={work}/fixtures.img,format=raw,if=virtio',
'-netdev', 'user,id=n0', '-device', 'virtio-net-pci,netdev=n0',
'-display', 'none', '-serial', f'file:{work}/serial.log', '-monitor', 'none', '-no-reboot']
(work / 'qemu-command.json').write_text(json.dumps(command, indent=2) + '\n')
print(f'Prepared {work}. Guest results remain in fixtures.img; inspect serial.log and mount read-only after shutdown.')
if not args.prepare_only:
subprocess.run(command, check=True)


if __name__ == '__main__':
main()
123 changes: 123 additions & 0 deletions pkg/ebpf/gadgets/http/tests/validation-cos121.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,123 @@
# COS 121 verifier validation — 2026-10-16

Validated PR #1012 production source at `d448cce61d763ed4bccd537d15fb8c1248bed166`
against its base `67e455e040520a3f1f5a4ac20fbf5fe25f676d43`.

## Environment and identity

All execution was local on an x86_64 workstation using KVM/QEMU, four vCPUs
and 4 GiB RAM. The guest reported:

- COS `VERSION_ID=121`, `BUILD_ID=18867.624.2`.
- `6.6.157+ #1 SMP Sat Sep 26 10:00:07 UTC 2026 x86_64`.
- Vendor Docker `27.5.1`; IG `v0.48.1`.

Source payload: `https://storage.googleapis.com/cos-tools/18867.624.2/cos_full_payload_stable.bin`.
Payload MD5 (base64): `hl91B9GxQQ/T4mZr/MJNGg==`. Extraction checked individual
operation hashes and final partition SHA-256 values:

| Partition | SHA-256 |
| --- | --- |
| root | `77fb2cb94838dbb12c5585ff2d6e77474541804e772ff69a6456ca193d648e26` |
| kernel | `e821cec8c0beb62b2b4ea61bc63afbdce7449b9634e0bd791bd4b28edda4bc14` |

The original vendor kernel was unpacked without recompilation or patching.
The original root partition was mounted read-only beneath a writable overlay.
A BusyBox initramfs loaded the vendor virtio block module and launched a test
shell instead of COS systemd. The guest mounted proc/sys/dev, bpffs, debugfs,
and cgroup2; Docker used vfs storage with bridge/iptables management disabled.
Python and IG ran in a privileged container with guest host PID/network access.
The kernel command line was `console=ttyS0 rdinit=/init panic=-1`.
These boot/runtime differences are explicit: this is exact vendor kernel/rootfs
validation, not a normal GCE boot or a full host-agent deployment.

## Requested evidence

| Run | Result |
| --- | --- |
| Baseline ELF load | Rejected in `syscall__probe_ret_sendmsg`, with `invalid access to map value, value_size=32768 off=0 size=262144`; exit 1 |
| Patched load and capture | All 42 live TCP scenarios passed; exit 0 |
| Patched detach/reattachment | Separate tracer process using identical artifact; all 42 scenarios passed; exit 0 |

The baseline amd64 ELF was exactly 326,256 bytes with SHA-256
`d278a0cf36ffcd8e95dad1b97da80422ba2e9c7d92342da4b9ff1857baa1151d`,
matching issue #1009. The baseline OCI index was a local rebuild, **not** the
released OCI index; identity equivalence is established for the ELF bytes.
Patched amd64 SHA-256:
`ed3cc0343a3784f60be408be1f6f923386b41f1f432a3c285c89d1e28007eff3`.
Patched arm64 SHA-256:
`3f5b58c7263385ee64f57b2a63d95045cc62788c22216c5c8e91aecc1cd84192`.

Capture tested sendmsg, recvmsg, writev and readv; sizes immediately below,
at and above 16 KiB and 256 KiB; split/empty vectors and partial receives.
The existing runner checked payload bytes and chunk/syscall limits. Both
architectures compiled and retained direct size/capacity checks before helper
112. arm64 was not loaded on hardware.

## Repository checks

Before the upstream merge, after rebuilding `tracers.tar`, the full default Go suite,
HTTP race tests, vet for HTTP/containerwatcher, and Linux amd64 agent build
passed with Go 1.27.0. The pre-existing archive had an incompatible trace_open
schema; rebuilding resolved `TestOpenFields` without changing its expectations.
The unit command was:

```sh
env TMPDIR=/home/linux/.cache/http1009/tmp \
GOTOOLCHAIN=go1.27.0 GOCACHE=/tmp/node-agent-go-build \
go test -mod=readonly -p 2 \
-exec 'sudo env GOTOOLCHAIN=go1.27.0 GOPATH=/home/linux/go GOCACHE=/tmp/node-agent-go-build' \
-count=1 ./...
```

No Kubernetes component deployment, backend/profile integration, production
rollout, or arm64 kernel load is claimed.

## Evidence and replay

The local evidence bundle `http1009-cos121-evidence.tar.gz` includes full
verifier logs (including the decoded 5,413-line baseline trace), raw capture
events, exit statuses, kernel configuration, serial boot log, ELF files,
disassembly, OCI identities, test/build logs, initramfs and fixture disk.
The fixture disk contains the original runner, IG executable, imported gadget
archive and Python image. Inspect `fixtures/guest.sh` before replaying.

Extract the bundle, then run from the repository root with a **new** work directory:

```sh
python3 pkg/ebpf/gadgets/http/tests/local_cos121.py \
--work-dir /var/tmp/http1009-cos121-replay \
--initrd /path/to/extracted/cos121/initrd.cpio \
--fixtures /path/to/extracted/cos121/fixtures.img
```

The helper downloads the pinned public payload unless `--payload` supplies a
local copy. After guest shutdown, inspect `serial.log`; expected markers are
`RESULT baseline 1`, `RESULT patched 0`, `RESULT reattach 0`, and
`VALIDATION_FINISHED`. Mount the stopped fixture disk with `loop,ro,noload` to
inspect `results/`; require the exact baseline bounds error and 42 PASS lines
in each patched stdout log. QEMU exiting zero alone does not assert success.

## Validation after updating from main

On 2026-10-16, branch `fix/1009-http-bpf-verifier-bounds` was checked against
freshly fetched upstream `main` at
`493f1011f11be74d3b46477516eaaceb02de8871`. Merge commit
`76630452` already includes that revision without unresolved conflicts.
The verifier bounds fix remains unchanged. The production HTTP source differs
from the COS-tested revision only in upstream's local socket port byte-order
correction; therefore the recorded ELF hashes identify the reviewed revision,
not a rebuild of the merged source.

The merged checkout passed the Go 1.27 agent build, HTTP race tests and HTTP vet,
Python syntax checks for both validation helpers, the COS helper's command-line smoke check,
and `git diff --check`. The full repository suite and target-kernel capture
results above belong to the earlier validation; they were not rerun after the
merge.

The local evidence archive is
`/home/linux/.cache/http1009/http1009-cos121-evidence.tar.gz`, with SHA-256
`12b526deaea2ab6197184b810c8798fb0b7cd62741fcde3e0e4ac025b3e3bdc3`.
Publish the archive and checksum alongside the maintainer reply so reviewers
can inspect the logs and replay the validation. A local archive path is not
accessible to GitHub reviewers.
Loading
Loading