TCP Module #663

The generic problem. TCP is not a message transport: it offers a single ordered, reliable stream of bytes and nothing more. In particular it does not preserve the boundaries of the sender's writes — those boundaries are never transmitted, because to TCP they do not exist. The kernel is free to merge several writes into one segment, and actively tries to (that is what Nagle's algorithm does), and equally free to split one write across several. On top of that, a read() returns whatever bytes happen to be in the receive buffer at the instant it is called, which is neither one segment nor one write. So a receiver of application-level messages must treat two situations as equally normal: several messages arriving glued together in a single read, with no separator, and one message arriving split across two or more reads. Neither is an error, neither is packet loss, and neither has anything to do with IP fragmentation. Recovering message boundaries out of the stream is called framing, and it is the application's job: every TCP protocol solves it with a length fixed in advance, a length prefix, or a delimiter that cannot appear in the payload. UDP needs none of this, since the kernel hands over a whole datagram or nothing.

The consequence is always the same: you cannot ask the socket for one message. You accumulate bytes in a buffer and, after each read, ask whether the start of that buffer holds a complete message. Either it does — take it, remove exactly the bytes it occupied, and ask again, because another one may be glued right behind it — or it does not yet, and you go back to the socket. One rule carries the whole scheme: nothing may be removed from the buffer until a message has parsed in full, or the bytes already consumed are lost and every later read is misaligned.

Our case. A board's TCP connection carries protections and messages inbound, orders outbound, and the framing rule is chosen by the frame's leading 16-bit id, resolved against the ADJ. A protection has a fixed length that only the ADJ can give you: 2 bytes of id, 4 of timestamp, 2 reserved, then the triggering value, as wide as the watched measurement's wire type — 9 bytes for a boolean, 16 for an f64. Same kind of frame, different lengths, so no constant in the code can describe it. A message's length cannot be known in advance at all: origin and message are NUL-terminated strings of variable length (capped at 48 and 320 bytes), so the frame ends at the second NUL. There is no length prefix and no frame terminator anywhere in the format. The end of a frame is therefore never detected, it is deduced: read the id, look its shape up in the ADJ, read the fields that shape prescribes, and when the last one is in, the frame is over and its length is however many bytes that took. "Has it ended?" and "how long is it?" are the same question, and parsing answers it.

Both cases are real here. There is no IP fragmentation on the pod's network, but that is not the mechanism that matters: the glued case is the frequent one — a protection and a message in the same instant, a burst of messages at boot, a fault storm — and the split case is the less frequent but ordinary one, firmware writing header and payload in two write() calls. Note which one punishes the naive reader: reading into a buffer, parsing once and discarding the remainder does not break on the rare split, it silently drops frames on the frequent glued case. Worth adding that the Go backend never faced any of this, because it read straight off the connection and a blocking read waits mid-frame.

Why protocol stays as it is. Framing is per-connection state — a buffer, how much of it was consumed, when to give up — and protocol is deliberately stateless and socket-free so a sniffer or a simulator can reuse it over a pcap without dragging in tokio. That state belongs to whoever owns the socket, net::tcp. And protocol already provides exactly what the loop needs, unchanged: decode_next takes any impl Read, and &mut Cursor<&[u8]> is one, so it runs against the accumulated buffer without touching the socket; since Read is implemented for &mut R, the cursor outlives the call and cursor.position() is precisely how many bytes the frame occupied — the one number needed to advance the buffer. Parsing against a cursor consumes nothing from the buffer, which is the invariant everything rests on. That buffer is also what reconciles sync parsing with async I/O: tokio reads into it asynchronously, protocol parses out of memory synchronously, where it can never block. Making decode_next async instead would force tokio into the one crate meant to be reusable without it and buy nothing — it is the same parsing either way. And assuming the firmware emits no malformed frames, the loop needs no error taxonomy: a failed parse just means "not yet". One cheap safeguard covers the rest: 374 bytes is the largest frame the format allows, so a buffer growing past it without yielding a frame is desynchronised rather than waiting, and the connection should be closed and reopened.