Repository navigation
Conversation
Short urls are more reliably expressed in QR codes. The longer names are easier for human readers. Signed-off-by: Sam Barker <sam@quadrocket.co.uk>
Signed-off-by: Sam Barker <sam@quadrocket.co.uk>
Signed-off-by: Sam Barker <sam@quadrocket.co.uk>
Signed-off-by: Sam Barker <sam@quadrocket.co.uk>
Signed-off-by: Sam Barker <sam@quadrocket.co.uk>
…n and typos - define MIA and ghost batches before first use - split the duplicates section into focused subsections and retitle it - note the idempotent client assumption and mention KIP-1313 - cover dispatch on ack caveats and memory lifetime - intro the dispatch pattern tables and fix the backpressure opener - fix 'repsonse' typo in the post and nieve-dr.svg Signed-off-by: Sam Barker <sam@quadrocket.co.uk>
Signed-off-by: Sam Barker <sam@quadrocket.co.uk>
… self-contained' Signed-off-by: Sam Barker <sam@quadrocket.co.uk>
…d sequence Signed-off-by: Sam Barker <sam@quadrocket.co.uk>
… rule Signed-off-by: Sam Barker <sam@quadrocket.co.uk>
…ash window row Signed-off-by: Sam Barker <sam@quadrocket.co.uk>
ecec680 to
6b05486
Compare
- Fix asset path typo: nieve-dr.svg -> naive-dr.svg (git mv) - Fix 'shaow' -> 'shadow' typo - Add semicolon: 'every response; a sample would work well here' - Lowercase 'Stream branches' in Mind the gap intro - Fix comma splice in 'Dispatch on ack' bullet - Expand 'Dispatch on ack' bullet: resolve rhetorical question on partial successes, validate audit/telemetry use case, name the mechanical reality of decomposing and rebuilding the batch - Drop bullet: add idempotency/drop paradox note (PID discard penalty, enable.idempotence=false pattern for lossy branches) - Fix Leadership changes period -> colon for proper kramdown nesting - Convert dangling nobody* asterisk to proper footnote Signed-off-by: Sam Barker <sam@quadrocket.co.uk>
- Fix railway metaphor callback attribution (Post 1, not Post 2) - Frame replica routing section explicitly as per-request substitution, distinct from stream branching - Break up fetch-side wall-of-text paragraph - Add bridge sentence before tables acknowledging whole-request simplification - Add ack-lost-in-transit ambiguity sentence for dispatch-on-ack - Ground filter chains in Post 1 DAG concept with back-reference - Soften non-idempotent aside (no moralising) - Remove stray exclamation mark - Rewrite 'Know when to get off the train' section: replace bullet checklist with flowing prose consistent with series register Signed-off-by: Sam Barker <sam@quadrocket.co.uk>
Signed-off-by: Sam Barker <sam@quadrocket.co.uk>
6b05486 to
e005f85
Compare
Signed-off-by: Sam Barker <sam@quadrocket.co.uk>
Signed-off-by: Sam Barker <sam@quadrocket.co.uk>
|
|
||
| Yes, I know this looks like a DR solution, it's really not. By definition the branch is a best-effort copy; the DR cluster can have holes and duplicates that the primary doesn't — [Mind the gap](#mind-the-gap) goes into the gory details. You can't trust it for failover — the only way to verify nothing's missing is to compare both copies, and if both copies are online you don't need to fail over. If you need reliable replication to a DR cluster, use MirrorMaker 2 or [KIP-1279](https://cwiki.apache.org/confluence/display/KAFKA/KIP-1279%3A+Cluster+Mirroring) cluster mirroring — they were designed and built for that job. | ||
|
|
||
| Where pure duplication earns its keep is staging: a complete-ish copy of production traffic so you can test against real-shaped data, where the odd missing record doesn't affect whether your code works. |
There was a problem hiding this comment.
So I question I'd expect at this point from an awkward audience member. "Why would I want a staging tee on my production hot path? If staging starts to fail, the router will fail the pain, and so will the production client. Surely, tapping from the topic with a plain old consumer is a lower risk solution". I think the audit use-case is more compelling as you are not mixing production with lower level environments.
There was a problem hiding this comment.
I take your point, and a plain consumer is often the lower-risk way to feed staging. I've kept staging as the example though, because it's easier to picture and works better for explaining the golden rule: it's the branch you'd most expect to have a bad day, and that must never become the mainline's bad day. I think audit is an awkward fit here. This section leans on the best-effort nature of the branch, and with audit, stream integrity is highly relevant. I've reworded the paragraph to call staging the trivial example and to nod to the more realistic uses, and the next paragraph now picks up why staging would have a bad day.
|
|
||
| So far we've focused on produce requests. Branching on the fetch side took me a while to realise the value of, but there are two cases worth a look. | ||
|
|
||
| The first is a branch driven by the response: the primary's `FetchResponse` goes back to the client, and the router also sends a small record of what was read (per-consumer bytes, lag, an audit trail of who read what) down a branch line. The catch is that a delivered fetch isn't a processed one, so the numbers will drift from committed offsets, and a refetch counts the same records twice. There can still be a signal in there. Any mismatches tell operators something about what's going on out there, but it needs careful handling. |
There was a problem hiding this comment.
You could just tap into the OffsetCommitRequest/Response so you have certainty about that the group member has actually ack'd.
There was a problem hiding this comment.
Good suggestion. A successful OffsetCommitResponse is the nearest thing the protocol has to "the member says it processed this", so it's the better source if that's what you need. I've kept the fetch-driven branch for "what was delivered" and pointed at the commit tap in the text for when you need "what was acknowledged".
|
|
||
| Kafka supports rack awareness, and if you're running in the cloud you really do want it on — egress costs can be punishing. But to get it working you have to ensure every application uses `client.rack` with the correct values. If you're anything like me, you hate playing config whack-a-mole. A layer 7 proxy, one that understands the Kafka protocol, can centralise this. | ||
|
|
||
| There's one subtle obstacle though: even if a filter can see that a `FetchRequest` belongs in `eu-west-1a`, knowing that isn't enough. The client dialled broker 1 in `eu-west-1c`, so all its traffic must flow there. A filter could stamp a `rack_id` onto fetches from a misconfigured client. If the brokers are running `RackAwareReplicaSelector` the client will be pointed at a nearby replica, at the cost of an additional round trip now and again to redirect it. That only works if someone has configured the brokers for it. A router needs neither the brokers' help nor the client's. Given its own idea of which broker lives in which rack, it can send the fetch straight to a nearby replica, with no redirect. The Routing API makes that possible: the backend connection is no longer fixed at connection time, it can be chosen per request. We can now redirect the fetch to the broker in the appropriate rack rather than the one the client asked for. |
There was a problem hiding this comment.
Musing.. It occurs to me if the router could identify the client as a consumer as it connects, it could lie about a topic partition's leader in the metadata response. However, there's nothing in the protocol that tells you this connection is going to be a producer, consumer or admin. Indeed, at the protocol level, I believe a client could both produce and consumer down the same connection.
There was a problem hiding this comment.
Interesting thought, and I think you're right about the protocol: nothing in it says whether a connection is a producer, consumer or admin, and a single connection can happily do both. Lying about leadership would also be tricky to get right, since clients use leader information for more than choosing where to fetch from. I'd rather leave it out of the post, but it's one to keep in our back pocket.
|
|
||
| A Kafka log only ever appends. When a batch lands on one cluster and fails on the other, nothing can take the successful write back. That leaves the branch out of step with the primary in two ways, and we'll need names for them: | ||
|
|
||
| - An **MIA batch** is one the primary has and the branch doesn't. The branch is behind, and has a hole. |
There was a problem hiding this comment.
Expanded to "missing in action" at the definition, and at its first mention in the replica routing section.
|
|
||
| #### Filters that change the batch count break the numbering | ||
|
|
||
| Supporting idempotent producers on a branch line is no different from supporting them anywhere else in the filter chain, though the problems are much more obvious here. A filter that adds or removes batches breaks the numbering, so it has to manage the sequence itself. |
There was a problem hiding this comment.
This got me thinking about Record Encryption/AEAD/Stream branching. When we enhance RE allow the batch identifier to be bound, we'd have an undecryptable stream.
There was a problem hiding this comment.
I think the general point is that any binding of batch body to batch headers has to be made over the version the broker will persist. The binding can happen at any node in the DAG, but once it has, nothing downstream can change the batch, like a branch re-stamping its PID and epoch. That covers AEAD with the batch identifier bound in, and anything similar. The runtime can't readily police it, so it's down to whoever writes the config. I've added a paragraph to the branch identity section saying so, without naming Record Encryption.
|
|
||
| Both are the same failure. The client's retries for a partition move from one branch identity to another, and the branch can only deduplicate within a single identity. The mainline is unaffected — its PID belongs to the client, which holds it across every connection, every proxy instance, and every reconnect, so the primary recognises retries and discards the ones it has already committed. | ||
|
|
||
| You could fix this inside one process by letting the routers share a PID cache. But nobody[^single-proc] runs a single proxy process in production, and across processes there's nothing to share. For that you need an external state store. |
There was a problem hiding this comment.
Could you derive the the branch's PID, deterministically, from the client PID by combining with the virtual node id to this point in the DAG, so that every replica would arrive at the same result?
There was a problem hiding this comment.
Yes, and the broker won't stop you. ProducerStateManager.prepareUpdate() takes any PID and hands back an empty entry if it hasn't seen it before — no check that the ID was actually issued by this cluster. checkSequence() accepts a never-seen PID at any epoch as long as the first batch on that partition starts at sequence 0. So every proxy replica that handles the same client can derive the same branch PID without calling InitProducerId.
But it doesn't get us very far, because sequence is the harder half. After a reconnect or failover the new instance knows the right PID but not where the branch sequence left off. If sequence 0 is still in the broker's deduplication window (last 5 batches) it gets quietly discarded. If the sequence has moved past that window, seq 0 is out-of-order and the broker rejects it, stalling the PID on that partition until the router bumps the epoch and resets — at which point anything in the gap is lost silently. You can't get the sequence right without a state store, and once you have a state store you don't need the derived PID that badly — you can just store the broker-issued one alongside the sequence. So PID derivation might be worth exploring as a building block, but it doesn't change the fundamental answer for the post.
Summary
adds the next blog post in the series. All about stream branching.
It also sets up re-direct for my slide deck for current.
Files Changed
📄
_data/redirects/blog.yamlAdds eight new entries under
mappings, all in theroutingsubgroup. All of them point to the same absolute target. Some of them TBCcsandconnection-switchingsbandstream-branchingtwandtopic-weavingpwandpartition-weavingEach concept has both an abbreviated and a full-name alias. The existing
benchmark-datamapping is unchanged. The abbreviated names work a lot better for QR codes the long names are for human readers.