add metric for monitor max queued record num shard by flashmouse · Pull Request #905 · confluentinc/parallel-consumer

flashmouse · 2026-01-04T09:59:30Z

I want to add a metric to monitor the maximum number of queued records for the shard. This metric is especially useful for orderType.KEY. If this metric is high and all messages are consumed at the same speed, it implies that some shards may contain a hot key with lots of records.

Checklist

Documentation (if applicable)
Changelog

…onfluentinc#905) Cherry-pick of confluentinc#905 (author: flashmouse). Adds a SHARDS_MAX_SIZE gauge that reports the record count in the most-loaded shard. Useful with KEY ordering to detect hot-key bottlenecks. Also simplifies .keySet().size() to .size(). Upstream PR: confluentinc#905 Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

Add a "Parallel-safe work while PR #57 is in flight" section to docs/inflight.md recording, for each in-flight track, whether it collides with PR #57's metrics/state files (857, 909, 51 -> sequence after) or is parallel-safe (912, release, logging cleanup, security bumps, contributor fixes, #40, confluentinc#915, DLQ), ranked by readiness. Also refresh the confluentinc#859 entry to the consolidated PR #57 (bundles the confluentinc#893/confluentinc#905 cherry-picks, supersedes the closed #42->#43->#45 stack) and expand the confluentinc#912 entry (ready, pushed, no PR, vertx-isolated). Bump the last-updated date. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

…onfluentinc#905) Cherry-pick of confluentinc#905 (author: flashmouse). Adds a SHARDS_MAX_SIZE gauge that reports the record count in the most-loaded shard. Useful with KEY ordering to detect hot-key bottlenecks. Also simplifies .keySet().size() to .size(). Upstream PR: confluentinc#905 Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

Render backlink comments from an optional per-entry backlink field in upstream-map.yaml (source of truth) instead of a separate body, with the same {{FORK_REPO}}/{{FORK_REF}}/{{SUMMARY}}/{{ID}} placeholders; entries without it fall back to the generic templates. bug-859 uses it to explain the two-cause leak vs the already-merged upstream confluentinc#892. Make fork status honest about landed-ness. The old "fixed" conflated "fix written" with "shipped": every "fixed" entry is actually an OPEN, unmerged fork PR (or a branch with no PR). Replace with a lifecycle vocabulary (none|in-progress|ready|pr-open|merged|released|superseded|wontfix) and correct the entries: confluentinc#859/confluentinc#893/confluentinc#905 -> pr-open (in open PR #57), confluentinc#857 -> ready (branch-only). Add an optional per-entry todo: list for outstanding actions (merge the open PR, post the backlink) surfaced by "upstream-map.py todo", so "still to do" is explicit rather than implied by an open PR + null forwarded. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

…tus guard Audited PR #57 against issue confluentinc#859. The code matches the issue (registeredMeters List -> LinkedHashSet + prune, plus caching OffsetMapCodecManager in PartitionStateManager), but the wording framed the leak as rebalance-driven when the issue is commit-driven. Tighten bug-859 summary/notes/backlink: the List accumulated a duplicate Meter.Id on every registration (every commit); fork PR #57 fixes it via List->Set + PartitionStateManager caching (also closes #233); upstream confluentinc#892 covers the per-commit churn; confluentinc#893/confluentinc#905 are unrelated cherry-picks. Also fix upstream-backlink.sh: the fix-backlink status guard still checked the removed "fixed" value, so it refused every entry after the lifecycle-status change. Now allows ready|pr-open|merged|released and refuses none|in-progress. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

The description repeated itself ("queued records in the shards with the most queued records"). State it plainly: the maximum records queued in any single (most-loaded) shard. Upstream-PR: confluentinc#905 Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

…onfluentinc#905) Cherry-pick of confluentinc#905 (author: flashmouse). Adds a SHARDS_MAX_SIZE gauge that reports the record count in the most-loaded shard. Useful with KEY ordering to detect hot-key bottlenecks. Also simplifies .keySet().size() to .size(). Upstream PR: confluentinc#905 Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

The description repeated itself ("queued records in the shards with the most queued records"). State it plainly: the maximum records queued in any single (most-loaded) shard. Upstream-PR: confluentinc#905 Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

Record the user-visible changes this PR introduces under the unreleased 0.6.0.0 section: the PCMetrics memory-leak fix, the accurate-committed-offset fix, and the new shards.max.size metric. Follows the fork/upstream reference convention. Upstream-Issue: confluentinc#859 Upstream-PR: confluentinc#893 Upstream-PR: confluentinc#905 Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

Add docs/runbooks/pr57-post-merge.md - the exact, reviewed upstream comments to post after #57 merges: the confluentinc#859 fix-backlink (from the manifest backlink field), and tailored author-crediting notes for the carried confluentinc#893/confluentinc#905 PRs (the generic template is wrong for cherry-picked PRs), plus the manifest status flips. Update AGENTS.md backlink guidance: always pre-draft backlinks as a runbook committed to the PR - reviewable in-diff, tailored per target, context-aware across targets - rather than running the backlink script blind. Runbooks live in docs/runbooks/, deleted once executed; stragglers swept at the next major release. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

Cover the new max-queued-records-per-shard metric next to the existing SHARDS_SIZE assertion in PCMetricsTest - the larger of the two per-partition shard remainders. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

…ntinc#905 gauge perf Two review findings: static errorPolicy is set-once since #57, so multiple PC instances in one JVM with differing InvalidOffsetMetadataHandlingPolicy would all use the last-constructed policy (fix via instance field, part of the #233 de-static work); and the SHARDS_MAX_SIZE gauge re-walks shard queues, duplicating the SHARDS_SIZE traversal (negligible). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

Add a Shards Max Size entry to the metrics reference in README_TEMPLATE.adoc (next to Shards Size) and regenerate README.adoc. The confluentinc#905 gauge was user-visible but undocumented in the metrics list. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

…tinc#893 names itself The changelog-issue gate that master gained in #104 flags two entries on this branch. One is a real omission: upstream PR confluentinc#893 says "issue confluentinc#894" in its own body, and confluentinc#894 is "Offset reset when frequent rebalancing" - precisely the problem the entry describes. It now cites it. upstream-map.yaml recorded issues: [] for this cherry-pick, so the map was wrong rather than merely thin. Corrected to [894], with a note saying what it used to say, and last_checked moved to today. The other flagged entry, the shards.max.size metric, genuinely has no issue: upstream confluentinc#905 opens "I want to add a metric..." and cites nothing. Its related: [71] is "Health-checks", far too general to claim as the request. That one takes the gate's changelog-ref opt-out in the PR body rather than an invented citation - which is the trade the gate's own comments call for. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01QqHpNSXC39ANv9kG1ZvUzn

…he plan Phase 2 asks for a rehearsal on three throwaway issues before the bulk run. It was instead run on the six the fork has actually worked on - upstream #233, confluentinc#326, confluentinc#857, confluentinc#859, confluentinc#894, confluentinc#912 - mirrored as #117-#122. Real content tests the scheme harder than low-traffic issues chosen for being safe. Four things it changed, all of which would have bitten the bulk run: - AUTO-CLOSE REPLACES THE POST-MERGE STEP. "Fixes #NNN" in the fixing PR closes the mirror on merge. That deleted an artefact outright: pr57-post-merge.md existed only because closing and announcing had to be remembered by hand. - "ALREADY FIXED" NEEDS THREE STATES, NOT TWO. Only three of the six could honestly be closed. #233 is partial, confluentinc#857 is two-thirds fixed with the original deadlock still open, and confluentinc#326's landed fix only MIGHT be the same bug. A binary field would have produced three wrong closures, so the script must never infer "fixed" from a linked PR existing. - UPSTREAM PRs NEED COMMENTS TOO, though they are correctly not mirrored. confluentinc#893 and confluentinc#905 are carried in #57; their authors deserve to know their work ships, and readers deserve the pointer. - POSTING UPSTREAM IS GATED. Bodies are drafted and reviewed, but bulk posting into confluentinc/parallel-consumer was refused by tooling permissions. Worth knowing before Phase 2 step 4 assumes 78 comments can just be fired. Also records fork_issue in upstream-map.yaml as each mirror is created rather than as an end-of-migration sweep, which keeps the map honest in the interim. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01QqHpNSXC39ANv9kG1ZvUzn

The house convention is that a bare #NN means THIS fork and upstream is written `upstream #NN`. Bare `confluentinc#859` in source was merely ambiguous before; now that the mirror #120 exists - and fork confluentinc#859 will eventually exist and mean something else - it is wrong. 27 refs qualified across the Java sources and docs/refactoring.md this PR already touches: #200, #233, confluentinc#857, confluentinc#859, confluentinc#893, confluentinc#905. Comments and one assertion message only; no logic. This is the "correct opportunistically as files are touched" rule from #114's Phase 4, not the bulk rewrite that plan rejects. Deliberately left alone, because a mechanical sweep gets them wrong: - CHANGELOG.adoc - its own header states that entries below 0.6.0.0 predate the fork and their #NN already refer to upstream, and the 0.6.0.0 entries use fully qualified links. Rewriting either would be wrong. - upstream-map.yaml - every number in it is upstream by construction, so prefixing all of them is noise, not clarity. - AGENTS.md - the only hit is a commit-subject convention shown BY EXAMPLE; rewriting the example would silently change the documented convention. Note what this does NOT do: renumber references to fork issue numbers. #114 rejects that, and three of its four reasons still stand - git history is immutable, a mechanical remap cannot tell an issue from the #999/#123 test offsets in Java source, and fork numbering will eventually collide with upstream's range. The fourth reason, staying cherry-pickable upstream, no longer applies. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01QqHpNSXC39ANv9kG1ZvUzn

Antony Stubbs (astubbs) · 2026-08-04T23:43:44Z

I am carrying this PR in my fork, in astubbs#57, because it has not merged here. Credit and authorship are unchanged - it is cherry-picked as-is, so the shards.max.size metric is set to ship in the fork's 0.6.0 release.

The Confluent-hosted version of this project is no longer maintained - its README says so. Parallel Consumer itself continues in my fork: https://github.com/astubbs/parallel-consumer

Nothing in this repository is closed or changed by this comment.

The six mirrored issues (upstream #233, confluentinc#326, confluentinc#857, confluentinc#859, confluentinc#894, confluentinc#912) and the two carried PRs (confluentinc#893, confluentinc#905) all have their backlink comment as of 2026-08-05. Comment URLs recorded in upstream-map.yaml under forwarded: for the entries the map covers, plus fork_issue for confluentinc#857 -> #119 and confluentinc#912 -> #122. upstream #233 and confluentinc#326 have no map entry; their mirrors are the record. Two findings promoted into the plan, because both change what the Phase 1 script must do: POSTING NEEDS A HUMAN. Agent tooling refused to bulk-post into another org's repo, so these went out through a reviewed script run by hand. Phase 2 step 4 should not assume 78 comments can be fired by automation. WRITE FOR THE READER, NOT THE TRACKER. Five review passes cut about a third of the drafted text. What went: PR base branches and commit-counts, "so the fork mirror stays open" (our bookkeeping, not the reporter's problem), a walkthrough of LinkedHashSet vs List, and hedging - "a maintained community fork", "one of its original authors" - which understates a fork that is ~99% one person's commits. Two rules fall out: link the PR rather than describe its state, and never write a sentence that goes stale in a week. Also: "this project is no longer maintained" reads, on an upstream issue, as "Parallel Consumer is dead". Name the Confluent-hosted version specifically and say the project continues. Every one of the eight now does. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01QqHpNSXC39ANv9kG1ZvUzn

flashmouse · 2026-08-05T02:38:05Z

I haven't been following Kafka's official releases in a long time, and recently noticed that KIP-932 is now marked production-ready and may cover the majority of what this project does — except for the key-ordered parallel consumption mode. But I still think that capability is important. Thanks for your continued work on this!

I think we may switch to your fork once you've cut a new release.

Antony Stubbs (astubbs) · 2026-08-06T00:14:33Z

flashmouse cool - yeah KIP-932 is addressed here: https://github.com/astubbs/parallel-consumer#when-to-use. I'll be adding a better section about it in the upcoming documentation issue.

… at us The sweep searched `updated:>=<last_swept>`, and the backlink run commented on all 78 mirrored upstream issues - which bumped `updated` on every one of them. The report became the entire upstream tracker, with no way to see the two items a human had actually touched. A watcher that fires on everything is the same as no watcher. It now reports an item only when someone other than us moved it: opened inside the window, or commented in inside the window by an account that is not the authenticated user. On the current window that is 78 items down to 2 - upstream confluentinc#885, where vivahu replied to our 0.5.3.3 answer, and upstream PR confluentinc#905, where flashmouse replied about us carrying their PR. Both are exactly the outreach responses the sweep exists to surface. Also corrects a tracked ref the sweep caught: upstream PR confluentinc#548 was recorded `open`, the file header claimed `CLOSED`, and GitHub says merged 2023-04-03. The entry now says merged; the header's STATE NOTE is gone, since it duplicated per-entry status and had already drifted from it. last_swept bumped to 2026-08-06. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01QqHpNSXC39ANv9kG1ZvUzn

Commit 8923603 resolved five conflicts with `git checkout --theirs`, which takes master ENTIRE file rather than the conflicting hunk. Where a file had changes from both sides, everything this branch had done in it went with it. Two casualties, both caught by the reviewer running the tests rather than reading the diff: ShardManager lost the confluentinc#905 SHARDS_MAX_SIZE gauge - the field and its registration - while PCMetricsDef.SHARDS_MAX_SIZE and the PCMetricsTest assertion on it merged cleanly and still expected it. PCMetricsTest.metricsRegisterBinding was failing with expected 785.0 but was -1.0, the no-such-gauge sentinel, and CHANGELOG.adoc still advertised the metric as delivered. The TODO(refactor) marker and the .size() cleanup went the same way. PartitionStateCommittedOffsetTest lost offsetToCommitIsComputedOncePerCommit, the confluentinc#893 regression test. The production fix was untouched, so nothing was functionally broken - but the test that would catch a regression back to the two-call dirty read was gone, which is the worse half of the two. Restored both, keeping master side where the sides genuinely differed: confluentinc#857 in ShardManager, and the tree-wide sweep wording in the test. Neither restore reintroduces the role form the gate now rejects. Verified by running them: PartitionStateCommittedOffsetTest 6/6, PCMetricsTest 2/2. The lesson is that --theirs and --ours are file-level operations, and a conflict is hunk-level. Resolve the hunk, or diff the pre-merge tip afterwards to see what the shortcut discarded. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01QqHpNSXC39ANv9kG1ZvUzn

flashmouse added 2 commits January 4, 2026 16:54

add metric for max queued shard

dc1f9eb

add doc

c6d81d2

flashmouse requested a review from a team as a code owner January 4, 2026 09:59

Antony Stubbs (astubbs) mentioned this pull request Apr 18, 2026

feat: add max-queued-records-per-shard metric (cherry-pick Confluent #905) astubbs/parallel-consumer#43

Closed

2 tasks

claude Bot mentioned this pull request Aug 4, 2026

docs: audit the in-flight ledger against reality, then split it so PRs stop colliding astubbs/parallel-consumer#112

Merged

5 tasks

Antony Stubbs (astubbs) mentioned this pull request Aug 6, 2026

confluentinc#27: Micrometer metrics astubbs/parallel-consumer#125

Closed

Navigation Menu

Uh oh!

add metric for monitor max queued record num shard - #905

add metric for monitor max queued record num shard#905
flashmouse wants to merge 2 commits into
confluentinc:masterfrom
flashmouse:add-shards-metric

flashmouse commented Jan 4, 2026

Uh oh!

Antony Stubbs (astubbs) commented Aug 4, 2026

Uh oh!

flashmouse commented Aug 5, 2026

Uh oh!

Antony Stubbs (astubbs) commented Aug 6, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Uh oh!

Conversation

flashmouse commented Jan 4, 2026

Checklist

Uh oh!

Antony Stubbs (astubbs) commented Aug 4, 2026

Uh oh!

flashmouse commented Aug 5, 2026

Uh oh!

Antony Stubbs (astubbs) commented Aug 6, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants