Skip to content

Ring: don't publish instance state before tokens are generated - #7901

Open
danielblando wants to merge 3 commits into
cortexproject:masterfrom
danielblando:fix/ring-tokenless-instance-never-recovers
Open

danielblando wants to merge 3 commits into
cortexproject:masterfrom
danielblando:fix/ring-tokenless-instance-never-recovers

Conversation

@danielblando

Copy link
Copy Markdown
Contributor

What this PR does:

An ingester can end up registered in the ring owning no tokens, and never recover. It serves no
writes, serves no reads, and never passes its readiness check. Restarting does not help. Getting
into this state needs -ingester.tokens-generator-strategy=minimize-spread, which is not the
default, but once an instance is in it, it cannot get out regardless of the strategy.

How it happens:

  1. autoJoin sets the instance to its target state at the top of its CAS callback, before
    generating any tokens.
  2. Under minimize-spread, generation can legitimately come back empty. The generator admits only
    the earliest-registered tokenless instance in a zone and returns nothing to the rest when
    force is false, which is how autoJoin calls it. The default random generator always
    returns the full count, so this branch is unreachable there.
  3. The callback returns a retryable error, but the instance is already in the target state owning
    no tokens. The heartbeat writes i.GetState() to the ring on every tick, so that combination
    gets persisted. The callback re-runs on each CAS retry, so this is not a brief race.
  4. On the next startup, initRing finds the entry and adopts its state from the ring.
  5. autoJoin only runs while the instance is PENDING, on a timer that fires once per process, so
    it is never reached again. The restart is what re-adopts the bad state, which is why restarting
    does not clear it.

Under minimize-spread the damage is not confined to that instance. It holds the head of its
zone's queue, so no other tokenless instance in the zone can generate either. We hit this on a cell
where 13 ingesters in one zone went tokenless and none recovered. The two oldest sat that way for
about 19 hours with zero restarts, because they never got far enough to fail.

The fix:

  • autoJoin sets the state only after tokens are secured, on both paths that publish the instance.
    This closes the window rather than narrowing it. What gets written to the ring is unchanged:
    setState now runs immediately before the existing i.GetState() read on each success path, and
    nothing between the old and new position reads the instance state.
  • initRing resets an instance found in the ring without tokens back to PENDING, mirroring the
    JOINING reset directly above it. Instances already stuck recover on restart, instead of needing
    the ring edited by hand. Forgetting such an entry only sticks while the process is down, since a
    live one re-creates it on its next heartbeat.

Which issue(s) this PR fixes:
Fixes #7899

Checklist

  • Tests updated
  • Documentation added
  • CHANGELOG.md updated - the order of entries should be [CHANGE], [FEATURE], [ENHANCEMENT], [BUGFIX]
  • docs/configuration/v1-guarantees.md updated if this PR introduces experimental flags

Signed-off-by: Daniel Blando <ddeluigg@amazon.com>
@danielblando
danielblando requested a review from a team as a code owner October 28, 2026 20:58
@danielblando
danielblando requested a review from yeya24 October 28, 2026 20:58

@sandy2008 sandy2008 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I followed the trace in the description against lifecycler.go and MinimizeSpreadTokenGenerator.GenerateTokens, and it matches: with force=false a later-registered tokenless instance gets no tokens, and the early setState let the heartbeat persist the target state with zero tokens. Moving setState after token generation is equivalent on both success paths, and both new tests fail with lifecycler.go reverted to master and pass at the head. There's one problem with the initRing reset on memberlist (inline), plus a minor note about READONLY.

Comment thread pkg/ring/lifecycler.go Resolved
Comment thread pkg/ring/lifecycler.go Resolved
danielblando and others added 2 commits October 30, 2026 00:26
Signed-off-by: Daniel Blando <ddeluigg@amazon.com>
Signed-off-by: Daniel Blando <daniel@blando.com.br>

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Ingester stuck in LEAVING when its ring entry has 0 tokens

2 participants