Skip to content

fix(cresettings): enforce single active job per config type - #23789

Queued
bolekk wants to merge 1 commit into
developfrom
fix-cresettings-job-check-per-config-type
Queued

bolekk wants to merge 1 commit into
developfrom
fix-cresettings-job-check-per-config-type

Conversation

@bolekk

@bolekk bolekk commented Sep 21, 2026

Copy link
Copy Markdown
Contributor

Problem

The check enforcing a single CRESettings job instance in core/services/cresettings/delegate.go was broken in two ways:

  1. It was global. Since cre-4958 (cre-4958: manual shard assignments #23348) added shard assignment jobs, a shard_assignment job could not run while a settings job was active, even though they write to separate loop.AtomicSettings stores.
  2. It never fired. activeJobID was only ever Loaded and CompareAndSwaped — never stored — so the another CRESettings job is already active error was unreachable, and OnDeleteJob always logged job was not active.

Fix

  • Track the active job ID per config type (settings / shard_assignment) in a sync.Map.
  • ServicesForSpec validates the config type, then claims the slot atomically via LoadOrStore — one settings job and one shard_assignment job may now coexist, while a second job of the same config type is rejected.
  • OnDeleteJob frees the slot only if it is held by the deleted job's ID (CompareAndDelete), so deleting a rejected duplicate cannot evict the active job's slot.
  • Shared configType helper guarantees the start and delete paths derive the same slot key.

Deployment Validation

  • There are no duplicate specs in any envs but for completeness check for any unexpected errors from delegate.go

@github-actions

Copy link
Copy Markdown
Contributor

✅ No conflicts with other open PRs targeting develop

@github-actions

Copy link
Copy Markdown
Contributor

I see you updated files related to core. Please run make gocs in the root directory to add a changeset as well as in the text include at least one of the following tags:

  • #added For any new functionality added.
  • #breaking_change For any functionality that requires manual action for the node to boot.
  • #bugfix For bug fixes.
  • #changed For any change to the existing functionality.
  • #db_update For any feature that introduces updates to database schema.
  • #deprecation_notice For any upcoming deprecation functionality.
  • #internal For changesets that need to be excluded from the final changelog.
  • #nops For any feature that is NOP facing and needs to be in the official Release Notes for the release.
  • #removed For any functionality/config that is removed.
  • #updated For any functionality that is updated.
  • #wip For any change that is not ready yet and external communication about it should be held off till it is feature complete.

The single-instance check was broken in two ways: it was global, so a
shard_assignment job could not run alongside a settings job, and the
activeJobID pointer was never stored, so the 'already active' error
could never actually fire.

Track the active job ID per config type (settings / shard_assignment)
in a sync.Map. ServicesForSpec validates the config type and claims the
slot atomically via LoadOrStore, so one settings job and one
shard_assignment job may coexist while a second job of the same type is
rejected. OnDeleteJob frees the slot only if it is held by the deleted
job's ID, so deleting a rejected duplicate cannot evict the active job.
@bolekk
bolekk force-pushed the fix-cresettings-job-check-per-config-type branch from f84dfd9 to a0cf664 Compare September 21, 2026 18:36
@bolekk
bolekk marked this pull request as ready for review September 21, 2026 18:36
@bolekk
bolekk requested review from a team as code owners September 21, 2026 18:36
@trunk-io

trunk-io Bot commented Sep 21, 2026

Copy link
Copy Markdown

Static BadgeStatic BadgeStatic Badge

View Full Report ↗︎Docs

@cl-sonarqube-production

Copy link
Copy Markdown

@bolekk
bolekk enabled auto-merge September 21, 2026 19:20
@bolekk
bolekk disabled auto-merge September 22, 2026 00:13
@mchain0
mchain0 added this pull request to the merge queue Sep 22, 2026
Any commits made after this event will not be merged.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants