server: shorten capture removal delay on rolling restart

[asddongmen]

What problem does this PR solve?

Issue Number: close #6203

During a rolling restart, a capture whose etcd key has been deleted remains in the scheduler-visible node view for the full captureRemoveTTL (10 seconds by default). This can keep a new coordinator waiting for a bootstrap response from an exited capture after every live capture has responded. Removing the node immediately is unsafe while the old capture may still hold a valid write proof.

What is changed and how it works?

• Advertise the current write-lease protocol in capture metadata.
• After every local module and sink closes successfully, conditionally publish a lease-bound write-stopped marker before deleting the capture key. Observers remove a marked capture immediately.
• For an unmarked replacement with a different ID, the same address, a non-older start time, and current protocol support on both registrations, shorten the delay to the five-second maximum write-proof lifetime.
• Keep the existing conservative captureRemoveTTL for legacy, capability-unknown, mismatched, or failed-shutdown cases.
• Add focused server/orchestrator tests and update the write-lease design documentation.

Check List

Tests

• Unit test
  • go test ./pkg/orchestrator ./server -count=1
  • go test ./pkg/config ./pkg/metrics -count=1
  • Focused go test -race for the new paths
  • make fmt
  • make cdc

Questions

Will it cause performance regression or break compatibility?

No expected regression. Address matching scans membership only on capture events. The metadata fields are additive; legacy or unknown nodes retain the existing 10-second fallback.

Do you need to update user documentation, design documentation or monitoring documentation?

Yes. The capture write-lease design and overview, plus the delay metric description, are updated.

Release note

Reduce TiCDC capture removal delay during rolling restarts while preserving conservative write-fencing fallbacks.

Summary by CodeRabbit

• New Features

  • Capture registrations now report write-lease compatibility and whether writes have stopped.
  • After a clean shutdown, stopped captures can be removed immediately.
  • Matching same-address replacements can be removed after a shorter safety interval.
  • Unrecognized or ambiguous replacements continue to use a conservative removal delay.

• Bug Fixes

  • Prevented stale shutdown signals from affecting a newer capture registration.

• Documentation

  • Updated guidance on capture removal, replacement, safety, and observability.

[coderabbitai]

Navigate logical layers of code changes, visualize relationships, and explore their blast radius.

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info

⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Advanced

Run ID: 4620133d-b99b-4467-88aa-76294355dbbb

📥 Commits

Reviewing files that changed from the base of the PR and between 008f5d8 and 47fe025.

📒 Files selected for processing (1)

• tests/integration_tests/move_table/run.sh

Included review availability: Your plan provides up to 1 included review per hour; 0 remain after this review.

---

📝 Walkthrough

Walkthrough

The change adds write-lease state to capture metadata, publishes a lease-checked stopped marker during clean shutdown, and updates reactor removal scheduling. Matching same-address current-protocol replacements use a five-second delay; other cases use the conservative removal TTL. The move-table integration test also adds CDC server restart guards.

Changes

Capture write-lease removal

Layer / File(s)	Summary
Removal policy and safety contract docs/design/capture-write-lease-design.md, docs/design/capture-write-lease-overview.md	The design defines graceful-marker removal, five-second same-address replacement removal, and conservative TTL removal. It updates the safety proofs, examples, test coverage, and implementation references.
Registration and graceful shutdown pkg/config/capture.go, server/server.go, server/server_prepare.go, server/capture_registration_test.go	Capture metadata includes protocol and stopped-state fields. Clean shutdown writes the stopped marker only when module closure succeeds and the capture key still uses the expected lease.
Reactor removal scheduling pkg/orchestrator/reactor_state.go, pkg/metrics/write_lease.go	The reactor detects qualifying replacements, removes stopped captures immediately, and applies either the five-second fenced delay or configured fallback delay. The metric help text describes the conservative fallback.
Removal behavior validation pkg/orchestrator/reactor_state_capture_test.go, server/capture_registration_test.go	Tests cover marker identity, immediate removal, replacement ordering, shortened delays, conservative fallback, and state cleanup.

Move-table integration test guards

Layer / File(s)	Summary
Guard CDC servers during move-table validation tests/integration_tests/move_table/run.sh	The test starts all three CDC servers with restart guards, checks the guards during validation, and stops them after the final sync-diff check.

Priority: ➖ Normal

Estimated code review effort: 3 (Moderate) | ~25 minutes

Change: Bug fix · Severity of issue fixed: Medium

Sequence Diagram(s)

sequenceDiagram
  participant CaptureServer
  participant Etcd
  participant GlobalReactorState
  participant ReplacementCapture
  CaptureServer->>Etcd: publish lease-bound write-stopped metadata
  Etcd-->>GlobalReactorState: report capture-key deletion
  GlobalReactorState->>GlobalReactorState: remove stopped capture immediately
  ReplacementCapture->>GlobalReactorState: register same-address current-protocol capture
  GlobalReactorState->>GlobalReactorState: apply five-second fenced delay

Loading

Suggested reviewers: wk989898, 3aceshowhand

Merge Risk: ⚪ Minimal · up to 47fe0

The change appears ready to merge after normal checks. No actionable failure was established in the shortened removal paths or the updated integration test.

🚥 Pre-merge checks | ✅ 4 | ❌ 1

❌ Failed checks (1 warning)

Check name	Status	Explanation	Resolution
Docstring Coverage	⚠️ Warning	Docstring coverage is 5.88% which is insufficient. The required threshold is 80.00%. Docstring coverage is scoped to functions touched by this diff. Analyzed 17 functions across 8 files.	Write docstrings for the functions missing them to satisfy the coverage threshold.

✅ Passed checks (4 passed)

Check name	Status	Explanation
Title check	✅ Passed	The title clearly and concisely describes the main change: shortening capture removal delays during rolling restarts.
Description check	✅ Passed	The description includes the issue reference, problem statement, implementation details, tests, compatibility assessment, documentation updates, and release note.
Linked Issues check	✅ Passed	The PR implements the coding objective in [#6203]. CaptureInfo advertises the write-lease protocol and WriteStopped. The server publishes WriteStopped only after all module groups close successf…
Out of Scope Changes check	✅ Passed	The changed server code, reactor code, metrics text, tests, and write-lease design documents support the capture-removal and rolling-restart behavior in [#6203]. The move-table integration-test change…

• Fix all pre-merge checks with AI

✨ Finishing Touches

🧪 Generate unit tests (beta)

• Create a new PR

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{A rabbit watched the leases glow,
Then saw the stopped-marked capture go.
Five seconds passed for matching peers,
While fallback waits kept older gears.
The guards stood watch through every move,
And carrots danced to mark the groove.}

_{Comment @coderabbitai help to get the list of available commands.}

[asddongmen]

/test all

[ti-chi-bot]

[LGTM Timeline notifier]

Timeline:

• 2026-09-14 04:37:00.700589037 +0000 UTC m=+7066.638246631: ☑️ agreed by 3AceShowHand.
• 2026-09-14 07:14:22.491735811 +0000 UTC m=+16508.429393415: ☑️ agreed by wk989898.

[asddongmen]

/check-issue-triage-complete

[ti-chi-bot]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: 3AceShowHand, nongfushanquan, wk989898

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Details

Needs approval from an approver in each of these files:

• OWNERS [3AceShowHand,nongfushanquan,wk989898]
• pkg/config/OWNERS [nongfushanquan]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[asddongmen]

/test pull-cdc-mysql-integration-light-next-gen
/test pull-cdc-mysql-integration-heavy-next-gen

[asddongmen]

/test pull-cdc-mysql-integration-light-next-gen

[asddongmen]

/test pull-cdc-mysql-integration-light-next-gen

[asddongmen]

/test pull-cdc-mysql-integration-heavy-next-gen

[asddongmen]

/test all

[asddongmen]

/test all

[ti-chi-bot]

@asddongmen: The following tests failed, say /retest to rerun all failed tests or /retest-required to rerun all mandatory failed tests:

Test name	Commit	Details	Required	Rerun command
pull-cdc-mysql-integration-heavy-next-gen-legacy-safepoint	f0b6db1	link	unknown	/test pull-cdc-mysql-integration-heavy-next-gen-legacy-safepoint
pull-cdc-mysql-integration-heavy-next-gen	f0b6db1	link	true	/test pull-cdc-mysql-integration-heavy-next-gen
pull-cdc-pulsar-integration-heavy	47fe025	link	false	/test pull-cdc-pulsar-integration-heavy

Full PR test history. Your PR dashboard.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here.