Cross-server schema cache consistency without broadcast storms

[bitflicker64]

Scope and current status

Updated 2026-09-18. This discussion tracks cross-server schema-cache convergence, including missed notifications and excessive invalidation traffic. Restoring a watch connection is necessary, but does not by itself make schema caches consistent.

A schema change on Server A can remain invisible on Server B because the publisher emits no notification, the watch stops, an event is lost during disconnection, or the consumer fails to invalidate its cache. These are different failure points in the same propagation path.

flowchart LR
    A["Server A: schema mutation"] --> B["Publish invalidation"]
    B --> C["PD watch"]
    C --> D["Server B: invalidate cache"]
    V["Proposed durable schema version"] -. "periodic reconciliation" .-> D
    A -. "proposed version update" .-> V

Loading

• fix(pd): keep KvClient watches alive after reconnect failures #3157 repairs watch recovery, the transport segment in this diagram. It does not add event replay, missing schema notifications, or cache reconciliation.
• This discussion covers the end-to-end schema contract. The version/reconciliation path is a proposal, not shipped behavior or an approved release requirement.
• [Bug] Server graph metadata can permanently diverge from PD when graph watch events are missed #3151 tracks graph metadata divergence, a related but distinct scope: graph configuration/lifecycle visibility across Servers. Share recovery principles without treating graph lifecycle and schema-cache invalidation as interchangeable.

Timeline and related work

Dates below use UTC; merged dates are distinguished from issue reports and review updates.

Date	Item	What changed / what remains
2024-08-06	#2617, reported; now closed	Multi-server HStore schema visibility depended on which Server answered. The historical issue being closed does not establish complete cache convergence.
2026-05-06	#3011, merged	Added cross-server HStore schema-cache clear propagation. Updates/status changes and conditional removal notification remain relevant here.
2026-05-15	#3017, merged	Aligned actions on the local EventHub path; this is not a cross-server reconciliation mechanism.
2026-06-23	#3062, merged	Restored terminal-watch recovery for the etcd meta driver. A fresh subscription still does not replay the disconnected interval; this is a separate driver from PD KvClient.
2026-08-12	#3138, merged	Closed the creating Server's local graph-binding race. Other replicas' graph readiness remains under #3137.
2026-08-12	#3151 / #3152, opened	Separated missed graph-metadata events from PD watch permanently stopping after reconnect failure. Both remain open at this update.
2026-08-15	#3157, opened	Per-subscription PD watch recovery, stale-callback rejection, and separation of transport cleanup from client shutdown.
2026-09-17	#3157 review and fixes	eeb7041e8 preserves ordinary KV/lock timeout budgets and unpublishes SchemaDriver before cleanup. Five independent review lanes; 41 KvClient + 5 SchemaDriver tests passed.
2026-09-18	#3157 status checkpoint	Still open, head eeb7041e8; 24 CI checks passed. One legacy streaming override compatibility discussion remains. No event replay or reconciliation was added.

Recommended order: land the bounded watch-recovery fix in #3157 independently; verify and address the small publisher/consumer defects below; then agree and implement the convergence contract in this discussion. Do not expand #3157 into the full schema synchronization redesign, and do not close this discussion merely because #3157 merges.

Background

The remaining gap exists even with a healthy watch. CachedSchemaTransactionV2.updateSchema() updates its local cache but suppresses cross-server notification to avoid broadcast storms from background status changes. Removal notification is gated by task.sync_deletion. Reconnecting cannot deliver a notification that was never emitted, nor recover an event from before the fresh subscription.

Multiple Server pods behind one Kubernetes Service expose this as inconsistent schema visibility depending on the responding pod. The proposal below aims to bound stale caches without turning every mutation into a full cluster-wide reload.

Why the quick fix does not work

Re-enabling the suppressed notification looks like a one-line fix. The numbers say otherwise. An index rebuild flips status 2 to 3 times per label (IndexLabelRebuildJob.java:106,137,143). Take M = 1,000 status flips over 60 s on N = 10 servers with 5,000 schema elements per graph:

• 1,000 event puts become 10,000 watch deliveries;
• each delivery is a full schema-cache clear (clearSchemaCache, CachedSchemaTransactionV2.java:131-152), so 10,000 full clears;
• every cleared server lazily reloads all four schema types, and the V2 name-miss path runs a full reload inline on the request thread (:414-420), so concurrent readers pile up: roughly 40,000 prefix scans for the duration of the job, with latency spikes on every server.

Cluster-wide work scales as O(M x N). Any acceptable design has to make cluster-wide work independent of M. The same math applies to today's shipped add path: bulk-creating 500 schema elements already produces 500 uncoalesced events and 500 full clears per remote server.

What propagates today

Mode	add	update / status	remove	transport
hstore + PD (V2)	yes, one event per op	no, silent	gated off when task.sync_deletion=true	one PD key, full clear per event
HBase, N servers (V1)	local JVM only	local only	local only	RPC notifier exists but is off by default and needs a static peer list
rocksdb + raft	covered	covered	covered	raft log replay
single node	covered (#3017)	n/a	n/a	in-process EventHub

Two details make the multi-server rows worse than they look. The task.sync_deletion gate (maybeNotifySchemaCacheClear, CachedSchemaTransactionV2.java:360-367) assumes the caller invalidates other nodes synchronously, but syncWait only waits for the local task, so a removed label can survive in remote caches indefinitely. And the PD watch client re-subscribes after a leader change or error with no resume point (KvClient.java:~168-190), so any design whose correctness depends on event delivery is wrong by construction.

Proposed design: three rules

The historical tension exists because the event is the only carrier of truth: losing one loses the information forever, so nothing dared coalesce or drop them. The proposal separates truth from wake-up.

Rule 1: a version register, written synchronously on every mutation. Each graph gets a small version value stored next to the schema itself: a PD KV key for hstore (HUGEGRAPH/{cluster}/SCHEMA_VERSION/{graphSpace}/{graph}, an opaque token compared only for equality), a counter row on the existing counter table for HBase (no DDL, no migration). Every schema mutation, including updates and status flips, writes it on the DDL thread right after the schema write. One cheap put with zero fan-out; the storm was fan-out times full reloads, never the publisher's own puts. If the register put fails after the schema commit, the DDL call still succeeds, and a background retry with metrics (register_write_failures, register_dirty) takes over until it lands.

Rule 2: events become coalesced, lossy hints. A debounced flusher (500 ms window) publishes per-id invalidation hints on a new EVENT/GRAPH/SCHEMA/DELTA key, at most about 2 events per second per graph no matter how fast mutations come. Hints only invalidate cache entries; they can never mark a node as up to date. That makes dropping, merging and reordering them harmless, which is exactly what allows unlimited coalescing. For compatibility, bursts containing add/remove/clear also put one event per window on the legacy CLEAR key, byte-identical to today's payload, so old servers and store nodes see what they see today, only fewer of them.

Rule 3: a node advances its applied state only through full resync. A reconciler polls the register (default every 10 s per graph, one prefix scan per server per tick) and, on mismatch, clears the graph's caches and adopts the value it read before clearing. Nothing else advances the applied state, not even the hints. The proposed target is convergence within two successful reconcile intervals (20 s with a 10 s interval), after the version update is durably visible and under bounded read/reload latency. This is not an unconditional deadline during PD unavailability. A schema commit followed by a failed version write and publisher crash can lose an in-memory retry; durable recovery or an atomic publication contract must be established before claiming a bound after every successful DDL. Typical hint-path latency also needs measurement.

For the storm scenario above, this comes out to at most 121 events on the wire and about 40 full clears cluster-wide instead of 10,000, a roughly 250x reduction, and doubling M changes nothing on the receiver side. Schema read hit paths gain zero instructions; the register is only touched on writes and by the reconciler.

Proposed convergence targets, per deployment

Mode	staleness bound after a mutation	config needed
hstore + PD, N pods	Hint delivery for the fast path; target two successful reconcile intervals after durable version publication	Proposed on by default; not implemented
HBase, N servers	Poll-based convergence target; optional scope still undecided	See Q1; not implemented
rocksdb + raft	unchanged (raft already covers it)	none
single node	unchanged, zero	none

An emergency switch (schema.sync.enabled=false) restores the legacy event behavior. Rolling-upgrade compatibility is a design goal: keep legacy payloads readable, then validate mixed-version ordering, coalescing delays and recovery before claiming safety in any pod order.

Candidate preparatory fixes to verify independently

The original analysis identified the following publisher/consumer defects or optimization candidates. Recheck each on the target head before changing code; #3157 does not claim to fix them. The JSON payload item concerns Store-side SchemaDriver, not proof that a Server watch failed:

1. Store nodes mis-parse every schema-clear event today: SchemaDriver.schemaCacheClearHandler (SchemaDriver.java:137-147) still splits the payload on -, but the server has emitted JSON since fix(server): sync hstore schema cache clears #3011, so store-side schema caches silently never clear.
2. SchemaTransactionV2.saveSchema publishes notifyGraphVertexCacheClear/notifyGraphEdgeCacheClear PD events on every VertexLabel/EdgeLabel save, and nothing anywhere subscribes to them: pure write amplification inside the schema write path.
3. The task.sync_deletion gate described above, which also suppresses STORE_CLEAR/TRUNCATE propagation.
4. PdMetaDriver.extractValuesFromResponse returns null for a watch batch containing any non-Put event, swallowing the whole batch for every meta listener in the process.
5. The V2 name-miss path reloads the entire schema inline on the request thread, so concurrent misses after a clear stampede the backend.

Who this pings, and why

• @imbajin raised the constraint this design is built around (no broadcast storms) and most open decisions need a maintainer call.
• @dpol1 wrote fix(server): sync hstore schema cache clears #3011, fix(server): align cache event actions in legacy EventHub path #3017 and fix(core): self-heal etcd meta watch on transport reconnect #3062; this design extends and partly replaces that machinery, including the task.sync_deletion gate, so review from that side matters most.
• @haohao0103 reported [Bug] cacheEventListener in CachedSchemaTransactionV2 Work does not meet expectations #2617 from a real deployment and is best placed to confirm the failure mode described here matches what was observed.
• @Tsukilc has the most recent commits on the V2 cache and meta code and wrote SchemaDriver in hugegraph-struct, where bug 1 lives.
• @koi2000 for PD watch semantics (question 4 in the comment below).
• @Soyaazz for the store release question (question 7).
• @VGalaxies wrote much of SchemaTransactionV2/MetaManager and may know history that invalidates parts of this.
• @contrueCT for the graph-data cache side, which this proposal deliberately defers.
• @vaijosh for whether HBase multi-server deserves the optional mode at all (question 1).

Nothing here is settled. If a simpler mechanism reaches the same bounds, or the bounds themselves are wrong for the k8s release, that is exactly the feedback this thread is for. The full analysis, per-mode matrix, failure semantics and test plan exist as a longer document; the open questions and the implementation plan are in the first comment below.

Implementation decisions and the proposed phased plan are tracked in the first comment. Existing diagrams illustrate the proposal; timing/count annotations are model estimates, not measured production guarantees.

[bitflicker64]

Open questions and proposed implementation plan

Updated 2026-09-18. See the discussion timeline for merged fixes and the separate #3157 watch-recovery PR. The phases below are proposals, not additional scope for #3157 or an already-approved release gate.

Open questions

Numbered so replies can reference them directly ("Q3: 5s"). Q8, Q9 and Q11 block the start of implementation; Q1 gates only the optional HBase phase; Q3 needs an answer before the release; the rest calibrate claims or clean up.

1. Ship an HBase multi-server mode at all? HBase is marked for removal in 2.0 (BackendProviderFactory.java:40). The counter-based register is a small severable PR, but it reserves a HugeType code and adds a counter row to every schema store. Alternative: declare HBase multi-server out of scope and document the legacy RPC notifier as its only story. (@vaijosh, @imbajin)
2. HugeType code reservation for SCHEMA_VERSION, only if Q1 says yes: a code near META(40)/COUNTER(50), plus a one-time sweep of serializers that switch on type codes. Reserve an adjacent code for a possible future graph-data epoch at the same time?
3. Reconcile interval default: proposed 10 s, targeting two successful reconcile intervals after durable version publication. Confirm bounded read/reload latency and durable recovery of a schema commit whose version write fails before promising a deadline after every successful DDL. Acceptable for the k8s release, or should the chart pin 5 s (halves the bound, doubles a still-trivial poll cost)? (@imbajin)
4. PD watch and read semantics: are KvClient get/scan leader-routed, or can a follower serve stale reads? Does the watch replay anything after reconnect, and does it compact rapid puts to one key? Polling avoids reliance on watch delivery, but correctness and timing still depend on version-read visibility and durable publication. These guarantees need explicit confirmation. (@koi2000)
5. Legacy event-key deprecation timeline: new publishers dual-publish the legacy CLEAR key for add/remove/clear bursts. One minor release of dual-publish, or longer? Retiring it also upgrades those bursts from full-clear to per-id granularity. Related: formally deprecate the SOFA-RPC CacheNotifier's cache-clear role in the same cycle? (@imbajin)
6. Out-of-tree consumers of the dead notifyGraphVertexCacheClear/notifyGraphEdgeCacheClear keys: in-repo grep shows zero listeners, but the keys are visible on the wire. Does any external tooling watch them? If provably none, the escape-hatch flag can be dropped and the puts deleted outright.
7. Store release cadence for the SchemaDriver parser fix: deployed store nodes benefit only via a store artifact release. When can one ship, and do store artifacts deploy out of lockstep long-term? (@Soyaazz, @Tsukilc)
8. Delete the task.sync_deletion notification gate unconditionally? Its premise is verifiably false across JVMs, and it also suppresses STORE_CLEAR/TRUNCATE propagation. No config to restore it is proposed. It changes event emission for single-JVM test setups that asserted on notification counts. (@dpol1)
9. Replace the V2 name-miss inline full reload with V1-style point repair in the same PR that ships delta application: per-id deltas make name misses routine, and each miss currently triggers a request-thread full reload. This changes a subtle existing recovery behavior; is anything known to depend on the eager reload? (@dpol1, @Tsukilc)
10. SchemaStatus audit: the timing argument treats status().ok() (CREATED) as the only safe-to-use state, verified for GraphIndexTransaction's index gating. A pass over all SchemaStatus consumers (OLAP gates, Hubble displays) would make the classification airtight. (@imbajin)
11. DDL error and durable-publication contract: today a failed event put after a committed add fails the already-committed DDL call. The design instead returns success, logs, raises metrics and retries in the background, because reporting failure for a committed write misstates reality. It is an observable contract change that clients or tests may depend on. Confirm it is acceptable and how a committed schema change is recovered if the version write fails and the publisher crashes before its background retry. An in-memory dirty flag alone cannot establish the claimed convergence bound. The decided contract must be release-noted. (@imbajin, @dpol1)

Implementation plan

PR-sized phases, each independently landable. The proposal recommends Phases 0 and 1 plus integration validation for the k8s release; maintainers have not yet accepted this as a release gate; Phase 2 is optional and severable.

Phase 0, standalone preparatory fixes (tiny, land first):

• PR-0a: PdMetaDriver.extractValuesFromResponse stops swallowing watch batches containing non-Put events (~30 lines). Check the corresponding Struct helper too; transport recovery in fix(pd): keep KvClient watches alive after reconnect failures #3157 does not change this extraction contract.
• PR-0b: SchemaDriver accepts both the JSON and the legacy plain-string payload, with log-and-skip instead of assert (~25 lines). Fixes bug 1 from the post.
• PR-0c: remove the dead vertex/edge cache-clear publishes behind a default-off flag (~40 lines).

Phase 1, proposed core machinery on the hstore+PD path (release scope pending agreement):

• PR-1: sync package skeleton: register SPI, coalescing flusher with guaranteed final flush, per-graph synchronizer state, new DELTA key codec. Fully unit-testable, no call sites yet (~550 lines).
• PR-2: reconciler singleton with per-graph due-times, graph ownership and lifecycle (create lazily, drain on close, delete the version key on drop) (~400 lines).
• PR-3: wire the V2 transaction: register write plus coalesced events on add/update/remove/clear, delete the task.sync_deletion gate, name-miss point repair, subscribe to both event keys (~280 lines changed). Q8, Q9 and Q11 land here.
• PR-4: consumer hardening: the generation stamp against reads that straddle a clear, single-flight reloads (~285 lines). Should ride the same release train as PR-3.

Phase 2, optional HBase mode (only if Q1 says yes):

• PR-5: counter register with self-echo accounting (~350 lines).

Phase 3, validation and release:

• PR-6: multi-node integration suite on the existing 3 PD + 3 store + 3 server CI env, including a storm-regression test asserted on metrics (event ceiling, resync cadence, zero legacy events for status-only bursts) (~400 lines).
• PR-7: docs, conf templates, release notes covering the upgrade matrix and the Q11 contract change (~120 lines).
• Pre-release, out of repo: a helm-chart validation pass replaying the literal [Bug] cacheEventListener in CachedSchemaTransactionV2 Work does not meet expectations #2617 Hubble scenario across 2+ replicas, a mixed-image rolling upgrade, PD leader chaos, and a 1-hour soak. Two load figures (register-put absorption, scan sizing) are estimates until this pass confirms them.

Phase 4, deferred: graph-data cache events reusing the same machinery, orphan-key janitor, RPC notifier deprecation or repair, legacy-key retirement per Q5.

Next step

No implementation until Q8, Q9 and Q11 have answers and the overall direction has a maintainer nod. Alternative designs that reach the same bounds with less machinery are welcome; the storm numbers in the post are the bar any alternative has to clear.