Preserve quorum errors and expose exact RF3 admission failures

Author: KSemenenko

README.md

@@ -351,12 +351,20 @@ records the actual source, test results and remaining acceptance gates.
 
 The current source stage also adds protected incremental text-index maintenance
 with explicit generation reads, default-off public ANN reads, and bounded
-parallel document/partition queries across two configured RF3 groups. The current
-local solution build and formatter pass; 213 focused native operation scenarios
+parallel document/partition queries across two configured RF3 groups. That stage's
+local solution build and formatter passed; 213 focused native operation scenarios
 pass in both normal and scalar modes, and three FTS process-recovery scenarios
 pass. The native inventory contains 3039 tests. Exact-source Linux RF3 and complete
 acceptance gates remain open; these additions do not increase the accepted task count.
 
+The latest correction preserves exact failure diagnostics for individual replica
+discovery and returns `NoLeader` when the compatible voters cannot form a quorum.
+The current local solution build and formatter pass, and nine complete cohort
+flows pass in both normal and scalar modes. RF3 container rejection now reports
+the first failed admission check while retaining every original predicate. The
+full Linux normal and scalar unit steps for the prior source failed; their
+original reports and current RF3 qualification are still pending.
+
 Complete product functional coverage remains **unmeasured**. The
 [coverage contract](docs/Features/CodeQuality.md) admits whole operation flows,
 excludes load and comparison runs, and requires matching source and build reports.

docs/ADR/ADR-017-ownership-session-tokens.md

@@ -184,3 +184,10 @@ TASK-REP-READINESS-EVIDENCE-001 refines REQ-REP-004 / AC-REP-004 and REQ/AC-TEST
 Each existing actual native ResourceId log subscription records its closed started/running/completed/canceled/faulted state plus observation timestamps of first/last received log lines. These are observer times, not invented native event timestamps; observed faults are rethrown for existing joined disposal. Save snapshots this evidence without replacing its original failure. ResourceId-keyed task lifecycle is retained; no unproved reattachment is introduced. Current24-line buffers and80-line/8192-byte total artifact limits remain unchanged. Evidence groups put resource/readiness/discovery/subscription state first, followed by newest retained logs so prefix clipping cannot omit the latest signal. Existing failure markers remain.
 
 AC-REP-READINESS-EVIDENCE-001 maps to the actual AcEvent008AcceptedRestartCancellationResumesSameReplacementAndPreservesReplay flow: canceled accepted restart saves real diagnostic evidence before recovery; after real all-three SDK/MCP healthy replay, another one-shot observation must contain HTTP200 for all three owned discovered endpoints and actual log observation state. No fake response/source assertion/timer/retry/increased budget. Native execution and exact Linux original evidence remain root-owned and pending.
+
+
+### TASK-MTOKEN-ISOLATION-ADMISSION-EVIDENCE-001
+
+REQ/AC-MTOKEN-ISOLATION-003 retains every original exact native container namespace guard. The original46a4 inspected-container rejection has no retained compound predicate member. Evaluate the existing conditions in their original short-circuit order and retain the first observed mismatch as one closed enum: running state, privilege, PID namespace, network count/mode, capability count/value, exact name/image/user/build source/incarnation. Invalid JSON/required fields and exact GUID parsing still fail; no alias, alternate namespace, permission or admitted-cohort expansion. On an actual predicate rejection emit one bounded fixed-schema stderr row containing only schema/kind/closed mismatch, then throw the unchanged primary rejection. Retain any output failure after that primary through the existing ServerFailureObserver. No native names, addresses, image refs, identifiers, payloads or credentials are emitted.
+
+IntegrationTests ClusterReplication Contracts owns the closed enum; Validation owns native field evaluation; Diagnostics owns the bounded row and original failure order; Serialization reader consumes them at its existing admission boundary. Root joins these together and builds/formats; genuine Kl021ReachableFormerLeaderRejectsMinimumTokenWhileOtherVotersAcknowledgeNewerTerm and its original exact SDK/MCP/isolation/restoration gates remain the runtime evidence. A future observed row identifies only the actual failing predicate, not a consensus cause or successful fault. Original source/namespace guard, shutdown/resource retention, deadlines and all acceptance counts remain unchanged. ADR-017 owns this additive test evidence contract.

docs/ADR/ADR-082-native-cqrs-streams.md

@@ -337,3 +337,10 @@ SCAT binds its existing original/mismatch/corrected actual waves to the same sco
 REQ/AC-CRS-DIAG-007 extends AC-CRS-DIAG-002/005/006 without changing logger, admission, quotas or timing. Original30fb run37694136259 guard400 succeeded but GuardWarningWait reached original12min cancellation with empty rejection artifact; canonical case canceled after2m13s without retained waiting phase. Neither proves provider/parser/consensus defect. Retain fixed-size per-native-resource actual line/prefix/accepted/malformed/oversize counts and observer first/last timestamps; no raw lines/payloads/credentials. Original closed rejection schema remains exact; separate native sidecar only after subscriptions settle under existing16KiB artifact bound. Existing real malformed HTTP→closed warning→healthy SDK/MCP operation verifies accepted native count matches its original rejection.
 
 Canonical flow records closed phase before actual awaits, original SDK task status, token flags and whether an actual marker/receipt was observed. Save immutable pre-cleanup snapshot and terminal status with native file Flush, preserving primary/capture/cleanup failures. No identities/payloads copied; existing marker/ACK/read/replay assertions, deadlines, ownership and cleanup unchanged. This exposes next initiating boundary, not a consensus fix or pass. Integration owns ClusterRouting Helpers/Assertions; root builds/discovers/executes fresh proof. No product hook, alias, format, provider, selector or retry. ADR082 remains Accepted pending actual evidence.
+
+
+### TASK-CRS-COHORT-NO-QUORUM-DETAIL-001
+
+REQ/AC-CRS-002/005 and REQ/AC-SESSIONREAD-003 preserve the existing authenticated fixed-voter cohort and explicit majority. Only the final aggregate compatible-count-below-majority branch in ReplicaCohortAdmission.EnsureCompatibleCohortAsync returns the existing OwnershipLost/NoLeader diagnostic. SDK reads and MCP's pre-tool authenticated read therefore retain the same established no-quorum safe problem. Individual invalid/unavailable discovery, local transport-not-ready, substituted signature, wrong identity and incompatible reachable peer retain their existing strict diagnostics and rejection; no new catch, retry, threshold, deadline or cache authority.
+
+Root owns this single Orleans branch and extends the existing StoppedSocketsRemoveFreshObservationsAndRequireACompatibleMajority native whole-flow to distinguish exact individual InvalidDiscovery from exact aggregate NoLeader, preserve actual stopped sockets/cache eviction/two-voter survival, and re-admit a fresh healthy signed cohort. Existing transition/signature/cancellation/shutdown controls and real KL021 SDK/official MCP no-quorum/body/privacy/restoration flow remain required. Stage order: docs, code and focused native normal/scalar, full build/format, current native case-source binding, delivered exact-source Linux RF3. Original46a4 safe-detail mismatch is retained; source diagnosis does not identify every historical initiating branch. ADR-082 owns cohort admission; ADR-017 owns public session-read authority.

docs/Features/ClusterReplication.md

@@ -458,3 +458,17 @@ Code ownership adds feature-local `ReplicaPrefixReclaimer`, `ReplicaVerifiedPref
 An explicit reclamation call with an already-deleted prefix still verifies the current complete image and joins one native replica WAL rewrite, returning zero without a no-op atomic commit. This settles process interruption between deletion and rewrite. Original cancellation is checked again before that zero-deletion rewrite; once a positive deletion commits, the original rewrite is joined despite later cancellation. The normal maintenance trigger remains actual remaining prefix keys or its unchanged snapshot threshold. No persistent cursor/new recovery format is added.
 
 R2 GC test ownership: each actual native node/materializer operation is observed to terminal completion before every original disposal; all primary and cleanup identities remain retained. Invalid-image restoration observes deletion and original-image rename separately, preserving either failure and attempting both. ReplicaCrashNode construction/disposal joins Log, replica store and canonical store independently. These source corrections do not change GC semantics, fault cuts, resource caps or qualification.
+
+
+### TASK-CRS-COHORT-NO-QUORUM-DETAIL-001
+
+REQ/AC-CRS-002/005 and REQ/AC-SESSIONREAD-003 preserve the existing authenticated fixed-voter cohort and explicit majority. Only the final aggregate compatible-count-below-majority branch in ReplicaCohortAdmission.EnsureCompatibleCohortAsync returns the existing OwnershipLost/NoLeader diagnostic. SDK reads and MCP's pre-tool authenticated read therefore retain the same established no-quorum safe problem. Individual invalid/unavailable discovery, local transport-not-ready, substituted signature, wrong identity and incompatible reachable peer retain their existing strict diagnostics and rejection; no new catch, retry, threshold, deadline or cache authority.
+
+Root owns this single Orleans branch and extends the existing StoppedSocketsRemoveFreshObservationsAndRequireACompatibleMajority native whole-flow to distinguish exact individual InvalidDiscovery from exact aggregate NoLeader, preserve actual stopped sockets/cache eviction/two-voter survival, and re-admit a fresh healthy signed cohort. Existing transition/signature/cancellation/shutdown controls and real KL021 SDK/official MCP no-quorum/body/privacy/restoration flow remain required. Stage order: docs, code and focused native normal/scalar, full build/format, current native case-source binding, delivered exact-source Linux RF3. Original46a4 safe-detail mismatch is retained; source diagnosis does not identify every historical initiating branch. ADR-082 owns cohort admission; ADR-017 owns public session-read authority.
+
+
+### TASK-MTOKEN-ISOLATION-ADMISSION-EVIDENCE-001
+
+REQ/AC-MTOKEN-ISOLATION-003 retains every original exact native container namespace guard. The original46a4 inspected-container rejection has no retained compound predicate member. Evaluate the existing conditions in their original short-circuit order and retain the first observed mismatch as one closed enum: running state, privilege, PID namespace, network count/mode, capability count/value, exact name/image/user/build source/incarnation. Invalid JSON/required fields and exact GUID parsing still fail; no alias, alternate namespace, permission or admitted-cohort expansion. On an actual predicate rejection emit one bounded fixed-schema stderr row containing only schema/kind/closed mismatch, then throw the unchanged primary rejection. Retain any output failure after that primary through the existing ServerFailureObserver. No native names, addresses, image refs, identifiers, payloads or credentials are emitted.
+
+IntegrationTests ClusterReplication Contracts owns the closed enum; Validation owns native field evaluation; Diagnostics owns the bounded row and original failure order; Serialization reader consumes them at its existing admission boundary. Root joins these together and builds/formats; genuine Kl021ReachableFormerLeaderRejectsMinimumTokenWhileOtherVotersAcknowledgeNewerTerm and its original exact SDK/MCP/isolation/restoration gates remain the runtime evidence. A future observed row identifies only the actual failing predicate, not a consensus cause or successful fault. Original source/namespace guard, shutdown/resource retention, deadlines and all acceptance counts remain unchanged. ADR-017 owns this additive test evidence contract.

docs/Features/ClusterRouting/NativeCqrsRequestV2.md

@@ -998,3 +998,10 @@ SCAT binds its existing original/mismatch/corrected actual waves to the same sco
 REQ/AC-CRS-DIAG-007 extends AC-CRS-DIAG-002/005/006 without changing logger, admission, quotas or timing. Original30fb run37694136259 guard400 succeeded but GuardWarningWait reached original12min cancellation with empty rejection artifact; canonical case canceled after2m13s without retained waiting phase. Neither proves provider/parser/consensus defect. Retain fixed-size per-native-resource actual line/prefix/accepted/malformed/oversize counts and observer first/last timestamps; no raw lines/payloads/credentials. Original closed rejection schema remains exact; separate native sidecar only after subscriptions settle under existing16KiB artifact bound. Existing real malformed HTTP→closed warning→healthy SDK/MCP operation verifies accepted native count matches its original rejection.
 
 Canonical flow records closed phase before actual awaits, original SDK task status, token flags and whether an actual marker/receipt was observed. Save immutable pre-cleanup snapshot and terminal status with native file Flush, preserving primary/capture/cleanup failures. No identities/payloads copied; existing marker/ACK/read/replay assertions, deadlines, ownership and cleanup unchanged. This exposes next initiating boundary, not a consensus fix or pass. Integration owns ClusterRouting Helpers/Assertions; root builds/discovers/executes fresh proof. No product hook, alias, format, provider, selector or retry. ADR082 remains Accepted pending actual evidence.
+
+
+### TASK-CRS-COHORT-NO-QUORUM-DETAIL-001
+
+REQ/AC-CRS-002/005 and REQ/AC-SESSIONREAD-003 preserve the existing authenticated fixed-voter cohort and explicit majority. Only the final aggregate compatible-count-below-majority branch in ReplicaCohortAdmission.EnsureCompatibleCohortAsync returns the existing OwnershipLost/NoLeader diagnostic. SDK reads and MCP's pre-tool authenticated read therefore retain the same established no-quorum safe problem. Individual invalid/unavailable discovery, local transport-not-ready, substituted signature, wrong identity and incompatible reachable peer retain their existing strict diagnostics and rejection; no new catch, retry, threshold, deadline or cache authority.
+
+Root owns this single Orleans branch and extends the existing StoppedSocketsRemoveFreshObservationsAndRequireACompatibleMajority native whole-flow to distinguish exact individual InvalidDiscovery from exact aggregate NoLeader, preserve actual stopped sockets/cache eviction/two-voter survival, and re-admit a fresh healthy signed cohort. Existing transition/signature/cancellation/shutdown controls and real KL021 SDK/official MCP no-quorum/body/privacy/restoration flow remain required. Stage order: docs, code and focused native normal/scalar, full build/format, current native case-source binding, delivered exact-source Linux RF3. Original46a4 safe-detail mismatch is retained; source diagnosis does not identify every historical initiating branch. ADR-082 owns cohort admission; ADR-017 owns public session-read authority.

docs/Features/DocumentStorage.md

@@ -454,3 +454,10 @@ Independently literal DocumentResult and OwnedDocumentReadResultV1 comparisons u
 REQ/AC-OWNER-DOC-003/005/006 and REQ/AC-PQUERY-REMOTE-001..005 retain the existing bounded RFC 6901 field-grant/resource-policy contract. Remote native fixture field paths are `/title` and `/secret`; canonical stored JSON property names remain literal `title` and `secret`. Complete public redaction witnesses report `/secret`, matching the actual resource policy. No validation or authorization rule changes.
 
 The existing four RemoteDocumentNativeOwnerTests/RemoteDocumentNativeReadTests operations, five RemotePartitionQueryNativeTests operations, and RemotePartitionParallelCancellationTests operation retain their denial, fresh grant/policy change, native reopen/receipt replay, observed-work cancellation/no partial, complete unchanged images/separate cuts and literal healthy continuation assertions. This correction repairs shared real ConfigureResource seed admission; it adds no case identity or runtime qualification. Root must execute the existing native classes after a fresh build. ADR-100 and ADR-106 remain the owning read-cut/physical-owner boundaries; no boundary change requires a new ADR. Original R648 failures remain retained.
+
+
+### TASK-CRS-COHORT-NO-QUORUM-DETAIL-001
+
+REQ/AC-CRS-002/005 and REQ/AC-SESSIONREAD-003 preserve the existing authenticated fixed-voter cohort and explicit majority. Only the final aggregate compatible-count-below-majority branch in ReplicaCohortAdmission.EnsureCompatibleCohortAsync returns the existing OwnershipLost/NoLeader diagnostic. SDK reads and MCP's pre-tool authenticated read therefore retain the same established no-quorum safe problem. Individual invalid/unavailable discovery, local transport-not-ready, substituted signature, wrong identity and incompatible reachable peer retain their existing strict diagnostics and rejection; no new catch, retry, threshold, deadline or cache authority.
+
+Root owns this single Orleans branch and extends the existing StoppedSocketsRemoveFreshObservationsAndRequireACompatibleMajority native whole-flow to distinguish exact individual InvalidDiscovery from exact aggregate NoLeader, preserve actual stopped sockets/cache eviction/two-voter survival, and re-admit a fresh healthy signed cohort. Existing transition/signature/cancellation/shutdown controls and real KL021 SDK/official MCP no-quorum/body/privacy/restoration flow remain required. Stage order: docs, code and focused native normal/scalar, full build/format, current native case-source binding, delivered exact-source Linux RF3. Original46a4 safe-detail mismatch is retained; source diagnosis does not identify every historical initiating branch. ADR-082 owns cohort admission; ADR-017 owns public session-read authority.