Assert fresh native RF3 readiness in interrupted request flows

Author: KSemenenko

docs/ADR/ADR-082-native-cqrs-streams.md

@@ -377,3 +377,7 @@ REQ/AC-CRS-004/005 and REQ/AC-CRS-DIAG-004 retain the actual current-format thre
 Freeze a private failure-only stderr record at most256 UTF8 bytes, one record per original child, with exact prefix/version, actual phase enum ReadInput/ValidateRequest/OpenStore/ReadOutcome/DisposeStore/WriteReceipt, closed error-kind enum and optional existing KeyLoad ErrorCode or directly nested allowlisted native code1,2,5,9,11,13,16,20,22,24,28,35,40. No exception text/type name, path, identity, credentials, payload or arbitrary field. First initiating failure is frozen before store disposal; later cleanup cannot overwrite it. Original fatal exception priority and initiating plus output/disposal failure retention remain. Native V2 input/success receipt, failure exit2, stdin/receipt/capture bounds, deadlines, original task/lock joins and empty success stderr stay unchanged. Failure output never constitutes a success receipt.
 
 The Integration oracle validates the exact closed record before including that static diagnostic in its failure; it continues to require exit0, exact canonical no-effect/majority outcomes and complete native child settlement. Existing real wrong-node/incarnation, malformed input and corrupt scoped-outcome/locator child flows assert the actual closed failure shape and phase, retain all joins/unchanged bytes, then perform their real healthy child/store continuation. This is mechanism evidence only; current scalar RF3 must rerun to expose its actual cause before any behavioral repair. Root owns review/join/build/native/Linux delivery. Rollback removes only these private observations and their regression amendments; product protocol, persistence, authorization and timebounds are unchanged.
+
+### TASK-CRS-C1-FRESH-READINESS-ORACLE-002
+
+REQ/AC-CRS-005, REQ/AC-CLIENT-004/005 and AC-CRS-FRESH-READINESS-002 retain full interrupted-submit/replay/conflict/healthy effect and original native RF3 receipt contracts. Root freezes the linked NativeCqrsRequestV2 source-backed advisory-cache distinction, joins the existing owned HTTP caller plus128-byte fixture bound and two actual one-shot readiness call sites, builds/formats and delivers for exact Linux SDK/official-MCP flows. The unchanged native readiness endpoint checks authenticated compatible-cohort and catalog admission; no cached status assertion becomes authority. Preserve before/after identity/applied/placement and every literal document/full receipt assertion. No poll, retry, timeout, product/cache/discovery, public data, package or topology change. Original source47 failed artifact remains immutable. Accepted test-oracle implementation contract; runtime qualification pending; rollback removes the coherent three-source amendment only. Root owns compilation/source join/Git; the three parallel feature agents continue independent parent implementation.

docs/Features/ClusterRouting/NativeCqrsRequestV2.md

@@ -1038,3 +1038,11 @@ REQ/AC-CRS-004/005 and REQ/AC-CRS-DIAG-004 retain the actual current-format thre
 Freeze a private failure-only stderr record at most256 UTF8 bytes, one record per original child, with exact prefix/version, actual phase enum ReadInput/ValidateRequest/OpenStore/ReadOutcome/DisposeStore/WriteReceipt, closed error-kind enum and optional existing KeyLoad ErrorCode or directly nested allowlisted native code1,2,5,9,11,13,16,20,22,24,28,35,40. No exception text/type name, path, identity, credentials, payload or arbitrary field. First initiating failure is frozen before store disposal; later cleanup cannot overwrite it. Original fatal exception priority and initiating plus output/disposal failure retention remain. Native V2 input/success receipt, failure exit2, stdin/receipt/capture bounds, deadlines, original task/lock joins and empty success stderr stay unchanged. Failure output never constitutes a success receipt.
 
 The Integration oracle validates the exact closed record before including that static diagnostic in its failure; it continues to require exit0, exact canonical no-effect/majority outcomes and complete native child settlement. Existing real wrong-node/incarnation, malformed input and corrupt scoped-outcome/locator child flows assert the actual closed failure shape and phase, retain all joins/unchanged bytes, then perform their real healthy child/store continuation. This is mechanism evidence only; current scalar RF3 must rerun to expose its actual cause before any behavioral repair. Root owns review/join/build/native/Linux delivery. Rollback removes only these private observations and their regression amendments; product protocol, persistence, authorization and timebounds are unchanged.
+
+### TASK-CRS-C1-FRESH-READINESS-ORACLE-002 (frozen before code)
+
+REQ/AC-CRS-005 and REQ/AC-CLIENT-004/005 require actual healthy routed continuation after interrupted submit, complete same-ID replay/conflict, literal document revision/content and native RF3 receipts. Original source47 run37868406144/attempt1/job113620545601 passed11/12 KL014 cases; SdkSubmitReturnedCancellationReusesCommittedReceipt failed its after.RoutingReady assertion after the healthy commit returned. The source contract makes NodeStatus.RoutingReady a synchronous fresh-observation cache snapshot; actual discovery expires/removes observations before a bounded refresh. This source distinction does not establish the precise original runtime interleaving, and a cached boolean cannot prove current request admission.
+
+AC-CRS-FRESH-READINESS-002 replaces only the two advisory before/after boolean assertions with one actual GET health/ready at each same boundary through the already owned SDK HttpClient. Require HTTP200 and the exact two-field JSON object status=ready/voters=3, bounded to128 body bytes with original response/body disposal. That native endpoint independently executes existing compatible-cohort and catalog admission. Any non200, malformed/extra/missing field, oversized body, cancellation or disposal failure fails the whole case. Do not poll, retry, sleep, extend any deadline, change product/cache/discovery behavior or infer readiness from a prior snapshot. The original before/after incarnation/node/applied and placement comparisons, full immutable receipts, changed-content denial, minimum-token SDK/official-MCP reads, complete literal state and original receipt replay all remain mandatory.
+
+Root owns IntegrationTests ClusterRouting Helpers/RequestCqrsRf3Callers.cs (the original owned native HTTP operation), Contracts/RequestCqrsRf3Protocol.cs (fixture-only128-byte bound), Assertions/RequestCqrsFaultReplayContinuation.cs (two call sites) and the unchanged exact FaultSubmit/Returned SDK/MCP whole cases. Existing ADR082 owns this test-only observation contract; no new public schema, provider, trust boundary, storage or runtime implementation. Freeze, join, native build/format and exact original Linux normal/scalar task cases precede current closure. Preserve original failed artifact11589298518 SHA25635d4bffaeed1db10fcc365af728b60c1515fec33ada040a5f4cdf069ced5b688; rollback removes this coherent oracle amendment, never admits the failed original. Scope remains open until those genuine current-source RF3 flows pass.

docs/implementation/status.json

@@ -226,12 +226,12 @@
     },
     "currentBuild": {
       "command": "dotnet build KeyLoad.slnx --no-restore --configuration Release --verbosity normal --disable-build-servers",
-      "msbuildDebugDirectory": "/private/tmp/keyload-current-r832-kl008-native-unit-image-oracle-full-build-20261009-msbuild-debug",
+      "msbuildDebugDirectory": "/private/tmp/keyload-r835-c1-fresh-native-readiness-full-build-20261009-msbuild-debug",
       "nativeExitCode": 0,
       "buildCompletedSuccessfully": true,
-      "elapsedSeconds": 121.766,
-      "log": "/private/tmp/keyload-current-r832-kl008-native-unit-image-oracle-full-build-20261009.log",
-      "logSha256": "23ff2653090a47c96f151957349404aede7a9e98dda0a529bfa3de3f8a369d58",
+      "elapsedSeconds": 122.536,
+      "log": "/private/tmp/keyload-r835-c1-fresh-native-readiness-full-build-20261009.log",
+      "logSha256": "97a25422f7d4924a003e986ce59637c39525b62ff45c97d5674aeeffbb85f8d1",
       "inputCount": 5337,
       "inputDrift": [],
       "uniqueErrors": [],
@@ -248,8 +248,8 @@
         "minimal"
       ],
       "nativeExitCode": 0,
-      "elapsedSeconds": 99.497,
-      "log": "/private/tmp/keyload-current-r832-final-native-format-20261009.log",
+      "elapsedSeconds": 94.499,
+      "log": "/private/tmp/keyload-r835-c1-fresh-native-readiness-format-20261009.log",
       "logSha256": "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855",
       "inputCount": 5337,
       "inputDrift": [],
@@ -37317,6 +37317,173 @@
         "KL-014 current-source original RoutingReady=false failure must be resolved and receive fresh exact task receipts; prior f80 originals remain immutable",
         "Complete product coverage/endurance/power-loss/performance remain open"
       ]
+    },
+    "stageXXXIIIOriginalDevelopment": {
+      "sourceCheckoutBeforeCommit": "904d234c1c652626057673fd5fa127d924dd25f5",
+      "task": "TASK-CRS-C1-FRESH-READINESS-ORACLE-002",
+      "acceptance": "AC-CRS-FRESH-READINESS-002",
+      "implementation": "Two original whole-flow boundaries now execute one native health/ready call each, with HTTP200, exact status=ready/voters=3, 128-byte admission and original disposal; advisory cache snapshots no longer substitute for admission.",
+      "preserved": "Original unique authenticated SDK/MCP request, complete literal effect and RF3 receipts, retry/conflict, incarnation/node/applied/placement checks, all deadlines, node-local ownership and mandatory suites.",
+      "originalFailedBuild": {
+        "logPath": "/private/tmp/keyload-r834-c1-fresh-native-readiness-full-build-20261009.log",
+        "logSha256": "ae9f5019e05592fd19521da93055c2a9262c8fb2efbb7abd549e66a4cf2cb8bf",
+        "nativeExitCode": 1,
+        "cause": "Two KLD0001 diagnostics for protocol key literals; repaired with named fixture constants before the passing build."
+      },
+      "currentFullBuild": {
+        "command": "dotnet build KeyLoad.slnx --no-restore --configuration Release --verbosity normal --disable-build-servers",
+        "msbuildDebugDirectory": "/private/tmp/keyload-r835-c1-fresh-native-readiness-full-build-20261009-msbuild-debug",
+        "nativeExitCode": 0,
+        "buildCompletedSuccessfully": true,
+        "elapsedSeconds": 122.536,
+        "log": "/private/tmp/keyload-r835-c1-fresh-native-readiness-full-build-20261009.log",
+        "logSha256": "97a25422f7d4924a003e986ce59637c39525b62ff45c97d5674aeeffbb85f8d1",
+        "inputCount": 5337,
+        "inputDrift": [],
+        "uniqueErrors": [],
+        "qualification": "local development; source drift invalidates exact-current-source qualification"
+      },
+      "currentFullFormatter": {
+        "commandArgs": [
+          "dotnet",
+          "format",
+          "KeyLoad.slnx",
+          "--no-restore",
+          "--verify-no-changes",
+          "--verbosity",
+          "minimal"
+        ],
+        "nativeExitCode": 0,
+        "elapsedSeconds": 94.499,
+        "log": "/private/tmp/keyload-r835-c1-fresh-native-readiness-format-20261009.log",
+        "logSha256": "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855",
+        "inputCount": 5337,
+        "inputDrift": [],
+        "qualification": "native local source-bound formatter verification, not functional or Linux proof"
+      },
+      "relatedNativeWholeFlows": {
+        "normal20": {
+          "receiptPath": "/private/tmp/keyload-r835-related-storage-vector-normal20-20261009-receipt.json",
+          "receiptSha256": "9229528514fc0d5ea3ca4a9c1a1af856a6a147618187ac2199bf066477e2afe3",
+          "nativeExitCode": 0,
+          "elapsedSeconds": 6.884,
+          "originalTrxSha256": "95046d0a800527815fb0930e98824920d57882701ab4843d7dd8bdd56730e467",
+          "counts": {
+            "total": "54",
+            "executed": "54",
+            "passed": "54",
+            "failed": "0",
+            "error": "0",
+            "timeout": "0",
+            "aborted": "0",
+            "inconclusive": "0",
+            "passedButRunAborted": "0",
+            "notRunnable": "0",
+            "notExecuted": "0",
+            "disconnected": "0",
+            "warning": "0",
+            "completed": "0",
+            "inProgress": "0",
+            "pending": "0"
+          },
+          "sourceInputs": 5338,
+          "assemblyInputs": 553,
+          "sourceDrift": [],
+          "assemblyDrift": [],
+          "maximumParallelTests": 20,
+          "actualChildResourceUsage": {
+            "userCpuSeconds": 3.884585,
+            "systemCpuSeconds": 1.579283,
+            "childMaximumResidentBytesDarwin": 323862528,
+            "logicalCpuCount": 12,
+            "qualification": "actual operating-system child usage; no RF3 or benchmark inference"
+          },
+          "scope": "actual native 54 whole storage/vector operations; does not execute the changed RF3 readiness helper"
+        },
+        "scalar20": {
+          "receiptPath": "/private/tmp/keyload-r835-related-storage-vector-scalar20-20261009-receipt.json",
+          "receiptSha256": "e6c78e41cc2e8d1ddf03c3fc4f4c579e47477ecb3c4ccc70c567e59b64fc46ce",
+          "nativeExitCode": 0,
+          "elapsedSeconds": 7.909,
+          "originalTrxSha256": "9632ee84ec54cd3eb1c7aa602fc05575c5f594eda3a3818d5e5eac7ea18d009c",
+          "counts": {
+            "total": "54",
+            "executed": "54",
+            "passed": "54",
+            "failed": "0",
+            "error": "0",
+            "timeout": "0",
+            "aborted": "0",
+            "inconclusive": "0",
+            "passedButRunAborted": "0",
+            "notRunnable": "0",
+            "notExecuted": "0",
+            "disconnected": "0",
+            "warning": "0",
+            "completed": "0",
+            "inProgress": "0",
+            "pending": "0"
+          },
+          "sourceInputs": 5338,
+          "assemblyInputs": 553,
+          "sourceDrift": [],
+          "assemblyDrift": [],
+          "maximumParallelTests": 20,
+          "actualChildResourceUsage": {
+            "userCpuSeconds": 6.281072,
+            "systemCpuSeconds": 1.608818,
+            "childMaximumResidentBytesDarwin": 333856768,
+            "logicalCpuCount": 12,
+            "qualification": "actual operating-system child usage; no RF3 or benchmark inference"
+          },
+          "scope": "actual native 54 whole storage/vector operations; does not execute the changed RF3 readiness helper"
+        }
+      },
+      "nativeOverlap": {
+        "scope": "original local 54 whole-operation TRX intervals; no Linux or throughput promotion",
+        "lanes": [
+          {
+            "lane": "normal20",
+            "cases": 54,
+            "peakOverlappingActualNativeCaseIntervals": 17
+          },
+          {
+            "lane": "scalar20",
+            "cases": 54,
+            "peakOverlappingActualNativeCaseIntervals": 18
+          }
+        ]
+      },
+      "sourceManifestOriginals": [
+        {
+          "path": "/private/tmp/keyload-r835-c1-fresh-native-readiness-source-20261009/functional-coverage.production-source-manifest.json",
+          "sha256": "2ab49b8559b665126bd409cf0e08c68f6cd6c15cc059d1a309ccad753548efee",
+          "length": 1368488
+        },
+        {
+          "path": "/private/tmp/keyload-r835-c1-fresh-native-readiness-source-20261009/functional-coverage.test-image.recovery.json",
+          "sha256": "da9107518a312357110864ce8563773faf1adef64030f8c73f1c1292b4d857f1",
+          "length": 97340
+        },
+        {
+          "path": "/private/tmp/keyload-r835-c1-fresh-native-readiness-source-20261009/functional-coverage.test-image.rf3.json",
+          "sha256": "bef02f684df32679cfa3900057fe855470cf189b3a5e7ad96f5c3f6bd08fa15b",
+          "length": 352775
+        },
+        {
+          "path": "/private/tmp/keyload-r835-c1-fresh-native-readiness-source-20261009/functional-coverage.test-image.unit.json",
+          "sha256": "d26f11fad209c4e2b8b7c90e429cc5e948fad30261869102ea9c37bc95136c3d",
+          "length": 1080536
+        }
+      ],
+      "sourcePrepareExitCode": 0,
+      "sourceVerifyExitCode": 0,
+      "fullProductCoverage": null,
+      "qualification": "local source-bound development only; exact current Linux whole RF3 readiness flows pending",
+      "remaining": [
+        "Fresh exact-source Linux normal/scalar KL014 whole SDK/official-MCP RF3 cases",
+        "Complete current unit/recovery/RF3, functional coverage, endurance, power-loss and representative performance gates"
+      ]
     }
   },
   "TIME-PROVIDER-001": {
@@ -40250,7 +40417,15 @@
           "receiptPath": "/private/tmp/keyload-47-kl014-normal-original-20261009/authenticity-and-original-failures.json",
           "receiptSha256": "6313568de6d5992ddf553bfd2f9704600b7b7827bdab580921537ff2cab153d4"
         },
-        "remaining": "Investigate native RoutingReady false observation after healthy committed continuation; fresh exact-source normal/scalar flows required."
+        "remaining": "Fresh exact-source normal/scalar Linux whole RF3 cases must validate the one-shot native readiness observation; original source47 failed artifact and historical f80 acceptance remain immutable.",
+        "currentRepair": {
+          "task": "TASK-CRS-C1-FRESH-READINESS-ORACLE-002",
+          "acceptance": "AC-CRS-FRESH-READINESS-002",
+          "developmentStage": "TEST-PIPELINE-001.stageXXXIIIOriginalDevelopment",
+          "localBuildPassed": true,
+          "localFormatterPassed": true,
+          "currentLinuxRf3Qualified": false
+        }
       }
     }
   },

tests/KeyLoad.IntegrationTests/Features/ClusterRouting/Assertions/RequestCqrsFaultReplayContinuation.cs

@@ -50,6 +50,7 @@ private static async Task HealthyContinuationAsync(RequestCqrsRf3Callers callers
         await Assert.That(placement.Partition).IsEqualTo(reference.Partition);
         var before = await McpCallerAssertions.SdkSuccessAsync(await administrator.StatusAsync(cancellationToken)
             .ConfigureAwait(false)).ConfigureAwait(false);
+        await callers.AssertFreshRoutingReadyAsync(cancellationToken).ConfigureAwait(false);
         await Assert.That(before.Incarnation).IsEqualTo(placement.Incarnation);
         await Assert.That(originalReceipt.Token.Incarnation).IsEqualTo(placement.Incarnation);
         await Assert.That(originalReceipt.Token.AtomicPartitionId).IsEqualTo(reference.Partition.AtomicPartitionId);
@@ -69,10 +70,9 @@ await Assert.That(JsonDefaults.Serialize(healthyReceipt.Mutations[MutationIndex]
             .SequenceEqual(JsonDefaults.Serialize(mutation))).IsTrue();
         var after = await McpCallerAssertions.SdkSuccessAsync(await administrator.StatusAsync(cancellationToken)
             .ConfigureAwait(false)).ConfigureAwait(false);
+        await callers.AssertFreshRoutingReadyAsync(cancellationToken).ConfigureAwait(false);
         await Assert.That(after.Incarnation).IsEqualTo(placement.Incarnation);
         await Assert.That(after.NodeId).IsEqualTo(before.NodeId);
-        await Assert.That(before.RoutingReady).IsTrue();
-        await Assert.That(after.RoutingReady).IsTrue();
         var currentPlacement = await McpCallerAssertions.SdkSuccessAsync(await administrator.ReadAtomicPartitionPlacementAsync(
             new(PlacementVersion, reference.Partition), cancellationToken).ConfigureAwait(false)).ConfigureAwait(false);
         await Assert.That(JsonDefaults.Serialize(currentPlacement).SequenceEqual(JsonDefaults.Serialize(placement))).IsTrue();

tests/KeyLoad.IntegrationTests/Features/ClusterRouting/Contracts/RequestCqrsRf3Protocol.cs

@@ -18,6 +18,11 @@ internal static class RequestCqrsRf3Protocol
     internal const string FollowerLossScenario = "cluster-routing-c1-compatible-follower-loss";
     internal const string ReadyPath = "health/ready";
     internal static readonly Uri ReadyUri = new(ReadyPath, UriKind.Relative);
+    internal const int ReadyReplyMaximumBytes = 128;
+    internal const int ReadyReplyPropertyCount = 2;
+    internal const string ReadyReplyStatusProperty = "status";
+    internal const string ReadyReplyVotersProperty = "voters";
+    internal const string ReadyReplyExpectedStatus = "ready";
     internal const int NodeCount = 3;
     internal const int CurrentMajority = 2;
     internal const int ApplicationProtocolVersion = 4;