Join native query cancellation proof and require complete-flow coverage

Author: KSemenenko

AGENTS.md

@@ -162,6 +162,7 @@ Rule format:
 ## Global Skills
 The explicit owner instruction to enable Orleans distributed directory and activation repartitioning is consent to those two native experimental APIs. Confine compiler opt-in ORLEANSEXP003/ORLEANSEXP001 to their two configuration calls, with ADR-034 evidence; it does not authorize global NoWarn, suppression of quality diagnostics or changing analyzer severity.
 
+- On 2026-10-05 the owner explicitly authorized installing the Managed Code `quality` bundle from https://skills.managed-code.com/bundles/quality/ and running KeyLoad complexity, CRAP and related code-quality analysis. Use the catalog's current bundle skill names and native installation command; this scoped permission supersedes the historical skill-installation prohibition and older `mcaf-*` naming requirement for this bundle only. Preserve existing quality thresholds and use the Aspire-owned entry point for coverage-producing tests.
 - On 2026-10-01 the owner explicitly authorized installing the Orleans skill through the `dotnet skills` command. This rule-specific permission applies to the requested Orleans skill; other skill installation remains prohibited unless separately authorized. Read and apply its installed SKILL.md before continuing Orleans implementation.
 - Orleans 3.1.1 is installed globally at `/Users/ksemenenko/.codex/skills/orleans/SKILL.md`, from Managed Code catalog `2026.10.1.0` using `dotnet-skills` 0.1.242. It is applied to Orleans design, lifecycle, transport, routing and failure verification. This is the specifically authorized addition; the historical bootstrap installed no skills.
 - No MCAF or .NET skills are installed for this bootstrap. The owner explicitly instructed: do not install skills. Do not create skill directories, install tools or modify global agent configuration for this task.
@@ -182,7 +183,7 @@ The explicit owner instruction to enable Orleans distributed directory and activ
 - `format`: `dotnet format KeyLoad.slnx --verify-no-changes --no-restore` is the required formatter command and CI gate; source configuration does not establish a green formatter qualification.
 - `analyze`: solution Release build with TreatWarningsAsErrors=true, AnalysisLevel=latest-all, SDK/style analysis and centrally attached KeyLoad.Analyzers; compiler SARIF reports are retained under artifacts/code-quality.
 - `complexity`: numeric policy limits below are mandatory and are enforced by source-owned KLD0030/KLD0031/KLD0032/KLD0033 during ordinary consumer builds; the analyzer infrastructure has a real compiler source-inventory fixture in CI. Configuration or a scoped build MUST NOT be reported as a passing full gate without the exact-SHA complete build and fixture qualification.
-- `coverage`: no CI coverage collector or numeric baseline is configured yet. Configure an MTP/TUnit-compatible collector before claiming the coverage thresholds below pass; no invented coverage result is allowed.
+- `coverage`: owner direction 2026-10-05 requires native MTP/TUnit-compatible code-coverage collection through the Aspire-owned test entry point, retained original reports, and module/file/line/branch gap analysis. Only functional operation tests may contribute; load, stress, performance and database-comparison runs MUST NOT contribute to coverage totals. Until a complete source-bound report exists, report coverage as unmeasured rather than inventing a numeric baseline or claiming the thresholds below pass.
 - `governance`: `node scripts/Features/RepositoryGovernance/verify.mjs` validates the real inventory and policy-preservation record; it is static installation validation, not a runtime test result.
 - Active runner: Microsoft.Testing.Platform in global.json; mandatory framework: TUnit. Existing xUnit/VSTest references are migration debt, not a permitted second framework.
 - .NET target is net10.0. C# 14.0 is explicitly pinned in Directory.Build.props. Root .editorconfig is the source of truth for formatting/style/analyzer severity; nested files require a concrete subtree purpose.
@@ -344,7 +345,7 @@ Local `AGENTS.md` files may tighten these values, but they must not loosen them
 - Each `AC-*` criterion in the owning feature specification MUST be covered by one or more automated tests or by an explicit written exception.
 - Test names, display names, or comments should reference the relevant `AC-*` ID when that improves traceability.
 - Every behaviour change needs new or updated automated tests with meaningful assertions. New tests are mandatory for new behaviour and bug fixes.
-- Tests must prove the real user flow or caller-visible system flow, not only internal implementation details.
+- Owner correction 2026-10-05 requires every test to contain a complete real operation or caller-visible workflow: arrange actual preconditions, execute the operation through its real contract, and verify its outcome plus the resulting state or fields changed during that operation. Getter/setter checks, property-only assertions and implementation-mirroring tests do not satisfy this rule or acceptance; a negative flow must execute the rejected operation and verify its error and preserved state.
 - Tests should be as realistic as possible and exercise the system through real flows, contracts, and dependencies.
 - Tests must cover positive flows, negative flows, edge cases, and unexpected paths from multiple relevant angles when the behaviour can fail in different ways.
 - Prefer integration/API/UI tests over isolated unit tests when behaviour crosses boundaries.
@@ -356,6 +357,7 @@ Local `AGENTS.md` files may tighten these values, but they must not loosen them
 - Critical flows and public contracts MUST reach at least 90% line coverage with explicit success and failure assertions.
 - Repository or module coverage must not decrease without an explicit written exception. Coverage after the change must stay at least at the previous baseline or improve.
 - Coverage is for finding gaps, not gaming a number. Coverage numbers do not replace scenario coverage or user-flow verification.
+- Owner direction 2026-10-05 requires reviewing actual uncovered code when closing each module and adding meaningful complete-flow regressions for the missing success, failure and edge behaviours. Keep load, stress, performance and comparison tests out of the coverage evidence, preserve an explicit contributor inventory, and never add trivial tests solely to raise a percentage.
 - The task is not done until the full relevant test suite is green, not only the newly added tests.
 - If the stack is `.NET`, document the active framework and runner model explicitly so agents do not mix VSTest and Microsoft.Testing.Platform assumptions.
 - If the stack is `.NET`, after changing production code run the repo-defined quality pass: format, build, analyze, focused tests, broader tests, complexity, coverage, and any configured extra gates such as architecture, security, or mutation checks.

docs/ADR/ADR-103-scaled-fair-comparisons.md

@@ -110,3 +110,8 @@ cleanup failures, and run them through the canonical Aspire comparison entry.
 ComparisonTests owns this code, partition_pages owns its private guarded packet,
 and root owns integration/gates/evidence/commit. There is no data or wire migration;
 rollback cannot convert an unfinished original task into a passing qualification.
+The stage also replaces the collector's premature completed boolean with one
+memoized original completion task, preserves cancellation-callback failures
+while joining its original observation, and retains failed write settlement on
+repeated teardown. Its schema, byte/time bounds and unqualified categories stay
+unchanged. This lifecycle amendment precedes the private repair.

docs/Features/BenchmarkComparisons/ScalingQualification.md

@@ -113,6 +113,12 @@ request cancellation or escalation through the original owner's supported API
 where available, retain the threshold failure, and ultimately await that same
 original task. Do not start a replacement operation, use an uncancellable shadow
 task, or dispose an owner while its observation/writer still uses it.
+The resource collector memoizes one original completion task. Concurrent or
+repeated completion calls await that same task, including its failure; a boolean
+set before cancellation/join is not completed settlement. Retain any native stop
+cancellation-callback failure and still join the original observation before
+disposing the CTS or writing/copying evidence. A failed evidence write remains
+the same failed completion on a later teardown call.
 
 AC-SCALE-017 requires genuine native lifecycle regressions to prove that cleanup
 does not pass an unfinished original operation, and that cancellation and failure

docs/implementation/query-cancellation-development-2026-10-05.json

@@ -0,0 +1,138 @@
+{
+  "schemaVersion": 1,
+  "date": "2026-10-05",
+  "kind": "local-development",
+  "baseRevision": "88ce19ebd5e4405e2ca102a4ea5f2e1c61e46ed3",
+  "requirement": "REQ-DQUERY-PREREQ-005",
+  "acceptance": "AC-DQUERY-PREREQ-005",
+  "task": "TASK-DQUERY-CANCEL-OBSERVATION",
+  "scope": "Native synchronous partition-query cancellation observation and related functional regressions",
+  "source": [
+    {
+      "path": "tests/KeyLoad.UnitTests/Features/QueryExecution/Cases/PartitionQueryCancellationTests.cs",
+      "sha256": "0cfcbd1c35211eb19d923611dc1fa4e53b9e0928309d2ebf9fd88385b83e81ee"
+    },
+    {
+      "path": "tests/KeyLoad.UnitTests/Features/QueryExecution/Helpers/PartitionQueryCancellationFailures.cs",
+      "sha256": "36c057734c08ae373d7b58c86339cda16edcf5b21433ffb9b2076481cdf9649a"
+    },
+    {
+      "path": "tests/KeyLoad.UnitTests/Features/QueryExecution/Helpers/PartitionQueryCancellationObserver.cs",
+      "sha256": "cd1a9729069e70431ff743bf1d4246578989c9f93d2ce0a3ce98e2a9308d0193"
+    },
+    {
+      "path": "tests/KeyLoad.UnitTests/Features/QueryExecution/Helpers/PartitionQueryCancellationRun.cs",
+      "sha256": "8577045df2ba053e105f5a2894ae48cd10ba640e9c0301273fa734d9a5de527f"
+    }
+  ],
+  "preserved": {
+    "nativeZoneTreeRecords": 5000,
+    "operationDeadlineSeconds": 30,
+    "originalRequestAndCallerCancellationToken": true,
+    "exactCancellationTokenAsserted": true,
+    "noPartialResultAsserted": true,
+    "healthyFollowUpAsserted": true,
+    "originalObserverJoinedBeforeTokenEventAndStorageDisposal": true
+  },
+  "build": {
+    "command": "dotnet build KeyLoad.slnx --no-restore --configuration Release",
+    "exitCode": 0,
+    "warnings": 0,
+    "errors": 0,
+    "log": "/private/tmp/keyload-stage88b-query-cancellation-build-20261005.log",
+    "sha256": "9ba350ada494338827611cc0444cc68d718b4f25f2613f7d2a79e04d8030a350"
+  },
+  "initialBuildFailure": {
+    "errors": 8,
+    "warnings": 0,
+    "cause": "Six CA1031 catch diagnostics and two IDE0005 imports were repaired without suppression",
+    "log": "/private/tmp/keyload-stage88a-query-cancellation-build-20261005.log",
+    "logPresent": false
+  },
+  "formatter": {
+    "command": "dotnet format KeyLoad.slnx --verify-no-changes --no-restore",
+    "exitCode": 0,
+    "log": "/private/tmp/keyload-stage88-query-cancellation-format-20261005.log"
+  },
+  "governance": {
+    "command": "node scripts/Features/RepositoryGovernance/verify.mjs",
+    "exitCode": 0,
+    "actualInventory": "27 projects and 4 modules"
+  },
+  "nativeRuns": [
+    {
+      "suite": "unit",
+      "entryPoint": "Aspire-owned KeyLoad.AppHost",
+      "filter": "/*/*/PartitionQuery*/*",
+      "summary": {
+        "total": 25,
+        "passed": 25,
+        "failed": 0,
+        "skipped": 0,
+        "cancelled": 0,
+        "timedOut": 0,
+        "flaky": 0
+      },
+      "artifacts": [
+        {
+          "path": "TestResults/stage88c-partition-query/KeyLoad.UnitTests-macos-net10.0-report.html",
+          "sha256": "8586c40ac8679e708b4ae7e0d3b907b22198a9d342628a3b4a82abf2c8ff603e"
+        },
+        {
+          "path": "TestResults/stage88c-partition-query/KeyLoad.UnitTests-macos-net10.0.tunit-report.json",
+          "sha256": "4fd32978b0d9a18cdf2c15a8948c2bd22a34cf206f53baa3e41e165028eef651"
+        },
+        {
+          "path": "TestResults/stage88c-partition-query/KeyLoad.UnitTests_net10.0_arm64.trx",
+          "sha256": "baf6b0ce1e54d4e9c8f9341aa68082396d1a4d752783bb45322acd319cbba2c8"
+        },
+        {
+          "path": "TestResults/stage88c-partition-query/_konstantins-macbook-pro_2026-10-05_20_30_56.9942060/In/konstantins-macbook-pro/KeyLoad.UnitTests-macos-net10.0-report.html",
+          "sha256": "8586c40ac8679e708b4ae7e0d3b907b22198a9d342628a3b4a82abf2c8ff603e"
+        }
+      ],
+      "qualification": "local-development-only"
+    },
+    {
+      "suite": "unit-scalar",
+      "entryPoint": "Aspire-owned KeyLoad.AppHost",
+      "filter": "/*/*/PartitionQuery*/*",
+      "summary": {
+        "total": 25,
+        "passed": 25,
+        "failed": 0,
+        "skipped": 0,
+        "cancelled": 0,
+        "timedOut": 0,
+        "flaky": 0
+      },
+      "artifacts": [
+        {
+          "path": "TestResults/stage88d-partition-query-scalar/KeyLoad.UnitTests-macos-net10.0-report.html",
+          "sha256": "0c007716c69268da77faf5ff723128e5027e019d2d8ad16538853bcb2404175b"
+        },
+        {
+          "path": "TestResults/stage88d-partition-query-scalar/KeyLoad.UnitTests-macos-net10.0.tunit-report.json",
+          "sha256": "f26f1a9938fbebaae4f1f2ecb0cefbcdc142aecabab1b3591faafa81a1f2244a"
+        },
+        {
+          "path": "TestResults/stage88d-partition-query-scalar/KeyLoad.UnitTests_net10.0_arm64.trx",
+          "sha256": "6740fca30a0e89ceb19f199cb8b79eb95ca0eb031b35ef4f1506dd77bfff1b2d"
+        },
+        {
+          "path": "TestResults/stage88d-partition-query-scalar/_konstantins-macbook-pro_2026-10-05_20_36_00.8989700/In/konstantins-macbook-pro/KeyLoad.UnitTests-macos-net10.0-report.html",
+          "sha256": "0c007716c69268da77faf5ff723128e5027e019d2d8ad16538853bcb2404175b"
+        }
+      ],
+      "qualification": "local-development-only"
+    }
+  ],
+  "fullModuleAcceptanceClosed": false,
+  "deliveredLinuxQualified": false,
+  "remaining": [
+    "Full required unit/scalar/recovery/RF3 and exact-source Linux qualification remain required",
+    "Remote physical-owner fanout for KL-037 remains unimplemented",
+    "Functional coverage collection and module/file/line/branch gap review are separately required; these runs did not collect coverage"
+  ],
+  "priorFailures": "Original full-suite failures remain retained in partition-runtime-development-2026-10-05.json; focused passes do not replace them"
+}

docs/implementation/status.json

@@ -1433,14 +1433,25 @@
       "src/KeyLoad.Query/Features/QueryExecution/Queries/QueryEngine.PartitionQuery.cs",
       "src/KeyLoad.Client/Features/QueryExecution/Transport/PartitionQueryClient.cs",
       "tests/KeyLoad.UnitTests/Features/QueryExecution/Cases/PartitionQueryPublicContractTests.cs",
-      "tests/KeyLoad.IntegrationTests/Features/QueryExecution/Cases/PartitionQueryPublicRf3Tests.cs"
+      "tests/KeyLoad.IntegrationTests/Features/QueryExecution/Cases/PartitionQueryPublicRf3Tests.cs",
+      "docs/implementation/query-cancellation-development-2026-10-05.json"
     ],
     "sourceStage": "bounded_same_physical_owner_partition_query_public_sdk_http_mcp_and_native_orleans_read_path",
     "qualification": {
       "implementation": "source joined; local Release solution build has zero warnings/errors; focused Aspire unit validation passed 25/25",
       "acceptance": "open",
       "remotePhysicalOwnerFanout": "not implemented",
       "fullTaskComplete": false
+    },
+    "queryCancellationDevelopment": {
+      "requirement": "REQ-DQUERY-PREREQ-005",
+      "acceptance": "AC-DQUERY-PREREQ-005",
+      "receipt": "docs/implementation/query-cancellation-development-2026-10-05.json",
+      "nativeNormal": "25_of25",
+      "nativeScalar": "25_of25",
+      "releaseBuild": "0_warnings_0_errors",
+      "fullModuleAcceptanceClosed": false,
+      "deliveredLinuxQualified": false
     }
   },
   "KL-038": {

tests/KeyLoad.UnitTests/Features/QueryExecution/Cases/PartitionQueryCancellationTests.cs

@@ -1,14 +1,11 @@
-using System.Diagnostics;
 using KeyLoad.Core;
 using KeyLoad.Query;
-using KeyLoad.Query.Features.QueryExecution;
 
 namespace KeyLoad.UnitTests.Features.QueryExecution;
 
 internal sealed class PartitionQueryCancellationTests
 {
     private const int SeedCount = 5_000;
-    private static readonly TimeSpan ObservationBound = TimeSpan.FromSeconds(10);
 
     [Test]
     public async Task ObservedNativeReadProgressCancelsTheOriginalBudgetAndLeavesNoPartialResult()
@@ -18,22 +15,12 @@ public async Task ObservedNativeReadProgressCancelsTheOriginalBudgetAndLeavesNoP
         using var cancellation = new CancellationTokenSource();
         var budget = new ReadExecutionBudget(database.Database.Limits, cancellationToken: cancellation.Token);
         var engine = new QueryEngine(database.Database);
-        var operation = Task.Run(() => engine.ExecutePartitionQuery(PartitionQueryTestSupport.Principal,
-            PartitionQueryTestSupport.Request(database, 1), [database.Partition], budget));
-        var observedBytes = await WaitForReadProgressAsync(budget, operation);
-        await cancellation.CancelAsync();
-        OperationCanceledException? failure = null;
-        try
-        {
-            _ = await operation;
-        }
-        catch (OperationCanceledException error)
-        {
-            failure = error;
-        }
+        var outcome = PartitionQueryCancellationRun.Execute(database, engine, cancellation, budget);
 
-        await Assert.That(observedBytes).IsGreaterThan(0L);
-        await Assert.That(failure?.CancellationToken).IsEqualTo(cancellation.Token);
+        await Assert.That(outcome.QueryReturned).IsFalse();
+        await Assert.That(outcome.Cancellation?.CancellationToken).IsEqualTo(cancellation.Token);
+        await Assert.That(outcome.CancellationRequested).IsTrue();
+        await Assert.That(outcome.ObservedReadBytes).IsGreaterThan(0L);
         var healthy = engine.ExecutePartitionQuery(PartitionQueryTestSupport.Principal,
             PartitionQueryTestSupport.Request(database, 1), [database.Partition]);
         await Assert.That(healthy.Complete).IsTrue();
@@ -52,16 +39,4 @@ private static void SeedRows(TestDatabase database)
             PartitionQueryTestSupport.AddRows(database, database.Partition, rows);
         }
     }
-
-    private static async Task<long> WaitForReadProgressAsync(ReadExecutionBudget budget,
-        Task<PartitionQueryResultV1> operation)
-    {
-        var started = Stopwatch.GetTimestamp();
-        while (budget.ReadBytes == 0 && !operation.IsCompleted
-               && Stopwatch.GetElapsedTime(started) < ObservationBound)
-        {
-            await Task.Yield();
-        }
-        return budget.ReadBytes;
-    }
 }

tests/KeyLoad.UnitTests/Features/QueryExecution/Helpers/PartitionQueryCancellationFailures.cs

@@ -0,0 +1,44 @@
+using System.Runtime.ExceptionServices;
+using ManagedCode.Communication.CQRS;
+
+namespace KeyLoad.UnitTests.Features.QueryExecution;
+
+internal static class PartitionQueryCancellationFailures
+{
+    private const string FailureMessage = "The partition query and native cancellation observer did not settle cleanly.";
+
+    internal static void ThrowIfAny(Exception? primary, List<Exception> settlement)
+    {
+        var failures = new List<Exception>();
+        if (primary is not null)
+        {
+            failures.Add(primary);
+        }
+        foreach (var failure in settlement)
+        {
+            if (!failures.Contains(failure, ReferenceEqualityComparer.Instance))
+            {
+                failures.Add(failure);
+            }
+        }
+        ThrowWithFatalPriority(failures);
+    }
+
+    private static void ThrowWithFatalPriority(List<Exception> failures)
+    {
+        if (failures.Count == 0)
+        {
+            return;
+        }
+        var fatalIndex = failures.FindIndex(failure => CqrsRuntimeFailures.FindFatal(failure) is not null);
+        if (failures.Count == 1)
+        {
+            ExceptionDispatchInfo.Capture(failures[0]).Throw();
+        }
+        if (fatalIndex > 0)
+        {
+            (failures[0], failures[fatalIndex]) = (failures[fatalIndex], failures[0]);
+        }
+        throw new AggregateException(FailureMessage, failures);
+    }
+}