Repair crash-file readiness and retain native restart failure evidence

Preserve all current authorized source changes; qualification runs in GitHub Actions.

Author: KSemenenko

README.md

@@ -2,7 +2,7 @@
 
 KeyLoad is an experimental .NET 10 database that puts JSON documents, event streams, durable work queues, graphs, time series and search behind one API and one authorization model. An atomic command can update a document, append its event and enqueue work in the same transaction domain. The first server topology has three replicated nodes; each node owns its ZoneTree storage and journals.
 
-The intended cluster foundation is Orleans, with separate request grains, a distributed grain directory and activation migration. That migration, fully containerized RF3 execution and the official MCP surface are in progress; the published comparison baseline predates them. Follow the [architecture map](docs/Architecture.md) and [qualification tracker](docs/implementation/status.json) for the actual implemented and verified boundaries.
+The cluster foundation is Orleans, with separate request grains and a distributed grain directory. Docker/Aspire RF3 operations are exercised through the real .NET and official MCP SDK clients in GitHub Actions. Actual activation migration remains unqualified, and the published comparison baseline predates this replacement. Follow the [architecture map](docs/Architecture.md) and [qualification tracker](docs/implementation/status.json) for the implemented and verified boundaries.
 
 The original load-testing prototype has been replaced. This repository implements the new [architecture and development plan](docs/design/architecture-v0.3.uk.md), with the [original HTML edition](docs/design/architecture-v0.3.uk.html) preserved alongside it.
 
@@ -237,7 +237,7 @@ Tests use TUnit and Microsoft.Testing.Platform and execute in GitHub Actions. De
 
 The root `.editorconfig` is copied directly from Prostir. Every project enables SDK/static/style analysis with warnings as errors. Edit custom rules under `src/KeyLoad.Analyzers/Features/CodeQuality/` and add real-compilation cases under `tests/KeyLoad.Analyzers.Tests/Features/CodeQuality/`. They attach centrally to consumer projects and report in the IDE and build. Compiler SARIF 2.1 reports live under `artifacts/code-quality/<project>/<configuration>/<framework>/diagnostics.sarif`; CI retains them even when the build fails. See [CodeQuality](docs/Features/CodeQuality.md) for the rule catalog, authoring and applicability, and [current evidence](docs/implementation/code-quality.md) for actual gates.
 
-The latest completed runtime checkpoint [fa80c701475d8faf65bb43b49b6e1c7afb33979a](https://github.com/managedcode/KeyLoad/commit/fa80c701475d8faf65bb43b49b6e1c7afb33979a) passed exact GitHub solution build, formatter, governance and118/118 analyzer cases on all three OSes in [run37021991878](https://github.com/managedcode/KeyLoad/actions/runs/37021991878). Unit results are787/788 on each OS, with three distinct remaining failures. Process recovery is115/115 on Linux/macOS and104/115 on Windows. RF3 caller tests are9/26; three separate file-receipt tests also pass. Comparison is2/4. The previous four unit failures now pass across all OSes. New preserving fixture/MCP framing repairs require renewed CI; [the runtime ledger](docs/implementation/runtime-qualification-20261002.md) records exact failures and evidence. Coverage and server-resource qualification remain open.
+The [runtime ledger](docs/implementation/runtime-qualification-20261002.md) preserves exact GitHub source SHAs, native suite counts, failures and artifact hashes; [AdminDashboard](docs/Features/AdminDashboard.md) records its separately verified RF3 and browser evidence. Preserving recovery refinements cover optional checkpoint metadata absence and all expected source/target ownership under the original deadline. They and the new bounded failed-restart diagnostics require their own delivered-source CI. Recovery, comparison completion and intermittent startup gates remain open. Coverage and server-resource qualification remain open.
 
 The comparison library/sole CLI host split under [ADR-043](docs/ADR/ADR-043-comparison-library-host.md), immutable harness under [ADR-044](docs/ADR/ADR-044-benchmark-immutable-contracts.md), owned PostgreSQL schema repair under [ADR-045](docs/ADR/ADR-045-postgres-schema-ownership.md), private storage owners under [ADR-046](docs/ADR/ADR-046-storage-private-owners.md) and genuine BenchmarkDotNet library under [ADR-047](docs/ADR/ADR-047-embedded-benchmark-host.md) are source joins awaiting full runtime qualification. The read-only product contract migration preserves JSON/base64 and fingerprints under [ADR-041](docs/ADR/ADR-041-read-only-public-collections.md). Exact-SHA run [37015193756](https://github.com/managedcode/KeyLoad/actions/runs/37015193756) passed all88 analyzer cases and the enabled solution builds; subsequent repairs require renewed verification. Compatible coverage collection, container export and its numeric baseline remain pending. Historical source stages are retained in the [code-quality record](docs/implementation/code-quality.md); the [runtime ledger](docs/implementation/runtime-qualification-20261002.md) records the latest completed candidate run.
 

docs/Features/ClusterReplication.md

@@ -24,6 +24,51 @@ Slice map: `src/KeyLoad.Replication/Features/ClusterReplication/` owns protocol/
 
 Traceability: AC-REP-001/003/004 map to `ClusterTests` and the new replica recovery tests; AC-REP-002 maps to real CrashHost interruption scenarios; AC-REP-005 maps to authorization/failover client cases. TASK-REP-LOG, TASK-REP-TRANSPORT, TASK-REP-INTEGRATE and TASK-REP-VERIFY are defined in [execution plan](../implementation/orleans-foundation.plan.md). Qualification is GitHub Actions only. Historical CI 36926803549 qualifies the old implementation, not this replacement. Power-loss and endurance remain pending.
 
+TASK-RUNTIME-REPLICA-READINESS-W4 maps REQ-REP-004 / AC-REP-004 and
+REQ/AC-STORAGE-012. Exact main b533c80 / CI37032546228 fails the Windows
+SnapshotInstalled reopen on target/database/tree/0.meta.wal; the prior two-store
+barrier probes only owner.lock and commands.wal. Root replaces each store's
+probe with the shared StorageRecovery exclusive-file probe, including metadata
+when present, inside the existing single five-second/25ms loop. Cancellation
+stops before another probe. No file creation, recovery retry, timeout increase,
+weakened snapshot/tail/receipt predicate or attribution of the unknown holder
+is permitted. A disjoint test worker owns new ClusterReplication real-file
+barrier cases: hold canonical or replica metadata exclusively, verify pending
+then release and success; cancel a held wait and pre-cancel an unlocked wait;
+assert a permanent holder still fails under the unchanged bound. Use actual
+closed CrashHost target stores and observe/dispose pending tasks and holders
+before deleting their root. Existing native SnapshotInstalled process-kill and
+ordered-tail tests are the primary regression. ADR-035/036/041 ownership and
+lifetime contracts suffice; public/data/dependency boundaries are unchanged.
+Root reviews both helper/caller scopes together, development-builds/formats,
+and qualifies the full three-OS recovery suite in GitHub at the delivered SHA.
+
+The same W4 criterion includes exact1bee2609 / CI37036628601's source-store
+reopen failures. For the six typed snapshot/transfer crash boundaries, the child
+also opened `source/database` and `source/replica`. Enumerate these exact owned
+stores alongside the target stores inside the same single five-second/25ms loop.
+Use the crash boundary's explicit source-ownership contract, never filesystem
+existence to decide whether required ownership/journal files are optional. Other
+boundaries retain their target-only path and must not create source storage.
+Root owns the CrashHost store-path/source-boundary helpers, preserving the
+existing scenario predicate, and ReplicaProcessTrial/ReplicaProcessFiles join.
+The disjoint readiness test worker extends only its real-store fixture and cases:
+each source canonical/replica owner, journal and metadata holder keeps the actual
+combined wait pending until released. Existing source snapshot import and private
+transfer crash cases remain the primary regression and retain all assertions.
+
+The two permanent-holder elapsed checks retain their five-second lower and
+six-second upper observation bounds. Run only those individual wall-clock
+measurement cases with TUnit's keyless method-level `NotInParallel`, which the
+pinned1.72.10 package documents as exclusive execution. Exact Windows evidence
+shows the failed check overlapped74 distinct cases with16 active at peak; timer
+or continuation delay is a supported inference, without a threadpool trace.
+No class/assembly serialization, retry or timeout increase is allowed. Other
+holder-release/cancellation tests and every real process crash remain parallel.
+This is intrinsic readiness timing qualification, not a loaded-system latency
+claim. Source review, native case discovery and full exact-SHA CI must verify
+that the same bounds and all original crash cases remain.
+
 ## Actors, entry points and failure boundaries
 
 Actors are authenticated SDK/MCP callers, the node-local replica host, fixed voters, the membership provider and the recovery operator. Current source is composed by [ServerApplication](../../src/KeyLoad.Server/Features/ClientApi/ServerApplication.cs): it starts the node-local [PartitionHost](../../src/KeyLoad.Server/Features/StorageRecovery/PartitionHost.cs) and Orleans silo, whose [replica service](../../src/KeyLoad.Orleans/Features/ClusterReplication/PartitionReplicaGrainService.cs) routes peer operations. Aspire declares three Docker nodes with separate data mounts. This source is not yet qualified as a delivered RF3 deployment: current tests do not force request-activation migration and then verify storage ownership and a durable caller-visible outcome. Public HTTP/.NET transport belongs to ClientApi; required official MCP parity is pending. Frontend is N/A because replica consensus has no independent UI. Shared contracts stay in Abstractions and the exact ClusterReplication/StorageRecovery slice owners above.

docs/Features/StorageRecovery.md

@@ -3,13 +3,37 @@
 REQ-STORAGE-012 / AC-STORAGE-012 (TASK-RUNTIME-WINDOWS-RECOVERY-W3) preserves
 the existing killed-child filesystem readiness bound. After actual process exit,
 exclusive readiness covers owner.lock, commands.wal and the actual ZoneTree
-tree/0.meta.wal which failed to reopen in Windows CI37021991878. The prior receipt
+tree/0.meta.wal when present, which failed to reopen in Windows CI37021991878. The prior receipt
 does not identify the sharing holder; this is a test-fixture observation gap,
 not proof of a storage/dependency defect. Keep the five-second bound and25ms poll,
 all50 seeded trials, original trial deadlines, WAL/snapshot/atomic assertions and
 permanent sharing failure. Cancellation stops before another probe; no retry of
 the recovery operation, default timeout increase or filesystem bypass is allowed.
 
+TASK-RUNTIME-RECOVERY-W4 refines this same criterion using exact main
+b533c80 / CI37032546228. During checkpoint installation the disposed live tree
+is moved aside before InstallPrepared and JournalSwapped callbacks; the child
+can therefore exit with no tree directory. Only FileNotFoundException and
+DirectoryNotFoundException while opening the optional metadata WAL mean that
+this file has no holder. owner.lock and commands.wal remain required exclusive
+probes; sharing violations and all other I/O errors retain the original bound.
+No readiness probe creates files or directories. Real missing-directory and
+missing-metadata-file regressions assert successful readiness without recreating
+either path, then restore the actual store files and prove subsequent reopen
+and commit. Missing required ownership/journal files must still fail, and the
+existing held-file/cancel/pre-cancel/permanent-lock tests remain intact.
+
+The common synchronous probe is internal test infrastructure shared with
+ClusterReplication's existing store barrier. That caller retains one
+five-second deadline across target stores and, only at typed snapshot/transfer
+boundaries, both source stores, with its25ms poll; it does not perform consecutive
+independently bounded waits. Root owns that caller join;
+the StorageRecovery worker owns only the existing helper, real-file tests and
+fixture. All checkpoint process-kill, seeded trials and replica snapshot/tail
+assertions remain unchanged. Tests-first source hashes, numeric/lifetime review,
+enabled development build/format and full exact-SHA GitHub recovery all OSes
+are the join; rollback reverts only the preserving helper/caller/test refinement.
+
 New real-file tests first hold the metadata WAL exclusively after an actual
 ZoneTree store closes: the shared barrier must remain pending, complete only
 after releasing that holder, reject cancellation and fail under the same bounded
@@ -52,7 +76,7 @@ new scoped-read contract; ADR-032 records existing layout migration debt.
 | REQ-STORAGE-008: cohesive private provider owners meet numeric gates without changing storage/caller contracts | AC-SQ-001..008 in storage-quality.acceptance.md | ADR-046 TASK-MP-010AF-R/T/C/L/B; source join and real lifetime test source exist; enabled provider development build clean, complete exact-SHA runtime qualification pending |
 | REQ-STORAGE-009: validation and apply share one private mutation projection per staged generation | AC-PSW-001..004, AC-MP-006/012 | ADR-035 TASK-MP-016P-W/L; first-authored PreparedTransactionTests plus existing FrameBudget/recovery/RF3 proof; source and qualification pending |
 | REQ-STORAGE-011: startup identity metadata is finite and failed read releases physical ownership | AC-BSM-001/003/005 | [ADR-048](../ADR/ADR-048-bounded-storage-metadata.md), Metadata* real-file constructor/restore/reopen checks under BackupRestore; complete source and GitHub execution pending |
-| REQ-STORAGE-012: killed-child readiness includes the real metadata WAL and preserves cancellation and the original failure bound | AC-STORAGE-012 | `tests/KeyLoad.RecoveryTests/Features/StorageRecovery/KilledProcessFileReadinessTests.cs`: actual closed-store pending/release, cancellation, pre-cancellation and permanent-lock cases; `RecoveryTests.cs` retains every process-kill scenario; TASK-RUNTIME-WINDOWS-RECOVERY-W3, full three-OS GitHub recovery qualification pending |
+| REQ-STORAGE-012: killed-child readiness includes the real metadata WAL when present, permits only precise sanctioned metadata absence, and preserves cancellation and the original failure bound | AC-STORAGE-012 | `tests/KeyLoad.RecoveryTests/Features/StorageRecovery/KilledProcessFileReadinessTests.cs`: actual closed-store pending/release, cancellation, pre-cancellation, permanent-lock, missing optional directory/file without creation and missing required file cases; `RecoveryTests.cs` retains every process-kill scenario; TASK-RUNTIME-WINDOWS-RECOVERY-W3 and TASK-RUNTIME-RECOVERY-W4, full three-OS GitHub recovery qualification pending |
 
 Ownership: common public storage contracts stay in Abstractions/Storage;
 provider helpers in Storage.ZoneTree/Features/StorageRecovery, tests mirror that

docs/Features/TestInfrastructure.md

@@ -109,6 +109,59 @@ and exact-SHA GitHub first-failure/report/recovery artifacts supplement those
 assertions. Environmental watch/runner failures
 must not be replaced by mocks or counted as green qualification.
 
+TASK-RUNTIME-RESTART-RECEIPT-W4 refines REQ/AC-TEST-007 after exact main
+b533c80 / CI37032546228. The failed native restart attempt is currently missing
+its container generation: the first receipt shows the killed old node and later
+receipts show a successful restoration. Capture the public Aspire current
+ResourceId, closed state/health, exit code and creation/start/stop timestamps
+before Start, after its successful command and immediately after an existing
+restart failure. These are sampled snapshots, not a native snapshot version or
+proof of the transition which caused the failure. Native Version and readiness
+internals are unavailable and must not be reflected or inferred.
+
+The failure receipt includes the existing verified pre-kill Docker ID/start time,
+closed failure stage/type, Start success and one immediate read-only Docker
+inspection of ID/status/exit/OOM/start/finish/error-present. No Docker error text,
+environment, properties, endpoints, credentials, arbitrary method/type text or
+payload is included. Only validated native identifiers and timestamps are
+retained; unknown states/types become a fixed unavailable/other marker. At most
+32 earliest restart receipts plus one latest view are retained, with80-line/8KiB
+UTF-8 bounds and filenames containing only the local sequence. The diagnostic
+has its own three-second native-inspection deadline after the original failure.
+Every path after process creation owns termination, reaping and observation of
+both redirected readers, including setup, wait and reader faults. Mandatory
+cleanup can extend wall time if operating-system termination is delayed; no
+hard end-to-end three-second return or detached process/reader claim is allowed.
+Termination checks an exit race and retries a failed live-child tree kill once;
+a final live-child kill error is retained through mandatory task joining. If
+the operating system permanently refuses termination, finite cleanup cannot be
+guaranteed. This environmental branch needs native CI evidence or source review,
+never a synthetic process failure or a passing lifecycle claim.
+It releases/awaits its actual CLI process and redirected readers on cancellation,
+then rethrows the original restart exception even if diagnostic inspection or
+file writing fails. Existing synchronous whole-file persistence keeps its
+80-line/8KiB limit; no hard three-second filesystem-latency claim or detached
+writer task is allowed.
+No Start, health, runtime identity, zero-exit, cancellation, timeout, retry count,
+quorum, atomicity, data or cleanup assertion changes.
+
+Root owns ContainerRuntimeControl and the closed receipt-kind join in
+ClusterFailureReceipts; one disjoint worker owns only new ClusterReplication
+failure-capture/projection/CLI-lifetime helpers and real temporary-file tests.
+Tests first prove only selected native public snapshot fields survive with
+canary properties/env/URL/state/identifier/type rejected, whole valid bounded
+files preserve the first failed generation across later receipts, and32-file
+retention remains exact. Native-shaped Docker records cover valid identities,
+safe error-presence projection and malformed ID/state/exit/OOM/timestamp fields;
+actual stream readers prove output is drained after its retained prefix is full.
+Do not fake native services, Docker or process failures.
+The original RF3 leader-loss/minority scenario remains the real lifecycle test;
+an environmental failure branch additionally requires exact-SHA GitHub artifact
+review and source lifetime/predicate comparison. If the next run succeeds, do
+not claim its unexecuted failure path qualified. ADR-035/036/039 existing
+privacy/lifetime contracts suffice; no product/public/dependency boundary changes.
+Rollback reverts only this additive diagnostic join and its helpers/tests.
+
 ## Platform, dependency та release qualification
 
 B3 keeps the exact task lifetime above while satisfying enabled CA1031: a private