Checkpoint all C1 controls and fresh due attempts

Full working-tree checkpoint requested by the owner. Release build79f failed with compiler and analyzer errors; C1 RF3 and fresh due runtime acceptance remain unverified. Continue repairs and Aspire-owned verification from this committed source.

Author: KSemenenko

docs/Features/ClusterRouting/NativeCqrsRequestV2.md

@@ -683,3 +683,70 @@ claim log privacy from filtered diagnostics or report remote-voter death.
 Root runs the two actual Aspire RF3 cases and required exact-source Linux gates;
 source alone cannot close these criteria. Existing ADR-034/058 and C1 control
 contracts cover this test surface without public/format/topology changes.
+
+### C1 guarded offline outcome observation
+
+TASK-CRS-C1-OUTCOME-INSPECTION supplies the missing native-store oracle for
+AC-CRS-004/005. Root accepts the existing guarded current-format recovery open
+after complete RF3 shutdown. This is a logical canonical-state observation;
+native journal recovery may replay, truncate an incomplete tail and flush.
+It is not a byte-preserving forensic reader, format converter, power-loss test
+or public command-outcome API. Existing ADR-034/058 and the guarded-store
+ownership contract cover this test-only addition; persisted records and
+generated native serializer contracts remain unchanged.
+
+1. Add the distinct CrashHost mode `c1-outcome-inspect`. Its closed V1 private
+   UTF-8 input contains only `Version`, `Directory`, `ExpectedNodeId`,
+   `Incarnation`, `PrincipalId` and `CommandId`. Reject unknown/duplicate/missing
+   or null fields, invalid UTF-8, trailing content, empty IDs, non-canonical
+   paths and principal identifiers over 256 UTF-8 bytes. Bound input to 8192
+   bytes and JSON depth to four before the native existing-store open.
+2. Open only through `ZoneTreeExistingStore.Open`, with the supplied observed
+   node ID and same-wave incarnation. Do not bootstrap, create directories,
+   add a cache/fault observer, decode keys/records independently, use reflection
+   or call a format migration. Construct the normal DatabaseEngine with its
+   persisted-store policy implementation and call `Outcome(principal,id)`.
+   The result oracle is only null versus non-null.
+3. On successful native close, emit a closed V1 receipt containing only
+   `Version`, `NodeId`, `Incarnation`, `FormatVersion`, `Position` and
+   `OutcomePresent`, bounded to 4096 UTF-8 bytes. Never emit a path, principal,
+   command, result content, credential or exception message/data/stack. Invalid
+   input or failed open/read/close exits with fixed code 2 and no receipt.
+   Preserve original failures internally and always close the owned native
+   store; successful opening alone is insufficient.
+4. Root joins the additive dispatcher, project reference and internal friend
+   metadata. The existing guard inspector and recovery modes stay exact. A
+   parent invoked by the Aspire-owned runner holds the actual outer
+   `node.owner.lock` through the original child exit, bounded pipe drains and
+   process-handle close. The child acquires the canonical inner owner lock;
+   the parent must not hold that same inner lock.
+5. The two held-authority scenarios capture actual node status IDs and the
+   wave incarnation while the real cluster runs. Before revocation, a real
+   caller command writes a separate positive-control document and records its
+   command ID. Capture the no-effect baseline after that control. After the
+   original held request, callbacks, public callers, diagnostics and all
+   Aspire resources have joined and every owner-lock check passes, inspect the
+   held and positive-control IDs on each actual node store. Require absence of
+   the held outcome on all three nodes and presence of the positive control on
+   at least the acknowledged quorum of two. Do not assume every follower had
+   applied the latest command merely because the client obtained an ACK.
+6. Add actual existing-store process regressions for present/absent outcomes,
+   wrong node/incarnation fail-closed behavior and strict malformed input.
+   Retain the original child and cleanup failures. Execute these through the
+   same Aspire unit/scalar caller. Root then runs the two actual RF3 authority
+   cases and exact-source Linux gates; private code or local mechanism tests
+   do not qualify AC-CRS-004/005 by themselves.
+
+Agent graph: cluster_wave Luna/high owns only new CrashHost ClusterRouting
+Contracts/Serialization/Helpers files prefixed C1OutcomeInspection, privately
+against exact absent bases. lifecycle_wave Luna/high owns new CrashHost
+ClusterRouting Processes files prefixed C1OutcomeInspection for one shared
+bounded parent lifetime and new UnitTests ClusterRouting Cases/Helpers files
+with that prefix. Unit and RF3 callers reuse the child DTOs and that one parent
+lifetime; do not duplicate either. Parent execution and cleanup bounds are 30
+and 15 seconds respectively; neither bound substitutes a completed wrapper for
+joining its original child/pipe tasks. Root owns shared dispatch, metadata,
+RF3 status capture, positive-control/cleanup ordering and the parent-call join.
+Workers must not edit the checkout or start builds/tests/processes/Git; deliver
+complete private source, patch, hashes and self-review. Root reads every byte,
+integrates, runs the gates and commits the complete stage scope.

src/KeyLoad.AppHost/Features/ClusterReplication/Resources/ClusterResources.cs

@@ -50,8 +50,9 @@ internal static IResourceBuilder<ContainerResource>[] Add(IDistributedApplicatio
         var nodeNames = ReadNodeNames(benchmarkNodeCount);
         ClusterProfileStore.Validate(profile);
         var root = Path.GetFullPath(dataRoot);
-        ClusterProfileStore.PrepareDirectory(root);
         var images = ProtocolCohortImages.Read(builder, ephemeral, benchmarkNodeCount);
+        var probes = RequestCqrsProbeProfile.Read(builder, root, ephemeral, benchmarkNodeCount, images);
+        ClusterProfileStore.PrepareDirectory(root);
         var signing = builder.AddParameter(SigningParameter, profile.SigningKey, secret: true);
         var peer = builder.AddParameter(PeerParameter, profile.PeerSecret, secret: true);
         var admin = builder.AddParameter(AdminParameter, profile.AdminKey, secret: true);
@@ -82,6 +83,7 @@ internal static IResourceBuilder<ContainerResource>[] Add(IDistributedApplicatio
                 .WithEnvironment(HttpPortEnvironment, HttpPort.ToString(CultureInfo.InvariantCulture))
                 .WithHttpHealthCheck(ReadyPath, endpointName: HttpEndpoint);
             ClusterResourceSettings.Apply(builder, resource, containerUser);
+            probes?.Apply(resource, name);
             ApplyPeers(resource, nodeNames, benchmarkNodeCount.HasValue);
             nodes[index] = resource;
         }

src/KeyLoad.AppHost/Features/ClusterRouting/Contracts/RequestCqrsProbeOwnerRecord.cs

@@ -0,0 +1,3 @@
+namespace KeyLoad.AppHost.Features.ClusterRouting;
+
+internal sealed record RequestCqrsProbeOwnerRecord(int Version, string Kind, string SessionId, string Voter);

src/KeyLoad.AppHost/Features/ClusterRouting/Helpers/RequestCqrsProbeOwnerReader.cs

@@ -0,0 +1,94 @@
+using System.Text.Json;
+using KeyLoad.Storage.IO;
+
+namespace KeyLoad.AppHost.Features.ClusterRouting;
+
+/// <summary>Validates the exact private Owner record with bounded native JSON parsing.</summary>
+internal static class RequestCqrsProbeOwnerReader
+{
+    internal const int MaximumOwnerBytes = 8_192;
+    internal const UnixFileMode PrivateFileMode = UnixFileMode.UserRead | UnixFileMode.UserWrite;
+    private const int MaximumJsonDepth = 1;
+    private const int RequiredFieldCount = 4;
+    private const int OwnerVersion = 1;
+    private const string VersionField = "Version";
+    private const string KindField = "Kind";
+    private const string SessionField = "SessionId";
+    private const string VoterField = "Voter";
+    private const string OwnerKind = "Owner";
+    private const string InvalidConfiguration = "RequestCqrsProbeConfigurationInvalid";
+
+    internal static RequestCqrsProbeOwnerRecord ReadFile(string path)
+    {
+        if (OperatingSystem.IsWindows())
+        { throw new InvalidOperationException(InvalidConfiguration); }
+        var identity = OfflineRegularFile.Inspect(path);
+        if (identity.Length is <= 0 or > MaximumOwnerBytes
+            || File.GetUnixFileMode(path) != PrivateFileMode)
+        { throw new InvalidOperationException(InvalidConfiguration); }
+        using var input = OfflineRegularFile.OpenWithIdentity(path, identity, FileAccess.Read,
+            FileShare.Read, bufferSize: 4_096);
+        if (input.Length != identity.Length || File.GetUnixFileMode(path) != PrivateFileMode)
+        { throw new InvalidOperationException(InvalidConfiguration); }
+        var bytes = new byte[checked((int)identity.Length)];
+        input.ReadExactly(bytes);
+        if (input.ReadByte() != -1 || input.Length != identity.Length
+            || OfflineRegularFile.Inspect(path) != identity)
+        { throw new InvalidOperationException(InvalidConfiguration); }
+        return Read(bytes);
+    }
+
+    internal static RequestCqrsProbeOwnerRecord Read(byte[] bytes)
+    {
+        if (bytes.Length is <= 0 or > MaximumOwnerBytes)
+        { throw new InvalidOperationException(InvalidConfiguration); }
+        using var document = JsonDocument.Parse(bytes, new JsonDocumentOptions { MaxDepth = MaximumJsonDepth });
+        if (document.RootElement.ValueKind != JsonValueKind.Object)
+        { throw new InvalidOperationException(InvalidConfiguration); }
+        var seen = new HashSet<string>(StringComparer.Ordinal);
+        var version = 0;
+        string? kind = null;
+        string? session = null;
+        string? ownerVoter = null;
+        foreach (var field in document.RootElement.EnumerateObject())
+        {
+            if (!seen.Add(field.Name))
+            { throw new InvalidOperationException(InvalidConfiguration); }
+            ReadField(field, ref version, ref kind, ref session, ref ownerVoter);
+        }
+        if (seen.Count != RequiredFieldCount || version != OwnerVersion || kind != OwnerKind
+            || session is null || ownerVoter is null)
+        { throw new InvalidOperationException(InvalidConfiguration); }
+        return new(version, kind, session, ownerVoter);
+    }
+
+    private static void ReadField(JsonProperty field, ref int version, ref string? kind,
+        ref string? session, ref string? voter)
+    {
+        switch (field.Name)
+        {
+            case VersionField when field.Value.ValueKind == JsonValueKind.Number && field.Value.TryGetInt32(out var value):
+                version = value;
+                break;
+            case KindField:
+                kind = ReadString(field.Value);
+                break;
+            case SessionField:
+                session = ReadString(field.Value);
+                break;
+            case VoterField:
+                voter = ReadString(field.Value);
+                break;
+            default:
+                throw new InvalidOperationException(InvalidConfiguration);
+        }
+    }
+
+    private static string ReadString(JsonElement value)
+    {
+        if (value.ValueKind != JsonValueKind.String)
+        { throw new InvalidOperationException(InvalidConfiguration); }
+        return value.GetString() is { Length: > 0 } text
+            ? text : throw new InvalidOperationException(InvalidConfiguration);
+    }
+}

src/KeyLoad.AppHost/Features/ClusterRouting/Helpers/RequestCqrsProbeProfile.cs

@@ -0,0 +1,66 @@
+using KeyLoad.AppHost.Features.ClusterReplication;
+using Microsoft.Extensions.Configuration;
+
+namespace KeyLoad.AppHost.Features.ClusterRouting;
+
+/// <summary>Admits the private per-voter phase-control mount only for an ephemeral RF3 host.</summary>
+internal sealed class RequestCqrsProbeProfile
+{
+    internal const string Section = "KeyLoadTests:RequestCqrsProbe";
+    private const string MountRoot = "/request-probes";
+    private const string EnabledEnvironment = "KeyLoad__RequestCqrsProbe__Enabled";
+    private const string RootEnvironment = "KeyLoad__RequestCqrsProbe__Root";
+    private const string SessionEnvironment = "KeyLoad__RequestCqrsProbe__SessionId";
+    private const string EnabledValue = "true";
+    private const string InvalidConfiguration = "RequestCqrsProbeConfigurationInvalid";
+
+    private readonly IReadOnlyDictionary<string, string> nodeRoots;
+
+    private RequestCqrsProbeProfile(string sessionId, IReadOnlyDictionary<string, string> nodeRoots)
+    { SessionId = sessionId; this.nodeRoots = nodeRoots; }
+
+    internal string SessionId { get; }
+
+    internal static void ValidateMode(IConfiguration configuration)
+    { _ = RequestCqrsProbeProfileSettingsReader.Read(configuration); }
+
+    internal static RequestCqrsProbeProfile? Read(IDistributedApplicationBuilder builder, string dataRoot,
+        bool ephemeral, int? benchmarkNodeCount, IReadOnlyDictionary<string, RuntimeContainerImage> images)
+    {
+        ArgumentNullException.ThrowIfNull(builder);
+        var settings = RequestCqrsProbeProfileSettingsReader.Read(builder.Configuration);
+        if (settings is null)
+        { return null; }
+        if (!ephemeral || benchmarkNodeCount is not null)
+        { throw new InvalidOperationException(InvalidConfiguration); }
+        ArgumentException.ThrowIfNullOrWhiteSpace(dataRoot);
+        ArgumentNullException.ThrowIfNull(images);
+        ValidateImages(images);
+        var nodeRoots = RequestCqrsProbeProfilePaths.Validate(settings.Root, settings.SessionId, dataRoot);
+        return new(settings.SessionId, nodeRoots);
+    }
+
+    internal void Apply(IResourceBuilder<ContainerResource> resource, string node)
+    {
+        ArgumentNullException.ThrowIfNull(resource);
+        if (!nodeRoots.TryGetValue(node, out var nodeRoot))
+        { throw new InvalidOperationException(InvalidConfiguration); }
+        resource.WithBindMount(nodeRoot, MountRoot)
+            .WithEnvironment(EnabledEnvironment, EnabledValue)
+            .WithEnvironment(RootEnvironment, MountRoot)
+            .WithEnvironment(SessionEnvironment, SessionId);
+    }
+
+    private static void ValidateImages(IReadOnlyDictionary<string, RuntimeContainerImage> images)
+    {
+        if (images.Count != RequestCqrsProbeProfileSettingsReader.VoterNames.Count
+            || RequestCqrsProbeProfileSettingsReader.VoterNames.Any(node => !images.TryGetValue(node, out var image) || image is null))
+        { throw new InvalidOperationException(InvalidConfiguration); }
+        var voters = RequestCqrsProbeProfileSettingsReader.VoterNames;
+        var reference = images[voters[0]].Reference;
+        if (string.IsNullOrWhiteSpace(reference)
+            || !string.Equals(images[voters[1]].Reference, reference, StringComparison.Ordinal)
+            || !string.Equals(images[voters[2]].Reference, reference, StringComparison.Ordinal))
+        { throw new InvalidOperationException(InvalidConfiguration); }
+    }
+}

src/KeyLoad.AppHost/Features/ClusterRouting/Helpers/RequestCqrsProbeProfilePaths.cs

@@ -0,0 +1,105 @@
+using System.Text.Json;
+
+namespace KeyLoad.AppHost.Features.ClusterRouting;
+
+/// <summary>Validates pre-created private control paths without creating, changing or deleting them.</summary>
+internal static class RequestCqrsProbeProfilePaths
+{
+    private const string OwnerFileName = "owner.json";
+    private const string InvalidConfiguration = "RequestCqrsProbeConfigurationInvalid";
+    private const int MaximumPathCharacters = 4_096;
+    private const UnixFileMode PrivateDirectoryMode = UnixFileMode.UserRead | UnixFileMode.UserWrite | UnixFileMode.UserExecute;
+
+    internal static IReadOnlyDictionary<string, string> Validate(string configuredRoot, string sessionId,
+        string dataRoot)
+    {
+        try
+        {
+            ValidatePlatform();
+            var root = CanonicalDirectory(configuredRoot);
+            var data = CanonicalDirectory(dataRoot);
+            if (IsWithin(root, data) || IsWithin(data, root))
+            { throw new InvalidOperationException(InvalidConfiguration); }
+            ValidateDirectoryAndAncestors(root, PrivateDirectoryMode);
+            ValidateRootEntries(root);
+            var nodes = new Dictionary<string, string>(StringComparer.Ordinal);
+            foreach (var voter in RequestCqrsProbeProfileSettingsReader.VoterNames)
+            {
+                var directory = Path.Combine(root, voter);
+                ValidateDirectoryAndAncestors(directory, PrivateDirectoryMode);
+                ValidateOwnerDirectory(directory, sessionId, global::ClusterResources.Origin(voter));
+                nodes.Add(voter, directory);
+            }
+            return nodes;
+        }
+        catch (Exception error) when (IsPathOrFileFailure(error))
+        { throw new InvalidOperationException(InvalidConfiguration); }
+    }
+
+    private static string CanonicalDirectory(string path)
+    {
+        if (path.Length is 0 or > MaximumPathCharacters || !Path.IsPathFullyQualified(path))
+        { throw new InvalidOperationException(InvalidConfiguration); }
+        var full = Path.GetFullPath(path);
+        var trimmed = Path.TrimEndingDirectorySeparator(full);
+        if (!string.Equals(full, path, StringComparison.Ordinal)
+            || !string.Equals(trimmed, full, StringComparison.Ordinal)
+            || string.Equals(trimmed, Path.GetPathRoot(full), StringComparison.Ordinal))
+        { throw new InvalidOperationException(InvalidConfiguration); }
+        return full;
+    }
+
+    private static void ValidatePlatform()
+    {
+        if (OperatingSystem.IsWindows())
+        { throw new PlatformNotSupportedException(InvalidConfiguration); }
+    }
+
+    private static void ValidateDirectoryAndAncestors(string path, UnixFileMode? expectedMode)
+    {
+        if (OperatingSystem.IsWindows())
+        { throw new InvalidOperationException(InvalidConfiguration); }
+        for (var current = path; current is not null; current = Path.GetDirectoryName(current))
+        {
+            var attributes = File.GetAttributes(current);
+            if ((attributes & FileAttributes.Directory) == 0 || (attributes & FileAttributes.ReparsePoint) != 0)
+            { throw new InvalidOperationException(InvalidConfiguration); }
+            if (string.Equals(current, path, StringComparison.Ordinal) && expectedMode is { } mode
+                && File.GetUnixFileMode(current) != mode)
+            { throw new InvalidOperationException(InvalidConfiguration); }
+        }
+    }
+
+    private static void ValidateRootEntries(string root)
+    {
+        var voters = RequestCqrsProbeProfileSettingsReader.VoterNames;
+        var entries = Directory.EnumerateFileSystemEntries(root).Take(voters.Count + 1).ToArray();
+        if (entries.Length != voters.Count
+            || voters.Any(voter => !entries.Contains(Path.Combine(root, voter), StringComparer.Ordinal)))
+        { throw new InvalidOperationException(InvalidConfiguration); }
+    }
+
+    private static void ValidateOwnerDirectory(string directory, string sessionId, string voter)
+    {
+        var entries = Directory.EnumerateFileSystemEntries(directory).Take(2).ToArray();
+        if (entries.Length != 1 || !string.Equals(Path.GetFileName(entries[0]), OwnerFileName, StringComparison.Ordinal))
+        { throw new InvalidOperationException(InvalidConfiguration); }
+        var ownerPath = Path.Combine(directory, OwnerFileName);
+        var owner = RequestCqrsProbeOwnerReader.ReadFile(ownerPath);
+        if (!string.Equals(owner.SessionId, sessionId, StringComparison.Ordinal)
+            || !string.Equals(owner.Voter, voter, StringComparison.Ordinal))
+        { throw new InvalidOperationException(InvalidConfiguration); }
+    }
+
+    private static bool IsWithin(string candidate, string root)
+    {
+        var relative = Path.GetRelativePath(root, candidate);
+        return relative == "." || (!Path.IsPathRooted(relative) && relative != ".."
+            && !relative.StartsWith(".." + Path.DirectorySeparatorChar, StringComparison.Ordinal)
+            && !relative.StartsWith(".." + Path.AltDirectorySeparatorChar, StringComparison.Ordinal));
+    }
+
+    private static bool IsPathOrFileFailure(Exception error)
+        => error is IOException or UnauthorizedAccessException or ArgumentException or NotSupportedException
+            or System.Security.SecurityException or JsonException or global::KeyLoad.KeyLoadException;
+}