Use generated Orleans serialization for internal database payloads

Move owned typed persistence, signed claims, grain replies and replica payloads to versioned native contracts. Preserve public JSON, canonical identity digests, node-local ZoneTree WAL ownership, ordered durability barriers and RF3 authority.

Reject malformed collection counts, opaque typed references, unsupported codec shapes and cyclic or expanding DOM graphs before effects. Validate complete backup journal cuts before publishing restored stores and fail closed when existing data lacks a valid identity. Add genuine native-codec and real-file regressions; exact-source qualification runs in GitHub Actions.

Author: KSemenenko

README.md

@@ -213,7 +213,9 @@ System projection APIs require cluster administration. A consumer defines its in
 
 The canonical backup includes the checksummed redo journal, database identity and a SHA-256 manifest. Domain data, schemas, credentials, outcomes and inbox receipts are journaled together. The journal can begin with a verified checkpoint followed by newer transaction frames. ZoneTree files can be rebuilt from that canonical history. Restore validates every manifest file, creates a new incarnation, resets consensus routing metadata and leaves queue dispatch paused.
 
-The current atomic-WAL source uses generated Orleans binary mutation payloads with the native raw-byte memory codec, explicit Put/Delete kinds, frame3 and identity4; native ZoneTree raw bytes, disk-flush ordering and checkpoint2 remain. Upgrading JSON/frame1 or prior unqualified binary/frame2 journals requires stopping all RF3 writers, Compact with the matching previous binary, and a verified compacted backup before upgrading every node. A remaining full legacy journal header is refused. [ADR-057](docs/ADR/ADR-057-orleans-atomic-wal.md) defines this offline transition. The [current native receipt](docs/implementation/atomic-wal-qualification-37084177131.json) qualifies cf630751e source:1407/1407 unit cases in each normal/scalar mode,66/66 WAL cases in each,136/136 recovery and63/63 RF3, with no skips and1000 atomic process-kill cuts. Full Release, formatter/governance and118 analyzer cases pass. The separate comparative workflow was still running at capture; measured acceleration, power-loss/endurance, coverage and independent migration/resource proofs remain open.
+The owner-directed native serialization source uses generated Orleans binary codecs for internal typed records, claims, metadata, grain replies and replica payloads, with journal4/identity5/checkpoint3/backup2, KLT2 and replica2 fences. Public HTTP/MCP JSON, exact user document content, canonical command digests, sortable keys and ZoneTree raw-byte Sync WAL retain their existing contracts. Missing/legacy store identities fail before data recreation; Compact does not convert opaque record values. A qualified offline converter is not delivered, so preserve existing stores and matching old binaries. Replica upgrades require stopped writers and homogeneous versions. [ADR-060](docs/ADR/ADR-060-native-internal-serialization.md) defines these contracts; exact-source native CI and performance qualification remain pending.
+
+The [historical WAL receipt](docs/implementation/atomic-wal-qualification-37084177131.json) qualifies only cf630751e's frame3/identity4/checkpoint2 source: normal/scalar unit, WAL, process recovery, RF3 and analyzer gates passed without skips. It does not qualify the broader new formats. No measured acceleration, power-loss durability, endurance or production readiness is claimed.
 
 ```sh
 # Stop the node before using the offline CLI.

docs/ADR/ADR-060-native-internal-serialization.md

@@ -32,3 +32,64 @@ The old Compact operation preserves opaque JSON values; it is not a migration. N
 Authenticate exact peer scope and sender before replay-slot admission. Native header projection/skipping may not allocate decoded opaque payloads or grant a malformed message a nonce slot; control/data pools and all external capacity contracts stay exact. Existing state ownership, synchronous barriers, unknown outcomes, cancellation and caller-visible sanitized diagnostics remain.
 
 Native codec/type-family/DOM tests, exact record byte accounting, actual metadata corruption/legacy fixtures, claims/tamper/version tests, replica admission allocation/malformed/scope tests, existing canonical hashes, real-process recovery and genuine RF3 SDK/MCP form the acceptance chain. Required commands are enabled solution restore/build and formatter/static governance, then canonical GitHub normal/scalar/recovery/analyzer/RF3 jobs. No local runtime tests, no removed assertions or invented load tests. Performance, full memory amplification, power-loss, endurance and production proof are separately unqualified. Publication is not attempted again without explicit approval after the previous automatic-review rejection.
+
+## Resumption repair contract, 2026-10-03
+
+The owner directs continuation, concrete implementation and GitHub qualification
+after reviewing the held candidate. TASK-IS-R4A implements missing-identity
+fail-closed creation with an actual acquired owner handle and real-file regressions
+(AC-IS-004/008). TASK-IS-R4B verifies the complete backup journal and manifest cut
+before destination publication, using existing checkpoint and atomic WAL readers,
+no source modification/truncation and bounded per-frame memory (AC-IS-004).
+The restore owner retains explicit new identity/incarnation/signing authority and
+paused dispatch; malformed or incompatible backups leave destination unchanged.
+Native scalar/count/type/reference/depth/work validation repairs stay in the shared
+InternalSerialization slice and preserve official generated wire encoding.
+No old-store conversion, running-cluster rollout or product release is inferred
+from development-source installation. Existing required qualification remains.
+
+TASK-IS-R4C shared preflight retains generated encoding and official scalar
+readers. It validates expected root compatibility before dynamic dispatch, walks
+wire tags iteratively, rejects invalid reference IDs and requires native collection
+counts/completion before allocation. Wire depth is bounded at264 (four structural
+array/property/node wrappers per existing64-level semantic depth plus8 envelope
+levels); this is an explicit safety fence, not full heap qualification.
+TASK-IS-R4D validates semantic graphs once per applicable nullability context,
+rejects cycles/depth overflow and bounds DOM expansion before materialization.
+DOM retains64 semantic levels and a32MiB encoded output ceiling corresponding to
+the existing maximum configurable public HTTP body. Actual lower domain/public
+admission and reply budgets still apply; these structural fences never grant
+capacity or replace required heap, allocation and performance measurements.
+Each worker owns disjoint code/tests; only lead owns this shared limits file.
+
+Final source closure removes OperationResult.Get's internal JSON fallback. Typed
+Core results and retained outcomes carry NativeValue; JSON-only or absent typed
+results fail with Corruption, while stored domain errors retain precedence.
+The already-applied/no-retained-outcome sentinel remains an empty internal result.
+HTTP/MCP still materialize typed JSON at their existing boundaries. Existing
+public JSON text/unicode/ownership/allocation regressions call JsonDefaults
+directly with unchanged thresholds; native result rejection/roundtrip tests cover
+the removed fallback under AC-IS-001/003. No test is skipped or removed.
+
+Native v2 evolution accepts bounded unknown fields but rejects a known typed
+reference to an opaque omitted-type unknown field. Orleans otherwise replays that
+field under the later expected type, potentially bypassing the first count scan.
+No homogeneous v2 generated producer requires this ambiguous deferred decode;
+support for it requires a separately specified bounded replay contract. Ordinary
+known-schema references remain supported. AC-IS-002 includes the concrete hidden
+underfilled float-array reference regression before native allocation.
+
+TASK-IS-R4E aligns replica inspection and malformed fixtures with Orleans10.3.1:
+explicit property Ids are body members, preceded by the empty constructor scope.
+The Server authentication projection follows the same genuine generated record
+shape. Official generated writer-to-inspector positive tests cover these joins;
+hand-authored malformed tests retain exactly their intended corruption and bounds.
+
+The inspected native type closure is intentionally limited to owned generated
+KeyLoad DTOs, existing scalar/date/byte/JSON DOM codecs, rank-one arrays, and the
+exact collection/surrogate shapes used by the contracts (ImmutableArray, List,
+Dictionary, KeyValuePair and Memory/ReadOnlyMemory). Other System surrogates and
+derived collection codecs fail closed before generated allocation, even below an
+object-typed field. Extending this closure requires a native shape specification,
+preflight/count/reference tests and homogeneous compatibility qualification.
+This is an internal concrete contract, not arbitrary Orleans codec compatibility.

docs/Architecture.md

@@ -645,16 +645,16 @@ flowchart LR
 [ADR-060](ADR/ADR-060-native-internal-serialization.md) and
 [InternalSerialization](Features/InternalSerialization.md) require native generated
 Orleans payloads with stable aliases and field IDs across owned persistence,
-replication, grain requests/replies, membership and signed claims. Attributed
-contracts and parts of persistence are applied; the shared codec, security-token
-and inter-node format changes remain review candidates awaiting explicit rollout
-approval after automatic approval rejection. The isolated shared-codec compiler
-preview passes; no expanded native runtime or fault qualification is claimed.
+replication, grain requests/replies, membership and signed claims. The owner-directed resumption installs the reviewed native source after rebasing
+against current work. Shared collection/reference/depth/graph/DOM validation and
+missing-identity/backup-cut regressions are being integrated before exact-source
+GitHub qualification. Historical isolated compilation and WAL-only qualification
+do not establish expanded runtime, fault or performance results.
 
 Public HTTP/MCP JSON, exact user content, frozen canonical fingerprint identities,
 sortable keys, raw ZoneTree values, fixed checksum/HMAC framing and blob bytes
 retain their concrete protocols. Native generated bytes are not canonical hashes
-for unordered values. The proposed journal4/identity5/checkpoint3/backup2 and
+for unordered values. The source journal4/identity5/checkpoint3/backup2 and
 KLT2/replica2 formats refuse old data or tokens; Compact preserves opaque records
 and cannot convert them. No offline converter is delivered. An old-store upgrade
 remains blocked, and replication requires a homogeneous stopped-writes rollout.

docs/Features/InternalSerialization.md

@@ -15,16 +15,27 @@ flowchart LR
 
 ## Requirements and acceptance
 
-REQ-IS-001 through REQ-IS-009 respectively require the complete attributed DTO closure, pooled strict native codec, typed persistence/accounting, versioned storage metadata, secure bounded replication, native grain/membership contracts, versioned native signed claims, explicit non-destructive migration and exact-source qualification. They map one-to-one to AC-IS-001..009 and the test/flow matrix in the root acceptance file. Each missing or malformed case must fail before effects; deserialize defaults are not authorization or semantic validation.
+REQ-IS-001 through REQ-IS-009 respectively require the complete attributed DTO closure, pooled strict native codec, typed persistence/accounting, versioned storage metadata, secure bounded replication, native grain/membership contracts, versioned native signed claims, explicit non-destructive migration and exact-source qualification. They map one-to-one to AC-IS-001..009 and the [test/flow matrix](InternalSerialization/Acceptance.md). Each missing or malformed case must fail before effects; deserialize defaults are not authorization or semantic validation.
 
 ## Ownership and verification
 
 [ADR-060](../ADR/ADR-060-native-internal-serialization.md) owns formats and ordered implementation. Abstractions owns Features/InternalSerialization and generated public DTOs; Core owns each model's private records/readers/accounting; Replication owns ClusterReplication persistence/protocols; Storage.ZoneTree owns StorageRecovery/BackupRestore metadata; Orleans owns ClusterRouting/ClusterReplication internal transport. Existing public ClientApi remains its actual JSON protocol boundary.
 
 Tests mirror InternalSerialization and the affected existing slices. Generated codecs and small real-file fixtures cover type/null/default/empty/byte/DOM/enum/polymorphism boundaries. Real process fault cuts and genuine Docker RF3 SDK/MCP prove recovery/runtime behavior only at the qualified SHA. Local runtime tests/benchmarks are forbidden; separate compiler, formatter and governance checks are static evidence.
 
-Current status: attributes are partly applied and dependent runtime changes remain under the already executed source hold. The fully joined temporary snapshot passed 25-project Release compilation with real analyzers, zero warnings/errors, and the required full formatter. Its 106 affected/new native test files contain 296 test declarations and 301 Arguments attributes; these are unexecuted source counts. [Static receipt](../implementation/native-internal-static-preview.json) records exact source/patch/log hashes and excluded moving owner work. That snapshot requires fresh rebase and checks before installation; it does not qualify latest owner source or runtime behavior.
+Historical static preview: attributes were partly applied and dependent runtime changes remained under the executed source hold. The fully joined temporary snapshot passed 25-project Release compilation with real analyzers, zero warnings/errors, and the required full formatter. Its 106 affected/new native test files contain 296 test declarations and 301 Arguments attributes; these are unexecuted source counts. [Static receipt](../implementation/native-internal-static-preview.json) records exact source/patch/log hashes and excluded moving owner work. That snapshot requires fresh rebase and checks before installation; it does not qualify latest owner source or runtime behavior.
 
 The final candidate includes attributed ChangeFeedClaims and native grain-to-HTTP/MCP principal binding, retaining existing admission/accounting and narrow strict grammar checks. Known ungenerated enum headers follow the official backing integer codec. An internal public-input profile permits only null reference collection elements so existing command/query validators and valid traversal-label behavior remain intact; required roots/members/initialized arrays, dictionaries, persisted records, claims and outputs stay strict. Public HTTP/MCP/CLI JSON, user JSON content, frozen identity material and the explicitly user-selected local-profile.json configuration file retain their concrete external boundaries.
 
-The active checkout retains earlier formats and the ValueOperand factory join remains staged. Automatic approval review rejected shared-codec persisted formats, KLT2 old-token invalidation and binary inter-node rollout pending direct consent. Native installation, offline conversion, general nested decoded/type/reference/work bounds and exact-source mandatory native qualification remain open. Prior ADR-057 WAL proof is historical and does not qualify this migration. No measured acceleration, power-loss or production result follows from this static preview.
+Current owner-directed resumption installs the native candidate after a fresh rebase, preserving unrelated phase telemetry, scoped term observations and SQL work. The previous automatic-review rejection and source hold are historical. Missing-identity, backup-cut and shared collection/reference/depth/graph/DOM guards are being repaired with real regression fixtures before exact-source GitHub qualification. Legacy stores remain fail-closed; a qualified offline converter and reverse conversion are not delivered. Prior ADR-057 WAL proof qualifies only its historical source, not this wider migration. No measured acceleration, power-loss or production result is claimed.
+
+Native v2 preserves bounded unknown fields and known-schema references. Typed
+references to opaque omitted-type unknown fields fail closed before allocation;
+deferred schema inference is outside this homogeneous rollout contract. Replica
+and authentication projections read the actual generated constructor/body scopes,
+including empty constructor scopes for explicitly attributed record properties.
+
+Native preflight supports the owning generated DTO closure and its specified
+scalar/array/collection/surrogate shapes. Unmodeled System surrogate and derived
+collection codecs are rejected before decode; they cannot bypass count guards
+through an object-typed member. General heap amplification remains unqualified.