Skip to content

Commit 1e8833c

Browse files
committed
Checkpoint C1 ownership, due recovery and bounded native work
Commit the full current checkout scope: native outcome inspection/probe controls, Aspire resource-log admission and joined cleanup, independent due/outbox replay oracles, bounded native partition pages and ANN terminal build observations. Freeze shortest-path requirements before integrating its private implementation. Validation: solution Release build (0 warnings/errors), formatter, governance, and 46 focused unit cases through the actual Aspire AppHost passed. Original whole-source/runtime inventories stayed unchanged during these runs. The ANN wide case used 10,000 records and real cancellation/deadline controls. Complete unit/scalar/recovery/RF3, current-image Linux fault acceptance, physical partition transfer and performance/endurance qualification remain pending. No original acceptance task is marked complete by this checkpoint.
1 parent 92c826e commit 1e8833c

96 files changed

Lines changed: 4421 additions & 526 deletions

File tree

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

‎docs/ADR/ADR-016-atomic-physical-placement.md‎

Lines changed: 19 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -24,6 +24,25 @@ ClusterRouting `REQ-ROUTE-001..003`/`AC-ROUTE-001..003`; ClusterReplication `REQ
2424

2525
Dependencies: [ADR-001](ADR-001-partition-identity-affinity.md), [ADR-003](ADR-003-durability-ack-barrier.md), [ADR-004](ADR-004-committed-read-views.md), [ADR-005](ADR-005-canonical-keyspace-codec.md), [ADR-007](ADR-007-replica-consensus-bootstrap.md), [ADR-008](ADR-008-backup-log-retention.md), and [ADR-017](ADR-017-migration-tokens.md). Stop if a plan transfers journal/lock ownership with an activation or changes logical AtomicPartitionId during a physical move.
2626

27+
## First accepted native page stage, 2026-10-05
28+
29+
TASK-PMOVE-PAGES and TASK-PMOVE-PAGE-ORACLES implement
30+
[REQ/AC-PMOVE-001..004](../Features/ClusterRouting/PartitionTransfer.md) as pure
31+
internal bounded record reads over an existing owned native committed view.
32+
Root freezes the listed reader signature, complete partition identity, exact raw
33+
bytes and record/retained/examined bounds. The implementation worker owns only
34+
new Core ClusterRouting Queries/Contracts/Validation files; the independent
35+
test worker owns only new UnitTests ClusterRouting Cases/Helpers. Root reviews
36+
both packets, joins them and runs strict build plus actual Aspire normal/scalar
37+
cases before checkpointing all code. No new package or project is required.
38+
39+
There is no persisted-format rollout in this stage. Removing the unused
40+
internal primitive is its rollback; no data, token or journal is rewritten.
41+
Installation remains inadmissible until partition-associated outcomes, shared
42+
authorization/catalog/blob accounting and source fencing are frozen and tested.
43+
The final movement stages and all process/RF3 gates above remain required;
44+
this stage does not mark this ADR Implemented or close KL-036/071/072.
45+
2746
```mermaid
2847
flowchart LR
2948
Many[Many stable AtomicPartitionIds] --> Pack[Physical shard placement]

‎docs/ADR/ADR-019-managed-ann.md‎

Lines changed: 24 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -2,6 +2,16 @@
22

33
Status: Accepted for the bounded first-party managed HNSW candidate stage on 2026-10-04. Online projection, persisted index format and public ANN capability remain unqualified and gated by later contracts.
44

5+
The accepted 2026-10-05 test-resource stage is REQ/AC-ANN-013 in
6+
[ManagedAnn](../Features/Search/ManagedAnn.md). Root freezes/reviews/joins; a Luna
7+
worker adds only the named Search test resource and native keyed TUnit attributes
8+
on the specified five heavyweight fixture methods. Test bodies, corpus sizes,
9+
product execution budgets and global runner concurrency remain unchanged. Verify
10+
the private base/post packet, full strict build/format/governance, Aspire normal
11+
and scalar suites and exact-source Linux originals before any qualification
12+
claim. Rollback removes only the resource key and attributes; there is no data,
13+
dependency, public API or production admission migration.
14+
515
## Context and decision
616

717
Exact vector scans are the correctness oracle and current public source path. The product prefers a managed-first ANN implementation; a native provider adds deployment, license, persistence, concurrency, deletion, and memory-ownership risks. Root selects an independently authored KeyLoad-owned packed managed HNSW candidate for the first computational stage, with no new package, native binary or project. This decision approves implementation of the bounded candidate and its independent real-store tests; it does not qualify recall, an online projection, a persisted format or a public capability.
@@ -111,3 +121,17 @@ class/value traversal and allocation tests, then retains full Release/static and
111121
Aspire native/scalar corpus evidence. Exact-source Linux and ANN lifecycle/RF3
112122
acceptance remain required. No persistence or public transport changes occur;
113123
rollback reverts these computational joins, with no store migration.
124+
125+
## Accepted unchanged-build observations, 2026-10-05
126+
127+
REQ/AC-ANN-014 and TASK-ANN-BUILD-OBSERVATION are frozen in
128+
[ManagedAnn](../Features/Search/ManagedAnn.md#accepted-construction-observation-contract-2026-10-05).
129+
The Luna worker owns only the four existing heavy Search case files and two new
130+
feature-local model/helper files in that contract; root owns review, join, strict
131+
checks, actual Aspire normal/scalar evidence and all-code commits. Measure the
132+
single existing synchronous Build after retaining its original budget. Emit
133+
safe counters, elapsed ticks and current-thread allocations only after it settles,
134+
with original/fatal failure priority. Preserve every original resource limit,
135+
input, assertion and admission setting. No dependency, production, data or wire
136+
change occurs; rollback removes only the diagnostic wrappers. This stage measures
137+
failure context without claiming a cause, deadline repair or performance gain.

‎docs/ADR/ADR-082-native-cqrs-streams.md‎

Lines changed: 14 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -119,3 +119,17 @@ Accepted before delegated writes: TASK-CRS-C1-BOUNDARY applies the existing REQ/
119119
Accepted before delegated implementation: TASK-CRS-C1-PHASE/CONTROL/FAULT use the exact native seams, optional silo-only observer, finite private AppHost controls, SDK/official-MCP outcome distinctions, authority/receipt/privacy oracles and worker scopes in NativeCqrsRequestV2. No public fault transport, canonical format, extra dispatcher or storage owner is added. Producer disposal is observed synchronously before deactivation scheduling, with every cleanup failure preserved. Migration of the stable command-partition activation still requires a separately frozen native scheduling/membership join and actual changed activation/silo evidence; a request activation's routine deactivation is not migration. Root owns control schema, AppHost/Server registration, fixture integration, original artifacts and final gates. Ordinary operation has no probe allocations or retained control state. Rollback disables the explicitly enabled ephemeral test profile; native request protocol/data contracts remain unchanged. This ADR remains Accepted until its full product and Linux qualification criteria pass.
120120

121121
The exact private Version1 Owner/Arm/Release/Marker schema, file/memory bounds, disabled-mode contract, silo-only registration and ordered Server/AppHost/fixture integration are accepted in NativeCqrsRequestV2's "Accepted private phase-control schema and join" section before worker writes. Native migration is excluded from that schema. Private patches and local unit passes do not establish actual RF3 or delivered-source acceptance.
122+
123+
## C1 native resource-log lifecycle contract, 2026-10-05
124+
125+
TASK-CRS-DIAG-DRAIN implements REQ/AC-CRS-DIAG-003 in NativeCqrsRequestV2:
126+
native subscriber admission precedes actual AppHost startup, the real malformed
127+
guard call awaits its exact sanitized live record, and actual stop is followed
128+
by native stream completion and original-watcher drain before artifact write.
129+
The feature freezes ordered stages, precise IntegrationTests helper ownership,
130+
bounded waiter/storage, cancellation and fatal/cleanup failure preservation,
131+
root-only joins and genuine current-image Linux RF3 verification. No parser,
132+
assertion, public contract, canonical data, topology or dependency is changed.
133+
Rollback removes the test-only lifecycle stage. Original failed artifacts
134+
remain evidence; mechanism tests and source review do not qualify real guard
135+
publication or the complete ADR.
Lines changed: 30 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,30 @@
1+
# ADR-098: Bounded authorized shortest paths
2+
3+
Status: Accepted; implementation and qualification pending, 2026-10-05.
4+
5+
KL-023 requires unweighted shortest paths and batched frontiers beyond existing
6+
reachability. Select one versioned directed same-partition path operator over
7+
native node-local ZoneTree adjacency in one current authorized committed cut.
8+
Explicit depth frontiers batch local work without per-edge RPC. Orleans retains
9+
one request grain and native ManagedCode CQRS; storage handles remain node-local.
10+
11+
The implementation contract is [ShortestPath](../Features/GraphTraversal/ShortestPath.md),
12+
REQ-GRAPH-007..010, AC-GRAPH-006..009, TASK-KL023-PATH. It freezes new version1
13+
aliases/IDs, deterministic BFS ties, empty/zero-hop results, conservative metadata
14+
retention, real read/time/cancellation caps, Q1.GraphPath.v1 literal/parameter
15+
grammar and shared deadline. Existing ADR-004/005/010/014/034/082 continue to own
16+
committed reads, keys, persisted authority and native routing/lifecycle.
17+
18+
Ordered stages and exact files/roles are in that contract: root freezes; private
19+
Luna Core/contracts packet; independently authored real-store tests; root SQL
20+
compile/parity; root Orleans/HTTP/SDK/official MCP and Aspire RF3 joins; full strict
21+
checks and original delivered-source Linux evidence. Root owns all shared/Git/
22+
acceptance joins. Workers cannot alter packages, shared configuration or formats.
23+
Neither source presence nor focused passes complete KL-023; normal/scalar,
24+
recovery and full RF3/client gates, including failover/revocation, remain required.
25+
26+
No canonical data, existing alias/ID, journal, placement or acknowledgement
27+
changes occur. There is no data migration. Rollout adds explicit v1 capabilities;
28+
unknown versions fail closed. Rollback removes new read registration without
29+
rewriting acknowledged data. Weighted/cross-partition paths, full SQL/protocol
30+
and measured performance leadership remain separate mandatory workstreams.

‎docs/Features/ClusterRouting.md‎

Lines changed: 5 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -24,6 +24,11 @@ this discovery does not choose the still-proposed token format, migrate a store,
2424
change a public API or satisfy runtime acceptance. The join requires concrete
2525
current file/method evidence and an ordered, disjoint implementation graph.
2626

27+
The first concrete page stage is frozen in
28+
[PartitionTransfer](ClusterRouting/PartitionTransfer.md), TASK-PMOVE-PAGES and
29+
TASK-PMOVE-PAGE-ORACLES. Exact canonical pages retain an owned native cut;
30+
complete ownership, installation, token and RF3 gates remain explicit.
31+
2732
The owner2026-10-04 native CQRS/result/long-operation requirement is specified in
2833
[NativeCqrs](ClusterRouting/NativeCqrs.md) and [ADR-082](../ADR/ADR-082-native-cqrs-streams.md).
2934
Only the real Graph/native-enumeration compatibility test stage is accepted for

‎docs/Features/ClusterRouting/NativeCqrsRequestV2.md‎

Lines changed: 38 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -750,3 +750,41 @@ RF3 status capture, positive-control/cleanup ordering and the parent-call join.
750750
Workers must not edit the checkout or start builds/tests/processes/Git; deliver
751751
complete private source, patch, hashes and self-review. Root reads every byte,
752752
integrates, runs the gates and commits the complete stage scope.
753+
754+
## Accepted C1 native resource-log lifecycle, 2026-10-05
755+
756+
REQ-CRS-DIAG-003 / AC-CRS-DIAG-003 extends DIAG-001/002: initialize the actual
757+
native subscriber observer, start the three original resource watchers and
758+
observe subscriber admission for all three actual RF3 resources before
759+
`DistributedApplication.StartAsync`. The real malformed MCP call retains its
760+
400 Validation assertion, then awaits the original node1
761+
`BodyMethodMismatch/ToolsCall` record under the existing wave cancellation
762+
deadline before the healthy SDK/official MCP follow-up. Use an event-driven
763+
bounded waiter; no delay, fabricated record or additional raw log retention.
764+
Keep the unchanged parser, V1 artifact, closed enums, 32-record-per-node and
765+
16-KiB artifact caps. Stream completion before the expected record is a visible
766+
failure. This lifecycle repair is a test-infrastructure stage, not evidence
767+
that the original empty captures had one proven cause.
768+
769+
After actual AppHost stop, complete the three native resource log streams and
770+
drain/join their original watcher tasks before writing either success or
771+
failure evidence. Cancellation is only a bounded cleanup fallback after
772+
completion/drain failure; it cannot replace an original task join. Preserve
773+
stop, observer, completion, watcher, cancellation, disposal, artifact and fatal
774+
failures with `ServerFailureObserver`; `SaveEvidence` remains fail-closed until
775+
the original joins have completed. Caller cancellation must not become a
776+
successful capture or erase its primary failure.
777+
778+
TASK-CRS-DIAG-DRAIN stages: freeze this contract, privately implement subscriber
779+
admission in `RequestCqrsRf3WaveStartup.cs`, bounded live-record observation
780+
and native completion/drain in `RequestCqrsRf3Diagnostics.cs`, the thin wave
781+
join in `RequestCqrsRf3Wave.cs`, and the real-call wait in
782+
`RequestCqrsRf3McpGuardEvidenceScenario.cs`. Keep these under IntegrationTests
783+
`Features/ClusterRouting/Helpers/`; additional populated Helpers files may
784+
separate watcher/waiter responsibilities without adding a second collector.
785+
Reuse `RequestCqrsRf3DiagnosticsSubscriberObserver` and the pinned Aspire
786+
13.6.0 native APIs. Root owns docs, packet review/join, strict build, format,
787+
Aspire mechanism cases and genuine exact-source Linux current-image RF3
788+
`AcCrsDiag002` acceptance. No product/public/persisted contract or dependency
789+
changes; rollback removes only the test lifecycle addition. ADR-082 remains
790+
Accepted until its complete qualification gates pass.
Lines changed: 82 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,82 @@
1+
# ClusterRouting: bounded partition record pages
2+
3+
Status: first implementation stage for KL-036/071/072; complete movement remains
4+
unqualified. Decisions: [ADR-016](../../ADR/ADR-016-atomic-physical-placement.md)
5+
and [ADR-017](../../ADR/ADR-017-migration-tokens.md). Initial serving remains RF3.
6+
7+
This stage reads exact canonical key/value bytes from an already owned committed
8+
`IKeyValueView`. The node-local caller retains that cut for all its pages. It
9+
changes no existing persisted format, public API, placement or writer authority.
10+
11+
| Requirement | Acceptance criterion and real oracle |
12+
|---|---|
13+
| REQ-PMOVE-001: use a closed source-owned partition-family inventory | AC-PMOVE-001: actual partition-key constructors are covered; every selected family uses `KeySpace.Partition` with the four complete partition components. Unknown families, invalid partitions and foreign continuation keys fail before copying. Global catalog, principal/API keys, global outcomes, physical watermarks and shared blob accounting are explicitly excluded. |
14+
| REQ-PMOVE-002: bound retained bytes, records and examined work independently | AC-PMOVE-002: real native ZoneTree pages cover empty/inclusive/one-over bounds, exact keys/values/order and charged lookahead. Retained bytes include every returned key/value buffer and the independent continuation buffer. Checked reservation precedes every copy, including continuation; an overlarge record or examined-byte exhaustion throws typed BudgetExceeded with no successful partial page. No full Scan, payload decode or whole-store materialization. |
15+
| REQ-PMOVE-003: preserve caller-owned cut, cancellation and identity | AC-PMOVE-003: paginated reads inside one actual native view reproduce exact selected records; equal key text across tenant/database/domain stays isolated. Cancellation before/during traversal settles the original call with no successful partial image. Reopen preserves bytes; grains acquire no file owners. |
16+
| REQ-PMOVE-004: a family page cannot stand in for a complete movable image | AC-PMOVE-004: no installer or public move endpoint exists until partition outcomes, shared catalog/authorization, blob accounting, retained cursor/transfer state and derived-index readiness have an accepted complete protocol. Later process-kill/fencing/RF3 SDK/MCP gates remain mandatory. |
17+
18+
The inventory covers documents/indexes/unique keys and epochs, graphs and
19+
adjacency, vectors/lineage/effects, samples/sequence/dedup/retention, blob heads/
20+
uploads/parts/metadata, every queue index/body/metadata/counter/inbox, recurring
21+
schedules/sagas/capacity, subscription state/window/completion/inbox, both remote
22+
transfer endpoints, event streams/topics/identities/sequence/feed/snapshots,
23+
outbox/heads/consumers/receipts and visibility epochs. Enumerate actual current
24+
constructors, including dynamic family arguments. There is no universal encoded
25+
partition prefix: the family precedes the four partition components.
26+
27+
## Frozen reader contract and ordered ownership
28+
29+
TASK-PMOVE-PAGES: root owns architecture, source inventory and joins. Luna
30+
implementation owns only new internal Core ClusterRouting Queries/Contracts/
31+
Validation files prefixed `PartitionRecord`. Exact contract:
32+
`PartitionRecordPageReader.Read(IKeyValueView view, PartitionRef partition,
33+
string family, int maxRecords, long maxRetainedBytes, long maxExaminedBytes,
34+
ReadOnlyMemory<byte> afterKey = default, CancellationToken cancellationToken = default)`
35+
returns an internal immutable `PartitionRecordPage` containing owned
36+
`ImmutableArray<KeyValueRecord> Records`, `bool HasMore`, `long RetainedBytes`,
37+
`long ExaminedBytes` and optional independently owned `ReadOnlyMemory<byte>`
38+
continuation. `PartitionRecordFamilies.All` is an immutable ordinal inventory.
39+
Use native VisitRange, charge its real observer before copying, verify the
40+
exclusive continuation belongs to this exact prefix and has canonical KeyCodec
41+
encoding, and preserve raw bytes. Validate the continuation's bounded length
42+
before decoding its key components; user value payloads are never decoded.
43+
Reserve and copy continuation independently from the last returned key; its
44+
bytes contribute to RetainedBytes and maxRetainedBytes. Native accounting and
45+
unexpected visitor stops fail closed.
46+
Positive bounds must be checked before traversal; overflowing counters fail
47+
closed. There is no serialization or inter-grain DTO in this first stage.
48+
49+
TASK-PMOVE-PAGE-ORACLES: an independent Luna worker owns only new UnitTests
50+
ClusterRouting Cases/Helpers prefixed `PartitionRecord`. Derive the criteria
51+
above against actual ZoneTree/TestDatabase primitives, without mocks, fake view,
52+
skips or weakened limits. Preserve callbacks' borrowed lifetime. Root reviews
53+
the complete private packets, then executes Aspire normal/scalar tests.
54+
55+
Complete ownership is a later root-owned stage. Current epoch7 outcome keys have
56+
no partition locator; their hash cannot recover one. Copying every global outcome
57+
or dropping outcomes is incorrect. Existing-store migration needs its exact
58+
accepted upgrade/rollback contract. Current persisted authorization remains
59+
authority and an acknowledged revocation must fence all serving groups. Source
60+
and destination group indexes are never directly comparable; explicit ownership
61+
epoch invalidation is the first candidate under KL-072, pending full freeze.
62+
63+
Slice map: Core owns this internal pure reader in ClusterRouting; Server's
64+
StorageRecovery retains native views/files and Replication retains ordered
65+
commit authority. Later Orleans orchestration uses one request grain and native
66+
ManagedCode.Communication asynchronous streams with bounded handoff. Public
67+
contracts, SDK/MCP, frontend/admin controls are N/A for this internal stage;
68+
their later actual move contract must be specified and qualified.
69+
70+
Verification: full strict Release build, formatter/governance, genuine Aspire
71+
unit and unit-scalar focused PartitionRecord cases for local development, then
72+
exact-source Linux CI. Later movement requires process cuts at every persisted
73+
state and Docker/Aspire RF3 SDK/MCP recovery, fencing, token and receipt oracles.
74+
Internal pages alone do not satisfy those movement acceptance criteria.
75+
76+
```mermaid
77+
flowchart LR
78+
Owner[Node local committed view] --> Family[Closed family and full partition prefix]
79+
Family --> Budget[Reserve examined and retained bytes]
80+
Budget --> Page[Exact owned bounded page]
81+
Page --> Later[Later complete ownership and fenced transfer]
82+
```

0 commit comments

Comments
 (0)