Skip to content

Commit fb586bf

Browse files
KSemenenkoKSemenenko
committed
Use owned packed similarity for ANN construction
Share the constrained traversal and preserve exact score bits, candidate order, work bounds and untrusted input validation. Join a dedicated real seed-cancel observer before disposing its owner-scoped synchronization signal and CTS. All current changes are included. Full Release build and format/governance pass. Actual Aspire Search194/194 passes in normal and scalar modes, including original 10000-record adaptive planner and cancellation controls. Original reports and frozen source/runtime inventories are retained. These are macOS development observations; full delivered-source Linux and ANN/RF3 qualification remain open. Freeze the next due fresh-attempt repair and held current-authority test stages.
1 parent 2044c53 commit fb586bf

21 files changed

Lines changed: 1021 additions & 89 deletions

‎docs/ADR/ADR-094-orleans-due-coordination.md‎

Lines changed: 12 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -58,3 +58,15 @@ sequenceDiagram
5858
Host-->>Coordinator: Native CQRS terminal outcome
5959
Coordinator-->>Service: Joined settlement or explicit failure
6060
```
61+
62+
Accepted TASK-DUE-FRESH-ATTEMPT repair,2026-10-05: a coordinator dispatch chooses
63+
one fresh command GUID after canonical barrier/creator reload, retaining it
64+
across its one unknown-result retry. Later sweeps get fresh IDs; the old internal
65+
v1 hint-to-command hash and obsolete hash-only test are removed. This fixes the
66+
original durable-denial replay trap without changing command outcome caching,
67+
canonical occurrence IDs, monotonic schedule state, saga revision CAS, serializer
68+
aliases/Ids, reader epochs or stored data. Exact scope, regression oracles, agent
69+
roles, ordered integration, rollout and remaining RF3 gates are in the accepted
70+
DueCoordination implementation contract. A caller retry of an actual existing
71+
command ID still replays that original terminal receipt. No automatic migration
72+
or reconstruction of historical command IDs is performed.

‎docs/Features/ClusterRouting/NativeCqrsRequestV2.md‎

Lines changed: 27 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -656,3 +656,30 @@ ADR-034/058 and the preceding accepted C1 control implementation contract alread
656656
cover these private controls and public CQRS behavior. No public API, canonical
657657
format, credential contract or deployment topology changes; rollback removes
658658
the test entry points/private controls without migrating canonical data.
659+
660+
### Accepted C1 held current-authority cases, 2026-10-05
661+
662+
TASK-CRS-C1-HELD-AUTHORITY implements two real RF3 write cases, SDK and official
663+
MCP, for AC-CRS-004/005. Reuse the exact accepted fresh-wave, private control,
664+
public persisted non-admin identity, signed marker, client, receipt and original
665+
cleanup contracts of TASK-CRS-C1-PUBLIC-INTERRUPTION. Arm Hold at the actual
666+
AuthorizationReload phase, which precedes GrainRequestAuthority.Reload. Await
667+
its signed physical voter/request marker. While that original request is held,
668+
revoke its persisted principal through a separate real administrator command
669+
and independently observe the revocation ACK and unchanged document. Release
670+
the exact original request without cancelling it. Its actual public SDK/MCP
671+
result must be the actual typed Unauthenticated, with no committed document/revision, outcome or
672+
outbox effect; no trusted caller role may override that result. Await genuine
673+
hold settlement and ProducerDisposed, join the original caller, and retire the
674+
arm across all voters. A fresh genuine administrator command with a new ID must
675+
then commit successfully; the revoked original principal remains rejected.
676+
677+
lifecycle_wave Luna/high owns only new IntegrationTests ClusterRouting
678+
Cases/Helpers/Models prefixed RequestCqrsAuthorityFault, emitted as private
679+
exact-base source and patch with complete hashes and self-review. Root owns
680+
shared StartProbedAsync and all runtime/Git joins. Do not duplicate controls or
681+
clients, edit production/shared sources, weaken bounds, synthesize replies,
682+
claim log privacy from filtered diagnostics or report remote-voter death.
683+
Root runs the two actual Aspire RF3 cases and required exact-source Linux gates;
684+
source alone cannot close these criteria. Existing ADR-034/058 and C1 control
685+
contracts cover this test surface without public/format/topology changes.

‎docs/Features/Messaging/DueCoordination.md‎

Lines changed: 39 additions & 7 deletions
Original file line numberDiff line numberDiff line change
@@ -146,13 +146,16 @@ write authority and does not call DatabaseEngine.Apply or SubmitNativeAsync.
146146

147147
The coordinator creates one Batch containing EmitRecurringOccurrences(max1) or
148148
ExpireSaga(expected revision), preserving full identity and current creator.
149-
Its command identity is SHA256 over the existing canonical KeyCodec encoding of
150-
`keyload-due-command-v1`, kind as Int64 (schedule0, saga1), full tenant/database/
151-
transaction-domain/partition/queue, item GUID in N format, revision, generation
152-
and ordinal. The GUID uses the first16 hash bytes; if empty, use the last16.
153-
Saga hints pin generation0 and ordinal0. This identity remains independent of
154-
request GUIDs; a public retry with the same creator, mutation and command ID
155-
converges with a service attempt under the existing canonical receipt contract.
149+
Each newly admitted ProcessDueAsync dispatch owns a fresh nonempty command GUID,
150+
chosen once after the real barrier and persisted creator reload. Keep that GUID
151+
and payload unchanged across its one uncertainty retry; each native request still
152+
uses a fresh request GUID. Later scans and post-restart dispatches are new attempts
153+
and use new command GUIDs, including after a definite denied or not-yet-due result.
154+
The canonical schedule generation, logged due time, monotonically advanced ordinal
155+
and occurrence message identity, and saga Waiting/revision CAS remain the durable
156+
effect authority. An identical external retry using the actual service-attempt
157+
command GUID, creator and body keeps the existing canonical receipt semantics.
158+
No public deterministic command-ID derivation from a hint is promised.
156159
For each attempt it creates a fresh IRequestGrain GUID, sets the existing bounded
157160
native request state and subject-only claims, signs via GrainRequestCodec, and
158161
drains the existing native CQRS stream. One5second dispatch deadline includes
@@ -263,3 +266,32 @@ flowchart LR
263266
Request --> Apply[Current authorization and logged-time RF3 apply]
264267
Apply --> Canonical[Atomic watermark or Waiting CAS plus enqueue]
265268
```
269+
270+
### Accepted definite-outcome retry repair, 2026-10-05
271+
272+
TASK-DUE-FRESH-ATTEMPT maps REQ/AC-DUE-002..004 to the concrete original CB5
273+
recurring revocation/restoration failure. The former deterministic v1 hint hash
274+
reused a durable PermissionDenied forever after authority restoration. It also
275+
retained a successful no-op if leader-logged time preceded the hinted due instant.
276+
Replace only that active internal attempt selection with one fresh Guid.NewGuid
277+
per newly admitted coordinator dispatch. Remove its obsolete deterministic
278+
helper and hash-only unit case; do not retain a runtime compatibility dispatcher.
279+
The existing original command outcomes and frozen canonical entity/occurrence
280+
digests are unchanged. No stored record, alias, Id, public mutation, codec or
281+
reader-format version changes. Old command receipts remain addressable by their
282+
original IDs through unchanged replay; no live store is rewritten.
283+
284+
cluster_wave Luna/high owns only the coordinator's one command-ID selection,
285+
removal of DueWorkCommandIdentity and its obsolete test, and new Messaging
286+
Cases/Helpers demonstrating on actual ZoneTree that a denied original command
287+
remains durably denied after current persisted grant restoration while a fresh
288+
command emits one exact canonical occurrence. Include a repeated original/fresh
289+
retry and original schedule/queue/outbox state oracle; do not weaken outcome
290+
caching or call direct Apply from Orleans. Root owns the docs/shared joins and
291+
actual Aspire gates. The unchanged full RF3 recurring autonomous case must pass
292+
without manual Emit, service disabling, fake time, threshold or wait increases.
293+
Real lost-ACK/native service retry, no-quorum/restart and complete Linux evidence
294+
remain mandatory separately; source or Core regression alone does not close S2.
295+
Rollout is homogeneous current capable nodes, retaining all original outcomes,
296+
watermarks and native recovery journals. Rollback of this repair reintroduces the
297+
denied-result retry defect and cannot be described as restoring correctness.

‎docs/implementation/ann-construction-development-2026-10-05.json‎

Lines changed: 232 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,232 @@
1+
{
2+
"schemaVersion": 1,
3+
"kind": "local-development-stage",
4+
"stage": "TASK-ANN-R1-CONSTRUCTION-VALUE-and-TASK-ASD-CANCELLATION-OBSERVER",
5+
"qualified": false,
6+
"acceptanceClosed": false,
7+
"source": {
8+
"gitHead": "2044c53cb70b4343884f106931d7b7c640657e14",
9+
"workingTree": "Exact frozen reviewed source/runtime inventories; HEAD alone is not compiled-source identity"
10+
},
11+
"requirements": [
12+
"docs/Features/Search/ManagedAnn.md",
13+
"docs/Features/Search/ManagedAnnSeed.md"
14+
],
15+
"decision": "docs/ADR/ADR-019-managed-ann.md",
16+
"implementation": [
17+
"One constrained packed-similarity traversal shares loops, ordering and work charges across prepared class and owned readonly value",
18+
"ANN insertion selects owned value without per-row prepared class allocation or repeated corpus preparation; raw/untrusted validation remains",
19+
"Dedicated native cancellation observer starts before real synchronous capture, observes charged bytes, retains cancellation/failures and joins its original Thread before owner-scoped signal/CTS disposal"
20+
],
21+
"caller": "dotnet run --project src/KeyLoad.AppHost --no-build --no-restore --configuration Release -- --KeyLoadTests:Suite=<unit|unit-scalar> --KeyLoadTests:Filter=/*/KeyLoad.UnitTests.Features.Search/*/*",
22+
"machine": {
23+
"machineName": "konstantins-macbook-pro",
24+
"operatingSystem": "macOS 27.0.1",
25+
"runtimeVersion": ".NET 10.0.12",
26+
"tunitVersion": "1.72.10.0"
27+
},
28+
"cohorts": [
29+
{
30+
"suite": "unit",
31+
"nativeExit": 0,
32+
"report": "TestResults/search78c-normal-20261005/KeyLoad.UnitTests-macos-net10.0.tunit-report.json",
33+
"sha256": "ee06683ec2ecdb3d57d39c64196af09585aa8e7226fc24040594e53984672432",
34+
"summary": {
35+
"total": 194,
36+
"passed": 194,
37+
"failed": 0,
38+
"skipped": 0,
39+
"cancelled": 0,
40+
"timedOut": 0,
41+
"flaky": 0
42+
},
43+
"durationMs": 46305.236,
44+
"selectedClasses": [
45+
{
46+
"class": "AdaptiveFilteredPlannerTests",
47+
"summary": {
48+
"total": 6,
49+
"passed": 6,
50+
"failed": 0,
51+
"skipped": 0,
52+
"cancelled": 0,
53+
"timedOut": 0,
54+
"flaky": 0
55+
},
56+
"testNames": [
57+
"AcFilter004AdaptiveWalkMayNavigateIneligibleVerticesToAnIsolatedEligibleEndpoint",
58+
"AcFilter004PlannerUsesCeilingBreadthAndCapsWithCheckedCorpusArithmetic",
59+
"AcFilter004SelectivityPlanAndPackedSearchMatchIndependentScalarCohortOracles",
60+
"AcFilter004CanceledPlannerLeavesTheNextPackedQueryHealthy",
61+
"AcFilter004SmallAndEmptyBitmapsEnumerateOnlyAdmittedRows",
62+
"AcFilter004InsufficientAdaptiveCandidatesUseChargedExactFallbackThenHealthySearch"
63+
]
64+
},
65+
{
66+
"class": "PackedAnnConstructionValueTests",
67+
"summary": {
68+
"total": 4,
69+
"passed": 4,
70+
"failed": 0,
71+
"skipped": 0,
72+
"cancelled": 0,
73+
"timedOut": 0,
74+
"flaky": 0
75+
},
76+
"testNames": [
77+
"AcAnn011ConstrainedScoreKeepsTypedOrdinalValidation",
78+
"AcAnn011ZeroDotProductsKeepExactOrdinalTieOrder",
79+
"AcAnn011OwnedAndClassTraversalMatchOnPersistedGraphsForEveryMetric",
80+
"AcAnn011WarmedOwnedTraversalAllocatesZeroBytesOnTheCallingThread"
81+
]
82+
},
83+
{
84+
"class": "AnnSeedCancellationTests",
85+
"summary": {
86+
"total": 2,
87+
"passed": 2,
88+
"failed": 0,
89+
"skipped": 0,
90+
"cancelled": 0,
91+
"timedOut": 0,
92+
"flaky": 0
93+
},
94+
"testNames": [
95+
"AlreadyCanceledBudgetRejectsBeforeSeedPublication",
96+
"InProgressChargedCaptureCancelsAndFollowingCaptureRemainsHealthy"
97+
]
98+
}
99+
],
100+
"inputInventory": {
101+
"head": "2044c53cb70b4343884f106931d7b7c640657e14",
102+
"source": {
103+
"count": 3521,
104+
"sha256": "df06e5b1c9493f592eb1c26b0ca8c0dfb2c5ab59d8522425ab413a30bd79f821"
105+
},
106+
"runtime": {
107+
"count": 544,
108+
"sha256": "0644ec147fe80e087fa8a89b3cf625dc646d8199bd49415f0674ef5e9de18b2b"
109+
},
110+
"beforeSha256": "767b52dd2950f8717c07e81e5c40ed918ec9a12c391c8b47cbc66bb9b8d678ee",
111+
"afterSha256": "7dae3b8d60305549cc5bd20873bdd5e870993d701453a102ce8c9c23e8e685a7",
112+
"unchanged": true
113+
}
114+
},
115+
{
116+
"suite": "unit-scalar",
117+
"nativeExit": 0,
118+
"report": "TestResults/search78c-scalar-20261005/KeyLoad.UnitTests-macos-net10.0.tunit-report.json",
119+
"sha256": "f70102bb1a6bfd8a7c5e66569b67379b1c0efcec711bb4458d88e31f87ca5186",
120+
"summary": {
121+
"total": 194,
122+
"passed": 194,
123+
"failed": 0,
124+
"skipped": 0,
125+
"cancelled": 0,
126+
"timedOut": 0,
127+
"flaky": 0
128+
},
129+
"durationMs": 63259.734,
130+
"selectedClasses": [
131+
{
132+
"class": "AdaptiveFilteredPlannerTests",
133+
"summary": {
134+
"total": 6,
135+
"passed": 6,
136+
"failed": 0,
137+
"skipped": 0,
138+
"cancelled": 0,
139+
"timedOut": 0,
140+
"flaky": 0
141+
},
142+
"testNames": [
143+
"AcFilter004CanceledPlannerLeavesTheNextPackedQueryHealthy",
144+
"AcFilter004InsufficientAdaptiveCandidatesUseChargedExactFallbackThenHealthySearch",
145+
"AcFilter004PlannerUsesCeilingBreadthAndCapsWithCheckedCorpusArithmetic",
146+
"AcFilter004AdaptiveWalkMayNavigateIneligibleVerticesToAnIsolatedEligibleEndpoint",
147+
"AcFilter004SelectivityPlanAndPackedSearchMatchIndependentScalarCohortOracles",
148+
"AcFilter004SmallAndEmptyBitmapsEnumerateOnlyAdmittedRows"
149+
]
150+
},
151+
{
152+
"class": "PackedAnnConstructionValueTests",
153+
"summary": {
154+
"total": 4,
155+
"passed": 4,
156+
"failed": 0,
157+
"skipped": 0,
158+
"cancelled": 0,
159+
"timedOut": 0,
160+
"flaky": 0
161+
},
162+
"testNames": [
163+
"AcAnn011ConstrainedScoreKeepsTypedOrdinalValidation",
164+
"AcAnn011ZeroDotProductsKeepExactOrdinalTieOrder",
165+
"AcAnn011OwnedAndClassTraversalMatchOnPersistedGraphsForEveryMetric",
166+
"AcAnn011WarmedOwnedTraversalAllocatesZeroBytesOnTheCallingThread"
167+
]
168+
},
169+
{
170+
"class": "AnnSeedCancellationTests",
171+
"summary": {
172+
"total": 2,
173+
"passed": 2,
174+
"failed": 0,
175+
"skipped": 0,
176+
"cancelled": 0,
177+
"timedOut": 0,
178+
"flaky": 0
179+
},
180+
"testNames": [
181+
"AlreadyCanceledBudgetRejectsBeforeSeedPublication",
182+
"InProgressChargedCaptureCancelsAndFollowingCaptureRemainsHealthy"
183+
]
184+
}
185+
],
186+
"inputInventory": {
187+
"head": "2044c53cb70b4343884f106931d7b7c640657e14",
188+
"source": {
189+
"count": 3521,
190+
"sha256": "df06e5b1c9493f592eb1c26b0ca8c0dfb2c5ab59d8522425ab413a30bd79f821"
191+
},
192+
"runtime": {
193+
"count": 544,
194+
"sha256": "0644ec147fe80e087fa8a89b3cf625dc646d8199bd49415f0674ef5e9de18b2b"
195+
},
196+
"beforeSha256": "0ff8b9fc18659e928e2604b865e3a51658c19c0629725ce713990de3045a8664",
197+
"afterSha256": "5bf7e1e76ae104750ba77b4dbbd612a7de645b636474c745099757deecf02af4",
198+
"unchanged": true
199+
}
200+
}
201+
],
202+
"build": {
203+
"nativeExit": 0,
204+
"warnings": 0,
205+
"errors": 0,
206+
"elapsedSeconds": 50.43
207+
},
208+
"formatter": {
209+
"nativeExit": 0,
210+
"command": "dotnet format KeyLoad.slnx --verify-no-changes --no-restore"
211+
},
212+
"governance": {
213+
"nativeExit": 0,
214+
"scope": "Static inventory only"
215+
},
216+
"retainedOriginals": {
217+
"manifest": "artifacts/qualification/ann-construction78c-20261005/manifest.json",
218+
"sha256": "3df1f294405777d932a32f3d9c6b64744eede1c90372f2564a22a0d2df880c7c"
219+
},
220+
"proof": [
221+
"Exact candidate IDs/order and double bits against prepared class and independent scalar score oracle for cosine/euclidean/dot-product on real persisted graphs",
222+
"Warmed complete owned traversal allocates exactly zero bytes on the calling thread for all three metrics",
223+
"All six unchanged adaptive planner cases, including both real10000-record deadline cases, pass in normal and scalar modes",
224+
"Both real seed cancellation controls pass; positive read charge/cancellation witness/no returned seed/following10000 capture are asserted"
225+
],
226+
"limitations": [
227+
"Search194 cases are a bounded development slice; not a passing full Unit/scalar/recovery/RF3 qualification",
228+
"Prior CB5 Linux full scalar3 failures remain failed original evidence; exact delivered-source full Linux rerun is required",
229+
"No comparable speedup, global allocation/memory improvement, production/endurance or public performance claim",
230+
"Durable ANN generations, online projection publication, public approximate search and their RF3 lifecycle gates remain open"
231+
]
232+
}

‎docs/implementation/status.json‎

Lines changed: 25 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -1133,7 +1133,8 @@
11331133
"tests/KeyLoad.UnitTests/Features/Search/Cases/AnnSeedReservationTests.cs",
11341134
"docs/implementation/runtime-qualification-37249197752.json",
11351135
"docs/implementation/ann-prepared-value-development-2026-10-05.json",
1136-
"docs/implementation/command-outcome-pin-development-2026-10-05.json"
1136+
"docs/implementation/command-outcome-pin-development-2026-10-05.json",
1137+
"docs/implementation/ann-construction-development-2026-10-05.json"
11371138
],
11381139
"sourceStage": "R2A_bounded_historical_seed_9_source_12_tests_34_cases_full_static23_and_unchanged_actual_Aspire_normal34_scalar34_local_pass_persisted_projection_replay_and_delivered_Linux_RF3_pending",
11391140
"runtimeQualified": false,
@@ -1155,6 +1156,16 @@
11551156
"receipt": "docs/implementation/command-outcome-pin-development-2026-10-05.json",
11561157
"durableGenerationAndOnlineProjection": "pending",
11571158
"qualified": false
1159+
},
1160+
"constructionValueDevelopment": {
1161+
"normal": "194_of194_Search_slice",
1162+
"scalar": "194_of194_Search_slice",
1163+
"newConstructionCases": "4_of4_each_mode",
1164+
"seedCancellation": "2_of2_each_mode",
1165+
"adaptivePlanner": "6_of6_each_mode",
1166+
"receipt": "docs/implementation/ann-construction-development-2026-10-05.json",
1167+
"qualified": false,
1168+
"acceptanceClosed": false
11581169
}
11591170
},
11601171
"KL-032": {
@@ -1884,15 +1895,26 @@
18841895
"tests/KeyLoad.UnitTests/Features/Search/Cases/VectorMetricValidationTests.cs",
18851896
"tests/KeyLoad.UnitTests/Features/Search/Cases/VectorMetricGoldenTests.cs",
18861897
"docs/implementation/performance-source-r11.json",
1887-
"docs/implementation/keycodec-crud-development-2026-10-04.json"
1898+
"docs/implementation/keycodec-crud-development-2026-10-04.json",
1899+
"docs/implementation/ann-construction-development-2026-10-05.json"
18881900
],
18891901
"simd": {
18901902
"ownerDirection": "confirmed_SIMD_dotNET_intrinsics_first_Rust_after_profiling",
18911903
"stage": "portable_Vector_finite_validation_only_existing_scoring_accumulation_unchanged",
18921904
"baseline": "five_public_metric_and_actual_store_tests_pass_each_OS_exactc486",
1893-
"candidate": "full_hardware_disabled_unit_suite_added_to_all3_OS_CI_own_SHA_execution_pending",
1905+
"candidate": "Linux_only_full_hardware_disabled_suite_CB5_failed3; current_local_Search194_passes; full_delivered_source_Linux_pending",
18941906
"measuredGain": "not_qualified",
18951907
"rust": "no_native_bridge_without_profile_evidence"
1908+
},
1909+
"constructionValueDevelopment": {
1910+
"normal": "194_of194_Search_slice",
1911+
"scalar": "194_of194_Search_slice",
1912+
"newConstructionCases": "4_of4_each_mode",
1913+
"seedCancellation": "2_of2_each_mode",
1914+
"adaptivePlanner": "6_of6_each_mode",
1915+
"receipt": "docs/implementation/ann-construction-development-2026-10-05.json",
1916+
"qualified": false,
1917+
"acceptanceClosed": false
18961918
}
18971919
},
18981920
"KL-078": {

‎src/KeyLoad.Query/Features/Search/Contracts/IPackedAnnSimilarity.cs‎

Lines changed: 6 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,6 @@
1+
namespace KeyLoad.Query.Features.Search;
2+
3+
internal interface IPackedAnnSimilarity
4+
{
5+
double ScorePacked(PackedAnnVectors vectors, int ordinal);
6+
}

0 commit comments

Comments
 (0)