Skip to content

Commit 5bbf30f

Browse files
committed
Checkpoint cache control and intensive comparison implementation
Commit all current eligible source, tests, documentation and CI configuration on main as requested. Include signed cache metadata primitives, image HTTP ownership, native comparison repairs and KeyLoad/Timescale intensive adapters. This is a work-in-progress checkpoint. The enabled R101 full development build failed 53 diagnostics; Node and Mongo corrections, final formatter and new-SHA native qualification remain open. Existing native results and source-only review are recorded separately, with no performance or readiness claim.
1 parent d45d7f2 commit 5bbf30f

186 files changed

Lines changed: 20555 additions & 286 deletions

File tree

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

‎.github/workflows/ci.yml‎

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -193,6 +193,10 @@ jobs:
193193
- name: Qualify exact native topology composition models
194194
run: |
195195
dotnet test --project tests/KeyLoad.ComparisonTests --no-build --no-restore --configuration Release --treenode-filter '/*/*/Isolated*Resource*/*'
196+
dotnet test --project tests/KeyLoad.ComparisonTests --no-build --no-restore --configuration Release --treenode-filter '/*/*/MongoReadinessResourceTests/*'
197+
dotnet test --project tests/KeyLoad.ComparisonTests --no-build --no-restore --configuration Release --treenode-filter '/*/*/MongoReadinessIdentityTests/*'
198+
dotnet test --project tests/KeyLoad.ComparisonTests --no-build --no-restore --configuration Release --treenode-filter '/*/*/MongoReadinessTaskFailureTests/*'
199+
dotnet test --project tests/KeyLoad.ComparisonTests --no-build --no-restore --configuration Release --treenode-filter '/*/*/IsolatedKurrentVolumeRegressionFailureTests/*'
196200
dotnet test --project tests/KeyLoad.ComparisonTests --no-build --no-restore --configuration Release --treenode-filter '/*/*/IsolatedKeyLoadAdmissionTests/*'
197201
dotnet test --project tests/KeyLoad.ComparisonTests --no-build --no-restore --configuration Release --treenode-filter '/*/*/ComparisonResourceLogBufferTests/*'
198202
- name: Qualify isolated replay admission and native diagnostics

‎README.md‎

Lines changed: 5 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -249,7 +249,7 @@ Tests use TUnit and Microsoft.Testing.Platform and execute in GitHub Actions. De
249249

250250
The root `.editorconfig` is copied directly from Prostir. Every project enables SDK/static/style analysis with warnings as errors. Edit custom rules under `src/KeyLoad.Analyzers/Features/CodeQuality/` and add real-compilation cases under `tests/KeyLoad.Analyzers.Tests/Features/CodeQuality/`. They attach centrally to consumer projects and report in the IDE and build. Compiler SARIF 2.1 reports live under `artifacts/code-quality/<project>/<configuration>/<framework>/diagnostics.sarif`; CI retains them even when the build fails. See [CodeQuality](docs/Features/CodeQuality.md) for the rule catalog, authoring and applicability, and [current evidence](docs/implementation/code-quality.md) for actual gates.
251251

252-
The [runtime ledger](docs/implementation/runtime-qualification-20261002.md) preserves exact GitHub source SHAs, native suite counts, failures and artifact hashes; [AdminDashboard](docs/Features/AdminDashboard.md) records its separately verified RF3 and browser evidence. The latest joined 06a9 run has unit and image-preparation failures; RF3 and measured comparison stages did not run. Historical successful cases do not qualify newer source. Environmental failure branches, coverage, activation movement, server-resource/performance, endurance and power-loss qualification remain open.
252+
The [runtime ledger](docs/implementation/runtime-qualification-20261002.md) preserves exact GitHub source SHAs, native suite counts, failures and artifact hashes; [AdminDashboard](docs/Features/AdminDashboard.md) records its separately verified RF3 and browser evidence. The earlier joined [397a run](docs/implementation/runtime-qualification-37097831105.json) fails17 unit cases; RF3, recovery and analyzer tests pass, while scalar and measured comparisons are skipped. Historical successful cases do not qualify newer source. Environmental failure branches, coverage, activation movement, server-resource/performance, endurance and power-loss qualification remain open.
253253

254254
The comparison library/sole CLI host split under [ADR-043](docs/ADR/ADR-043-comparison-library-host.md), immutable harness under [ADR-044](docs/ADR/ADR-044-benchmark-immutable-contracts.md), owned PostgreSQL schema repair under [ADR-045](docs/ADR/ADR-045-postgres-schema-ownership.md), private storage owners under [ADR-046](docs/ADR/ADR-046-storage-private-owners.md) and genuine BenchmarkDotNet library under [ADR-047](docs/ADR/ADR-047-embedded-benchmark-host.md) are source joins awaiting full runtime qualification. The read-only product contract migration preserves JSON/base64 and fingerprints under [ADR-041](docs/ADR/ADR-041-read-only-public-collections.md). Exact-SHA run [37015193756](https://github.com/managedcode/KeyLoad/actions/runs/37015193756) passed all88 analyzer cases and the enabled solution builds; subsequent repairs require renewed verification. Compatible coverage collection, container export and its numeric baseline remain pending. Historical source stages are retained in the [code-quality record](docs/implementation/code-quality.md); the [runtime ledger](docs/implementation/runtime-qualification-20261002.md) records the latest completed candidate run.
255255

@@ -300,9 +300,12 @@ No new intensive performance or live publication result is claimed.
300300

301301
The earlier isolated Linux baseline ([run37087909605](https://github.com/managedcode/KeyLoad/actions/runs/37087909605), source `b47409e`) passes source, unit, recovery and RF3 gates. Ten of27 native1/2/3 preflights fail, so the complete270-cell cohort and new performance publication remain blocked. Read/control admission, native diagnostics, document oracle and pinned gossip identity repairs are awaiting exact-source GitHub qualification; the separate intensive TimeSeries corpus/oracle source does not yet implement its native runner/adapters.
302302

303-
The bounded RAM pool, explicit embedded point cache, local coordinated binding, actual-owner observation and test-fixture repairs passed the development build, formatter and static governance checks recorded in the [R83 source receipt](docs/implementation/cache-memory-source-r83.json). The preceding [exact-source native run](docs/implementation/runtime-qualification-37093992229.json) failed:1522/1539 normal unit cases and62/63 RF3 cases passed; scalar was skipped, while164 recovery and118 analyzer cases passed. The17 unit failures are test-side null-key projection and byte/int assertion defects; RF3 failed an election-window queue receive without its required same-ID retry. Their source repairs require a new delivered SHA and complete native gates. The [signed metadata contract](docs/Features/ResourceExecution/CacheControlV1.md) is accepted for unused primitives only; Orleans RF3 controls, coverage and measured benefit remain pending, and server RF3 caches are disabled.
303+
The bounded RAM pool, embedded point cache and local binding passed the development checks in the [R83 source receipt](docs/implementation/cache-memory-source-r83.json). The newer [d45 native source gates](docs/implementation/cache-native-source-r93.json) pass1583/1583 normal and scalar unit cases with identical IDs and no skips,164/164 recovery plus1000 seeded process-cut rows,118 analyzer cases and63/63 SDK/MCP RF3 cases. All17 earlier identity failures and the byte77 oracle now pass. The comparison workflow concluded with24/27 successful preflight jobs and3 failures; the270-cell matrix did not execute. These source gates provide no performance result. The [R100 wire source receipt](docs/implementation/cache-wire-source-r100.json) binds the complete independent review of34 unused internal signed-metadata files and26 corrected test files. Fresh integrated source gates and delivered-SHA native qualification remain pending. Orleans RF3 controls, product coverage, measured benefit, endurance and power-loss gates remain open; server RF3 caches are disabled.
304304

305305

306306
The delivered `2f374fc34` [Linux run37093197474](https://github.com/managedcode/KeyLoad/actions/runs/37093197474) now passes the complete source gates,118 analyzer cases,1483 normal and1483 scalar unit cases,164 process-recovery cases and63 Docker RF3 cases without skips. The [original-source receipt](docs/implementation/isolated-source-qualification-37093197474.json) retains exact report/artifact hashes. Comparative image and diagnostic gates pass;27 native preflights finish16 job successes and11 failures, including two explicit unavailable Neo4j topologies among the successes. Complete270 performance and site publication remain unqualified. The separate TimeSeries pure30 cases pass in both modes, while its bounded runner has36 new declared TUnit cases prepared in source and native adapters/all30 cells remain pending.
307307

308308
The later `2ecbeee4d` [run37093992229](https://github.com/managedcode/KeyLoad/actions/runs/37093992229) fails17 normal unit cases and one RF3 queue-receive case; scalar and native comparison jobs are skipped. The [original-failure receipt](docs/implementation/isolated-current-main-baseline-37093992229.json) preserves the exact reports and artifact hashes. Current repairs and a prepared intensive TimeSeries runner remain unqualified until their delivered-source GitHub gates pass.
309+
310+
311+
The fixture-only `d45d7f253` [Linux source qualification](docs/implementation/isolated-source-qualification-37098964980.json) passes normal1583/scalar1583, recovery164, analyzer118 and RF363 without skips; all17 prior fixture failures pass in both modes. The [terminal comparison receipt](docs/implementation/isolated-comparison-terminal-37098964980.json) records24successful native preflight jobs and3failures (Mongo2 image import, Mongo3 priority takeover, Kurrent2 cleanup). Two successful jobs explicitly report unavailable Neo4j Community topologies. Full270 was not allocated; aggregation and refreshed site metrics remain unqualified. New ACK ownership, SDK TimeSeries adapter, SQL and transactional lifecycle source require their own delivered-SHA qualification; TimeSeries6/30 remains incomplete.

‎benchmarks/KeyLoad.Comparisons/Features/BenchmarkComparisons/KurrentClusterVerifier.cs‎

Lines changed: 3 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -35,9 +35,10 @@ public static async Task<KurrentClusterProof> VerifyAsync(string connectionStrin
3535
}
3636

3737
public static Task<ClusterEvidence> VerifyCopyAsync(KurrentDBClient writer, KurrentDBClient[] nodeClients,
38-
HttpClient[] httpClients, ComparisonTopology topology, string stream, KurrentEventData eventData, TimeSpan timeout,
38+
HttpClient[] httpClients, ComparisonTopology topology, string stream, KurrentEventData eventData,
39+
KurrentStreamOwnership ownership, TimeSpan timeout,
3940
CancellationToken cancellationToken)
40-
=> KurrentReplicaProbe.VerifyCopyAsync(writer, nodeClients, httpClients, topology, stream, eventData,
41+
=> KurrentReplicaProbe.VerifyCopyAsync(writer, nodeClients, httpClients, topology, stream, eventData, ownership,
4142
timeout, cancellationToken);
4243

4344
internal static async Task<KurrentGossipView[]> ReadReadyViewsAsync(HttpClient[] clients,

‎benchmarks/KeyLoad.Comparisons/Features/BenchmarkComparisons/KurrentConstants.cs‎

Lines changed: 8 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -94,11 +94,17 @@ internal static class KurrentConstants
9494
public const int ReadLimit = 2;
9595
public const ulong NativeFirstRevision = 0;
9696
public const int CanonicalFirstRevision = 1;
97-
public const int CleanupTimeoutSeconds = 20;
98-
public const int CleanupHostTimeoutSeconds = 30;
97+
public const int CleanupTimeoutSeconds = 120;
98+
public const int CleanupHostTimeoutSeconds = 180;
9999
public const int CleanupConcurrency = 16;
100100
public const string CleanupIncomplete = "KurrentOwnedStreamCleanupIncomplete";
101101
public const string CleanupDrainFailed = "KurrentOwnedStreamCleanupDrainFailed";
102102
public const string CleanupInvalidDiagnostic = "KurrentOwnedStreamCleanupInvalidDiagnostic";
103+
public const int OwnershipProbeCount = 2;
104+
public const string OwnershipInvalidDescriptor = "KurrentStreamOwnershipInvalidDescriptor";
105+
public const string OwnershipReservationRejected = "KurrentStreamOwnershipReservationRejected";
106+
public const string OwnershipTransitionRejected = "KurrentStreamOwnershipTransitionRejected";
107+
public const string OwnershipAlreadyInitialized = "KurrentStreamOwnershipAlreadyInitialized";
108+
public const string AppendAcknowledgementMissing = "KurrentStreamAppendAcknowledgementMissing";
103109
public const int ProbeWaitMilliseconds = 100;
104110
}
Lines changed: 39 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,39 @@
1+
using KurrentDB.Client;
2+
using KurrentEventData = KurrentDB.Client.EventData;
3+
4+
namespace KeyLoad.Comparisons.Targets;
5+
6+
internal static class KurrentOwnedStreamAppend
7+
{
8+
internal static async Task<IWriteResult> AppendAsync(KurrentDBClient writer, KurrentStreamOwnership ownership,
9+
string stream, KurrentEventData eventData, CancellationToken token)
10+
{
11+
ArgumentNullException.ThrowIfNull(writer);
12+
ArgumentNullException.ThrowIfNull(ownership);
13+
ArgumentNullException.ThrowIfNull(eventData);
14+
var nativeEventId = eventData.EventId.ToGuid();
15+
ownership.Reserve(stream, nativeEventId);
16+
IWriteResult result;
17+
try
18+
{
19+
result = await writer.AppendToStreamAsync(stream, StreamState.NoStream, [eventData], cancellationToken: token);
20+
if (result is not SuccessResult)
21+
{
22+
throw new ComparisonFailureException(KurrentConstants.AppendAcknowledgementMissing);
23+
}
24+
}
25+
catch (WrongExpectedVersionException)
26+
{
27+
ownership.Reject(stream, nativeEventId);
28+
throw;
29+
}
30+
catch (Exception)
31+
{
32+
ownership.MarkUnknown(stream, nativeEventId);
33+
throw;
34+
}
35+
// No conversion, copy validation or success result can precede this actual ACK transition.
36+
ownership.Acknowledge(stream, nativeEventId);
37+
return result;
38+
}
39+
}

‎benchmarks/KeyLoad.Comparisons/Features/BenchmarkComparisons/KurrentReplicaProbe.cs‎

Lines changed: 6 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -8,10 +8,11 @@ namespace KeyLoad.Comparisons.Targets;
88
internal static class KurrentReplicaProbe
99
{
1010
public static async Task<ClusterEvidence> VerifyCopyAsync(KurrentDBClient writer, KurrentDBClient[] nodeClients,
11-
HttpClient[] httpClients, ComparisonTopology topology, string stream, KurrentEventData eventData, TimeSpan timeout,
11+
HttpClient[] httpClients, ComparisonTopology topology, string stream, KurrentEventData eventData,
12+
KurrentStreamOwnership ownership, TimeSpan timeout,
1213
CancellationToken cancellationToken)
1314
{
14-
var cut = await AppendAndCaptureCutAsync(writer, stream, eventData, cancellationToken);
15+
var cut = await AppendAndCaptureCutAsync(writer, ownership, stream, eventData, cancellationToken);
1516
using var limit = CancellationTokenSource.CreateLinkedTokenSource(cancellationToken);
1617
limit.CancelAfter(timeout);
1718
try
@@ -59,11 +60,10 @@ public static async Task RequireOriginalEventAsync(KurrentDBClient client, strin
5960
}
6061
}
6162

62-
private static async Task<KurrentAppendCut> AppendAndCaptureCutAsync(KurrentDBClient writer, string stream,
63-
KurrentEventData eventData, CancellationToken cancellationToken)
63+
private static async Task<KurrentAppendCut> AppendAndCaptureCutAsync(KurrentDBClient writer,
64+
KurrentStreamOwnership ownership, string stream, KurrentEventData eventData, CancellationToken cancellationToken)
6465
{
65-
var result = await writer.AppendToStreamAsync(stream, StreamState.NoStream, [eventData],
66-
cancellationToken: cancellationToken);
66+
var result = await KurrentOwnedStreamAppend.AppendAsync(writer, ownership, stream, eventData, cancellationToken);
6767
var commit = checked((long)result.LogPosition.CommitPosition);
6868
var prepare = checked((long)result.LogPosition.PreparePosition);
6969
if (commit < prepare)

‎benchmarks/KeyLoad.Comparisons/Features/BenchmarkComparisons/KurrentSession.cs‎

Lines changed: 2 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -42,9 +42,8 @@ public async Task<OperationResult> ExecuteAsync(Scenario scenario, BenchmarkDocu
4242
{
4343
case Scenario.StreamAppend:
4444
var appendStream = target.StreamName(document);
45-
target.TrackStream(appendStream);
46-
await client.AppendToStreamAsync(appendStream, StreamState.NoStream,
47-
[KurrentTarget.CreateEvent(document)], cancellationToken: cancellationToken);
45+
await KurrentOwnedStreamAppend.AppendAsync(client, target.Ownership, appendStream,
46+
KurrentTarget.CreateEvent(document), cancellationToken);
4847
return new();
4948
case Scenario.StreamRead:
5049
return new(Event: await ReadEventAsync(document, cancellationToken));
Lines changed: 74 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,74 @@
1+
namespace KeyLoad.Comparisons.Targets;
2+
3+
internal sealed class KurrentStreamOwnership
4+
{
5+
private readonly object gate = new();
6+
private readonly Dictionary<string, KurrentStreamOwnershipEntry> entries = new(StringComparer.Ordinal);
7+
8+
internal KurrentStreamOwnership(ComparisonOptions options)
9+
{
10+
ArgumentNullException.ThrowIfNull(options);
11+
options.Validate();
12+
Capacity = checked(options.Documents + options.Repetitions * checked(options.Warmup + options.Operations)
13+
+ KurrentConstants.OwnershipProbeCount);
14+
}
15+
16+
internal int Capacity { get; }
17+
internal int Count
18+
{
19+
get
20+
{
21+
lock (gate)
22+
{
23+
return entries.Count;
24+
}
25+
}
26+
}
27+
28+
internal void Reserve(string stream, Guid nativeEventId)
29+
{
30+
ArgumentException.ThrowIfNullOrEmpty(stream);
31+
if (nativeEventId == Guid.Empty)
32+
{
33+
throw new ArgumentException(KurrentConstants.OwnershipInvalidDescriptor, nameof(nativeEventId));
34+
}
35+
lock (gate)
36+
{
37+
if (entries.Count >= Capacity || !entries.TryAdd(stream, new(nativeEventId, KurrentStreamOwnershipState.Reserved)))
38+
{
39+
throw new ComparisonFailureException(KurrentConstants.OwnershipReservationRejected);
40+
}
41+
}
42+
}
43+
44+
internal void Acknowledge(string stream, Guid nativeEventId)
45+
=> Complete(stream, nativeEventId, KurrentStreamOwnershipState.Acknowledged);
46+
47+
internal void Reject(string stream, Guid nativeEventId)
48+
=> Complete(stream, nativeEventId, KurrentStreamOwnershipState.Rejected);
49+
50+
internal void MarkUnknown(string stream, Guid nativeEventId)
51+
=> Complete(stream, nativeEventId, KurrentStreamOwnershipState.Unknown);
52+
53+
internal string[] SnapshotAcknowledged()
54+
{
55+
lock (gate)
56+
{
57+
return entries.Where(pair => pair.Value.State == KurrentStreamOwnershipState.Acknowledged)
58+
.Select(pair => pair.Key).ToArray();
59+
}
60+
}
61+
62+
private void Complete(string stream, Guid nativeEventId, KurrentStreamOwnershipState state)
63+
{
64+
lock (gate)
65+
{
66+
if (!entries.TryGetValue(stream, out var entry) || entry.NativeEventId != nativeEventId ||
67+
entry.State != KurrentStreamOwnershipState.Reserved)
68+
{
69+
throw new ComparisonFailureException(KurrentConstants.OwnershipTransitionRejected);
70+
}
71+
entries[stream] = entry with { State = state };
72+
}
73+
}
74+
}
Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,3 @@
1+
namespace KeyLoad.Comparisons.Targets;
2+
3+
internal readonly record struct KurrentStreamOwnershipEntry(Guid NativeEventId, KurrentStreamOwnershipState State);
Lines changed: 9 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,9 @@
1+
namespace KeyLoad.Comparisons.Targets;
2+
3+
internal enum KurrentStreamOwnershipState
4+
{
5+
Reserved,
6+
Acknowledged,
7+
Rejected,
8+
Unknown,
9+
}

0 commit comments

Comments
 (0)