Skip to content

Commit 31dd7ea

Browse files
KSemenenkoKSemenenko
committed
Report live native benchmark progress and retain cancellation snapshots
1 parent 377886f commit 31dd7ea

31 files changed

Lines changed: 1514 additions & 33 deletions

‎.github/workflows/benchmarks.yml‎

Lines changed: 7 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -133,6 +133,12 @@ jobs:
133133
run: dotnet run --project src/KeyLoad.AppHost --no-build --no-restore --configuration Release -- --KeyLoadTests:Suite=comparison '--KeyLoadTests:Filter=/*/*/IsolatedKeyLoadAdmissionTests/*' --KeyLoadTests:ResultsDirectory=TestResults/comparison-images/keyload-admission
134134
- name: Test bounded database log buffers
135135
run: dotnet run --project src/KeyLoad.AppHost --no-build --no-restore --configuration Release -- --KeyLoadTests:Suite=comparison '--KeyLoadTests:Filter=/*/*/ComparisonResourceLogBufferTests/*' --KeyLoadTests:ResultsDirectory=TestResults/comparison-images/log-buffer
136+
- name: Test live benchmark progress capture
137+
run: dotnet run --project src/KeyLoad.AppHost --no-build --no-restore --configuration Release -- --KeyLoadTests:Suite=comparison '--KeyLoadTests:Filter=/*/*/ComparisonProgress*/*' --KeyLoadTests:ResultsDirectory=TestResults/comparison-images/live-progress
138+
- name: Test live benchmark process entry
139+
run: dotnet run --project src/KeyLoad.AppHost --no-build --no-restore --configuration Release -- --KeyLoadTests:Suite=unit '--KeyLoadTests:Filter=/*/*/NativeBenchmarkProgressEntryTests/*' --KeyLoadTests:ResultsDirectory=TestResults/comparison-images/live-entry
140+
- name: Test benchmark phase and attempt counters
141+
run: dotnet run --project src/KeyLoad.AppHost --no-build --no-restore --configuration Release -- --KeyLoadTests:Suite=unit '--KeyLoadTests:Filter=/*/*/ComparisonLiveProgress*/*' --KeyLoadTests:ResultsDirectory=TestResults/comparison-images/live-counters
136142
- name: Test KeyLoad replay admission
137143
run: dotnet run --project src/KeyLoad.AppHost --no-build --no-restore --configuration Release -- --KeyLoadTests:Suite=comparison '--KeyLoadTests:Filter=/*/*/IsolatedKeyLoadReplayAdmissionTests/*' --KeyLoadTests:ResultsDirectory=TestResults/comparison-images/replay-admission
138144
- name: Test Redis replica error reporting
@@ -223,7 +229,7 @@ jobs:
223229
echo '::error::Benchmark preparation failed; no measurement data is available.'
224230
exit 1
225231
fi
226-
dotnet run --project src/KeyLoad.AppHost --no-build --no-restore --configuration Release -- --KeyLoadTests:Suite=comparison '--KeyLoadTests:Filter=/*/*/IsolatedNativeComparisonTests/*' --KeyLoadTests:TimeoutMinutes=140
232+
node scripts/Features/BenchmarkComparisons/run-workload.mjs
227233
- name: Record benchmark availability
228234
if: ${{ !cancelled() }}
229235
env:

‎AGENTS.md‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -404,6 +404,7 @@ For changes outside existing owner authorization, obtain direction before changi
404404
## Preferences
405405

406406
### Likes
407+
- Owner correction 2026-10-04 requires bounded live benchmark phase and attempt progress in GitHub Actions so a long workload cannot appear idle for tens of minutes. Retain cancellation diagnostics before teardown; keep original failures, required workload sizes, native topology and qualification gates intact.
407408
- Owner correction 2026-10-04 requires Benchmarks to finish every independent database workload and publish authenticated successful measurements even when another benchmark, including unfinished KeyLoad, fails. Record each failed workload as unavailable with no numeric data and its actual failure/job provenance; never invent results or label a failed job successful. This explicitly supersedes the all-success comparison-cohort publication restriction while preserving complete planned-cell accounting, source/run/attempt identity, native topology, fairness, site qualification and least-privilege deployment. Benchmark repair does not authorize taking over concurrent KeyLoad engine implementation.
408409
- Owner correction 2026-10-04 separates database measurements from website testing: Benchmarks owns native database preparation, independent workloads, result validation and authenticated JSON aggregation. Chrome, browser/site tests and website qualification MUST NOT be benchmark dependencies. A separate website-build action may run after the JSON is ready or be triggered through CI; the owner explicitly permits static site building at the end of Benchmarks. The selected implementation uses independent CI website jobs on own-main push/manual and completed Benchmarks events. Every website build MUST use the newest completed own-main Benchmarks run and its successful authenticated aggregate, across push/manual producers, rather than blindly using the triggering run or falling back after invalid latest evidence. Preserve exactly three workflows, failed/null cells, original provenance, website gates and predeploy freshness. Website failure cannot block database metrics production. This supersedes earlier integrated qualification placement and the interim static-build prohibition. KeyLoad engine implementation remains owned by the other agent.
409410
- Complete the coherent task's implementation before executing its runtime tests or benchmarks. Author the mapped tests with the code, then run the required validation after the implementation and self-review; do not let repeated intermediate verification replace implementation progress. This owner correction2026-10-03 supersedes test-first execution for the current work while preserving every final correctness, CI, fault and performance gate.

‎benchmarks/KeyLoad.Comparisons/Features/BenchmarkComparisons/ComparisonMeasurement.cs‎

Lines changed: 42 additions & 11 deletions
Original file line numberDiff line numberDiff line change
@@ -4,7 +4,7 @@
44

55
namespace KeyLoad.Comparisons;
66

7-
internal sealed class ComparisonMeasurer(ComparisonOptions options)
7+
internal sealed class ComparisonMeasurer(ComparisonOptions options, ComparisonProgressObserver observer)
88
{
99
private const string FailedAttemptDetail = "Failed attempts remain in raw samples and latency; useful throughput counts verified successes.";
1010

@@ -28,21 +28,26 @@ public async Task<ComparisonCase> MeasureAsync(IComparisonTarget target, Benchma
2828
private async Task<ComparisonCase> MeasureOwnedAsync(IComparisonTarget target, BenchmarkDataset dataset,
2929
Scenario scenario, int repetition, List<IComparisonSession> sessions, CancellationToken cancellationToken)
3030
{
31+
observer.Begin(ComparisonProgressPhase.Initialize, repetition + 1);
3132
var inputs = Enumerable.Range(0, options.Operations).Select(operation => dataset.Input(scenario, repetition, operation, false)).ToArray();
3233
for (var worker = 0; worker < options.Concurrency; worker++)
3334
{
3435
sessions.Add(await target.OpenSessionAsync(cancellationToken));
3536
}
3637

3738
await WarmupAsync(sessions, dataset, scenario, repetition, cancellationToken);
39+
observer.Begin(ComparisonProgressPhase.Prepare, repetition + 1);
3840
await ComparisonMutationPreparation.PrepareAsync(sessions, inputs, scenario, options.TimeoutSeconds, cancellationToken);
3941
var samples = new OperationSample[options.Operations];
4042
var outputs = new OperationResult?[options.Operations];
4143
await using var resources = new ClientResourceSampler();
44+
observer.Begin(ComparisonProgressPhase.Measure, repetition + 1, options.Operations);
4245
var clock = Stopwatch.StartNew();
4346
await ExecuteBatchAsync(sessions, inputs, scenario, samples, outputs, clock, cancellationToken);
4447
clock.Stop();
4548
var clientResources = await resources.StopAsync();
49+
observer.Begin(ComparisonProgressPhase.Validate, repetition + 1, samples.Length, samples.Length,
50+
samples.Count(sample => !sample.Success));
4651
if (ComparisonMutationPreparation.Required(scenario))
4752
{
4853
await ComparisonMutationValidation.ValidateAsync(sessions, dataset, scenario, inputs, samples, outputs, cancellationToken);
@@ -52,6 +57,7 @@ private async Task<ComparisonCase> MeasureOwnedAsync(IComparisonTarget target, B
5257
await ComparisonValidation.ValidateBatchAsync(sessions[0], dataset, scenario, inputs, samples, outputs, cancellationToken);
5358
}
5459
var measurement = ComparisonStatistics.Summarize(samples, clock.Elapsed.TotalSeconds) with { ClientResources = clientResources };
60+
observer.Begin(ComparisonProgressPhase.Complete, repetition + 1, measurement.Attempts, measurement.Attempts, measurement.Failures);
5561
return new(target.Profile.Name, scenario, repetition, measurement.Failures == 0 ? ComparisonStatuses.Measured : ComparisonStatuses.Failed,
5662
measurement.Failures == 0 ? null : FailedAttemptDetail, measurement,
5763
ImmutableCollectionsMarshal.AsImmutableArray(samples));
@@ -61,14 +67,25 @@ private async Task WarmupAsync(List<IComparisonSession> sessions, BenchmarkDatas
6167
int repetition, CancellationToken cancellationToken)
6268
{
6369
var warmup = Enumerable.Range(0, options.Warmup).Select(operation => dataset.Input(scenario, repetition, operation, true)).ToArray();
70+
observer.Begin(ComparisonProgressPhase.Prepare, repetition + 1);
6471
await ComparisonMutationPreparation.PrepareAsync(sessions, warmup, scenario, options.TimeoutSeconds, cancellationToken);
72+
observer.Begin(ComparisonProgressPhase.Warmup, repetition + 1, options.Warmup);
6573
for (var operation = 0; operation < options.Warmup; operation++)
6674
{
6775
using var deadline = ComparisonDeadline.Create(options.TimeoutSeconds, cancellationToken);
6876
var input = warmup[operation];
6977
var session = sessions[operation % sessions.Count];
70-
var output = await session.ExecuteAsync(scenario, input, deadline.Token);
71-
await ComparisonValidation.ValidateOperationAsync(session, dataset, scenario, input, output, deadline.Token);
78+
var success = false;
79+
try
80+
{
81+
var output = await session.ExecuteAsync(scenario, input, deadline.Token);
82+
await ComparisonValidation.ValidateOperationAsync(session, dataset, scenario, input, output, deadline.Token);
83+
success = true;
84+
}
85+
finally
86+
{
87+
observer.Settle(success);
88+
}
7289
}
7390
}
7491

@@ -87,18 +104,32 @@ await Task.WhenAll(sessions.Select(async (session, worker) =>
87104
}
88105

89106
cancellationToken.ThrowIfCancellationRequested();
90-
using var deadline = ComparisonDeadline.Create(options.TimeoutSeconds, cancellationToken);
91-
var started = clock.Elapsed.TotalMilliseconds;
92-
string? failure = null;
93-
try
94-
{ outputs[operation] = await session.ExecuteAsync(scenario, inputs[operation], deadline.Token); }
95-
catch (Exception error) when (!cancellationToken.IsCancellationRequested) { failure = ComparisonErrors.Safe(error); }
96-
samples[operation] = new(operation, worker, started, clock.Elapsed.TotalMilliseconds, failure is null, failure,
97-
Encoding.UTF8.GetByteCount(inputs[operation].Json), outputs[operation]?.Message?.Id, outputs[operation]?.Queue);
107+
await ExecuteAttemptAsync(session, inputs, scenario, samples, outputs, clock, operation, worker, cancellationToken);
98108
}
99109
}));
100110
}
101111

112+
private async Task ExecuteAttemptAsync(IComparisonSession session, BenchmarkDocument[] inputs, Scenario scenario,
113+
OperationSample[] samples, OperationResult?[] outputs, Stopwatch clock, int operation, int worker, CancellationToken cancellationToken)
114+
{
115+
using var deadline = ComparisonDeadline.Create(options.TimeoutSeconds, cancellationToken);
116+
var started = clock.Elapsed.TotalMilliseconds;
117+
var success = false;
118+
try
119+
{
120+
string? failure = null;
121+
try
122+
{ outputs[operation] = await session.ExecuteAsync(scenario, inputs[operation], deadline.Token); }
123+
catch (Exception error) when (!cancellationToken.IsCancellationRequested) { failure = ComparisonErrors.Safe(error); }
124+
samples[operation] = new(operation, worker, started, clock.Elapsed.TotalMilliseconds, failure is null, failure,
125+
Encoding.UTF8.GetByteCount(inputs[operation].Json), outputs[operation]?.Message?.Id, outputs[operation]?.Queue);
126+
success = failure is null;
127+
}
128+
finally
129+
{
130+
observer.Settle(success);
131+
}
132+
}
102133
}
103134

104135
internal static class ComparisonStatistics

‎benchmarks/KeyLoad.Comparisons/Features/BenchmarkComparisons/ComparisonProgressObserver.cs‎

Lines changed: 137 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,137 @@
1+
using System.Diagnostics;
2+
using System.Globalization;
3+
using System.Text;
4+
5+
namespace KeyLoad.Comparisons;
6+
7+
internal sealed class ComparisonProgressObserver : IAsyncDisposable
8+
{
9+
private const int HeartbeatSeconds = 30;
10+
private const string LineFormat = "KeyLoadBenchmarkProgress phase={0} repetition={1} completed={2} total={3} failed={4} elapsedSeconds={5:F3}";
11+
private static readonly CompositeFormat ProgressFormat = CompositeFormat.Parse(LineFormat);
12+
private readonly Action<string>? _progress;
13+
private readonly object _outputGate = new();
14+
private readonly Stopwatch _elapsed = Stopwatch.StartNew();
15+
private readonly CancellationTokenSource _lifetime = new();
16+
private readonly PeriodicTimer _timer = new(TimeSpan.FromSeconds(HeartbeatSeconds));
17+
private readonly Task _heartbeat;
18+
private ComparisonProgressState _state = new(ComparisonProgressPhase.Oracle, 0, 0);
19+
private bool _hasState;
20+
21+
internal ComparisonProgressObserver(Action<string>? progress)
22+
{
23+
_progress = progress;
24+
_heartbeat = progress is null ? Task.CompletedTask : Task.Run(ObserveAsync);
25+
}
26+
27+
internal void Begin(ComparisonProgressPhase phase, int repetition, int total = 0, int completed = 0, int failed = 0)
28+
{
29+
if (!Enum.IsDefined(phase) || repetition < 0 || total < 0 || completed < 0 || completed > total || failed < 0 || failed > completed)
30+
{
31+
throw new ArgumentOutOfRangeException(nameof(phase));
32+
}
33+
lock (_outputGate)
34+
{
35+
if (_hasState)
36+
{
37+
Emit();
38+
}
39+
Volatile.Write(ref _state, new(phase, repetition, total, completed, failed));
40+
_hasState = true;
41+
Emit();
42+
}
43+
}
44+
45+
internal void Settle(bool success) => Volatile.Read(ref _state).Settle(success);
46+
47+
internal void Complete()
48+
{
49+
var state = Volatile.Read(ref _state);
50+
Begin(ComparisonProgressPhase.Complete, state.Repetition, state.Total, state.Completed, state.Failed);
51+
}
52+
53+
public async ValueTask DisposeAsync()
54+
{
55+
await _lifetime.CancelAsync();
56+
_timer.Dispose();
57+
try
58+
{
59+
await _heartbeat;
60+
lock (_outputGate)
61+
{
62+
Emit();
63+
}
64+
}
65+
finally
66+
{
67+
_lifetime.Dispose();
68+
}
69+
}
70+
71+
private async Task ObserveAsync()
72+
{
73+
try
74+
{
75+
while (await _timer.WaitForNextTickAsync(_lifetime.Token))
76+
{
77+
lock (_outputGate)
78+
{
79+
Emit();
80+
}
81+
}
82+
}
83+
catch (OperationCanceledException) when (_lifetime.IsCancellationRequested)
84+
{
85+
// The owning runner stops and joins the observer before leaving its scope.
86+
}
87+
}
88+
89+
private void Emit()
90+
{
91+
if (_progress is null)
92+
{
93+
return;
94+
}
95+
var state = Volatile.Read(ref _state);
96+
var completed = state.Completed;
97+
var failed = Math.Min(state.Failed, completed);
98+
var line = string.Format(CultureInfo.InvariantCulture, ProgressFormat, PhaseName(state.Phase), state.Repetition,
99+
completed, state.Total, failed, _elapsed.Elapsed.TotalSeconds);
100+
try
101+
{
102+
_progress(line);
103+
}
104+
catch (IOException)
105+
{
106+
// A failed diagnostic output must preserve the native workload outcome.
107+
}
108+
catch (UnauthorizedAccessException)
109+
{
110+
// Diagnostic output may lose file access while a workload is active.
111+
}
112+
catch (ObjectDisposedException)
113+
{
114+
// A closed diagnostic writer must not fail the workload or observer cleanup.
115+
}
116+
}
117+
118+
private static string PhaseName(ComparisonProgressPhase phase) => phase switch
119+
{
120+
ComparisonProgressPhase.Oracle => Oracle,
121+
ComparisonProgressPhase.Initialize => Initialize,
122+
ComparisonProgressPhase.Warmup => Warmup,
123+
ComparisonProgressPhase.Prepare => Prepare,
124+
ComparisonProgressPhase.Measure => Measure,
125+
ComparisonProgressPhase.Validate => Validate,
126+
ComparisonProgressPhase.Complete => CompletePhase,
127+
_ => throw new ArgumentOutOfRangeException(nameof(phase))
128+
};
129+
130+
private const string Oracle = "oracle";
131+
private const string Initialize = "initialize";
132+
private const string Warmup = "warmup";
133+
private const string Prepare = "prepare";
134+
private const string Measure = "measure";
135+
private const string Validate = "validate";
136+
private const string CompletePhase = "complete";
137+
}

‎benchmarks/KeyLoad.Comparisons/Features/BenchmarkComparisons/ComparisonProgressPhase.cs‎

Lines changed: 12 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,12 @@
1+
namespace KeyLoad.Comparisons;
2+
3+
internal enum ComparisonProgressPhase
4+
{
5+
Oracle,
6+
Initialize,
7+
Warmup,
8+
Prepare,
9+
Measure,
10+
Validate,
11+
Complete
12+
}

‎benchmarks/KeyLoad.Comparisons/Features/BenchmarkComparisons/ComparisonProgressState.cs‎

Lines changed: 23 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,23 @@
1+
namespace KeyLoad.Comparisons;
2+
3+
internal sealed class ComparisonProgressState(ComparisonProgressPhase phase, int repetition, int total,
4+
int completed = 0, int failed = 0)
5+
{
6+
private int _completed = completed;
7+
private int _failed = failed;
8+
9+
internal ComparisonProgressPhase Phase { get; } = phase;
10+
internal int Repetition { get; } = repetition;
11+
internal int Total { get; } = total;
12+
internal int Completed => Volatile.Read(ref _completed);
13+
internal int Failed => Volatile.Read(ref _failed);
14+
15+
internal void Settle(bool success)
16+
{
17+
if (!success)
18+
{
19+
Interlocked.Increment(ref _failed);
20+
}
21+
Interlocked.Increment(ref _completed);
22+
}
23+
}

0 commit comments

Comments
 (0)