Stop owned Aspire test workloads on terminal resource failures

Author: KSemenenko

.github/workflows/ci.yml

@@ -103,6 +103,8 @@ jobs:
         run: node scripts/Features/RepositoryGovernance/verify.mjs
       - name: Test code analyzers
         run: dotnet run --project src/KeyLoad.AppHost --no-build --no-restore --configuration Release -- --KeyLoadTests:Suite=analyzers
+      - name: Test prompt termination on Aspire resource failures
+        run: dotnet run --project src/KeyLoad.AppHost --no-build --no-restore --configuration Release -- --KeyLoadTests:Suite=comparison '--KeyLoadTests:Filter=/*/*/AspireFailure*/*' --KeyLoadTests:ResultsDirectory=TestResults/aspire-failure --KeyLoadTests:ReportTrx=true
       - name: Test live benchmark progress capture
         run: dotnet run --project src/KeyLoad.AppHost --no-build --no-restore --configuration Release -- --KeyLoadTests:Suite=comparison '--KeyLoadTests:Filter=/*/*/ComparisonProgress*/*' --KeyLoadTests:ResultsDirectory=TestResults/live-benchmark-progress --KeyLoadTests:ReportTrx=true
       - name: Test live benchmark process entry

AGENTS.md

@@ -404,6 +404,7 @@ For changes outside existing owner authorization, obtain direction before changi
 ## Preferences
 
 ### Likes
+- Owner correction 2026-10-04 requires an actual Aspire/AppHost or required topology startup failure to terminate its owned test/workload promptly, retain the original failure and diagnostics, and clean up owned resources instead of waiting for the full workload deadline. Dashboard-disabled warnings alone are not topology failures; planned fault injection and independent database jobs retain their explicit contracts.
 - Owner correction 2026-10-04 requires bounded live benchmark phase and attempt progress in GitHub Actions so a long workload cannot appear idle for tens of minutes. Retain cancellation diagnostics before teardown; keep original failures, required workload sizes, native topology and qualification gates intact.
 - Owner correction 2026-10-04 requires Benchmarks to finish every independent database workload and publish authenticated successful measurements even when another benchmark, including unfinished KeyLoad, fails. Record each failed workload as unavailable with no numeric data and its actual failure/job provenance; never invent results or label a failed job successful. This explicitly supersedes the all-success comparison-cohort publication restriction while preserving complete planned-cell accounting, source/run/attempt identity, native topology, fairness, site qualification and least-privilege deployment. Benchmark repair does not authorize taking over concurrent KeyLoad engine implementation.
 - Owner correction 2026-10-04 separates database measurements from website testing: Benchmarks owns native database preparation, independent workloads, result validation and authenticated JSON aggregation. Chrome, browser/site tests and website qualification MUST NOT be benchmark dependencies. A separate website-build action may run after the JSON is ready or be triggered through CI; the owner explicitly permits static site building at the end of Benchmarks. The selected implementation uses independent CI website jobs on own-main push/manual and completed Benchmarks events. Every website build MUST use the newest completed own-main Benchmarks run and its successful authenticated aggregate, across push/manual producers, rather than blindly using the triggering run or falling back after invalid latest evidence. Preserve exactly three workflows, failed/null cells, original provenance, website gates and predeploy freshness. Website failure cannot block database metrics production. This supersedes earlier integrated qualification placement and the interim static-build prohibition. KeyLoad engine implementation remains owned by the other agent.

docs/ADR/ADR-086-aspire-terminal-failure.md

@@ -0,0 +1,69 @@
+# ADR-086: Stop owned test workloads on terminal Aspire failure
+
+Status: Accepted implementation contract; native qualification pending.
+Date: 2026-10-04. Related: REQ/AC-TEST-012..014, ADR-074, ADR-085.
+
+## Decision
+
+Use actual Aspire resource notifications and host stopping cancellation, rather
+than parsing CLI warning text. The disabled dashboard is optional infrastructure
+and cannot determine the test outcome. Original CI37202347856 reports show
+completed native tests with business failures, not a proven AppHost crash.
+
+Register observation before starting the owned application. Race native runner
+completion against the required dependency failure notifications. FailedToStart,
+RuntimeUnhealthy, Finished, Exited or an exit code end a runner wait; an absent
+original exit status is an error. Required long-lived dependencies cannot exit
+successfully during measurement. Derive dependencies from actual WaitAnnotations,
+following their graph once per resource, with bounded tasks proportional to the
+existing fixed topology. Expected WaitForCompletion exits remain valid, while
+wrong or missing exits fail. HealthStatus.Unhealthy by itself is not a terminal
+failure. Preserve native exit codes and fail before the full deadline.
+
+Cancel and join every owned observer on success, failure or cancellation. Observe
+startup concurrently with failure so a resource failure also cancels pending
+startup. Retain the initiating exception rather than replacing it with sibling
+cancellation. Existing stop/dispose and bounded diagnostic retention remain the
+owner's finally responsibility. No detached task or new timeout increase.
+
+Startup readiness for the shared RF3 fixture explicitly uses native
+StopOnResourceUnavailable and cancels/joins other startup waits when one fails.
+This startup guard ends at initial readiness. Continuous comparison guards end
+at runner completion, before intentional node loss/restart regression scenarios.
+Explicit WaitOnResourceUnavailable restart paths stay unchanged. Independent
+database jobs continue their own workloads after another database fails.
+
+## Implementation and verification
+
+1. Root owns requirements, this ADR, shared integration and Git delivery.
+2. progress_runner owns only new AppHost Features/TestInfrastructure native
+   lifecycle helpers and new ComparisonTests Features/TestInfrastructure tests.
+   Genuine Aspire notifications cover terminal/missing-exit, live dependency
+   failure, successful/failed bootstrap, no dependencies, cancellation and task
+   settlement. Native failures settle within five seconds of event publication.
+3. Root joins TestSuiteApplication, IsolatedNativeCase and shared ClusterFixture
+   startup, with any necessary test friend visibility. vector_runtime reviews
+   required resources and planned fault boundaries read-only. Concurrent engine
+   CQRS/upgrade code and its dedicated wave fixtures remain with their owner.
+4. Build/format/governance and focused TUnit tests through the canonical Aspire
+   entry, then retain exact-source Linux CI reports. Full unit/scalar/recovery/RF3
+   and native benchmark qualification remain required; unrelated failures are
+   reported honestly. Source/native service tests do not prove a Docker crash
+   was exercised. A real failure artifact supplies environmental-path evidence.
+
+Graph: CONTRACT -> NATIVE HELPERS/TESTS -> JOINS -> REVIEW -> EVIDENCE.
+Acceptance mapping is in TestInfrastructure. No production contract, package,
+data-format or topology migration. Rollback reverts only these lifecycle joins
+and helpers; the canonical Aspire entry and existing outcomes remain required.
+Frontend N/A: developer and CI lifecycle only.
+
+```mermaid
+flowchart LR
+    Start[Owned AppHost start] --> Race[Native runner and required resources]
+    Race --> Exit[Original runner exit]
+    Race --> Failure[Terminal required resource failure]
+    Exit --> Join[Cancel and join observers]
+    Failure --> Join
+    Join --> Evidence[Original failure and diagnostics]
+    Evidence --> Cleanup[Stop and dispose owned resources]
+```

docs/ADR/README.md

@@ -165,3 +165,8 @@ series-codec qualification stage; canonical layout, rewrite/migration and fault
 qualification remain separate. [ADR-081](ADR-081-awaited-native-search-execution.md)
 freezes awaited admitted native search execution and independent RF3 oracle
 repairs. Both retain source/runtime and exact-source Linux qualification gates.
+
+[ADR-086](ADR-086-aspire-terminal-failure.md) defines prompt termination on actual
+owned Aspire resource failures, preserving original outcomes, dependency
+completion, planned fault injection and independent comparison jobs. Native
+notification regressions and delivered-source qualification remain pending.

docs/Features/TestInfrastructure.md

@@ -16,6 +16,35 @@ ComparisonTests Features/TestInfrastructure; shared IntegrationTests fixture and
 CI composition are root-owned joins. No public database or persisted format change.
 Local entry runs are development evidence; delivered-source Linux gates remain.
 
+## Prompt termination on owned Aspire failure
+
+[ADR-086](../ADR/ADR-086-aspire-terminal-failure.md) defines the private lifecycle
+repair prompted by CI [37202347856](https://github.com/managedcode/KeyLoad/actions/runs/37202347856).
+Its original reports show RF3 87/88 passed (cold-upgrade authorization assertion)
+and unit 2947/2953 passed (six native CQRS cases); they do not establish an AppHost
+crash. Website tests were 201/202 passed, with a real Chrome node-selection
+assertion failure. Dashboard-disabled CLI warnings appeared before successful
+suites too. These facts come from the original RF3, unit and site TUnit artifacts
+at source `15ea5030c6fc010b29127b4b8de07bd255afc7ff`, not inferred CLI text.
+
+| Requirement | Acceptance and evidence |
+|---|---|
+| REQ-TEST-012: owned terminal Aspire failures stop waiting promptly | AC-TEST-012: a runner's FailedToStart, RuntimeUnhealthy, Finished/Exited without an original exit code, or a required long-lived dependency's terminal state fails its workload without consuming the suite deadline; genuine native notification tests settle within five seconds of publication |
+| REQ-TEST-013: completion and planned fault contracts remain intact | AC-TEST-013: expected WaitForCompletion exit is allowed; nonzero/missing completion exit fails; healthy/running and transient health snapshots continue waiting; no dependencies remains valid; guards are canceled and joined before intentional comparison node kill/restart assertions |
+| REQ-TEST-014: original outcome, ownership and cleanup remain intact | AC-TEST-014: native runner exit codes, startup exceptions and cancellation remain failures where applicable; AppHost stopping cancels waits; every notification task is canceled/joined, existing diagnostics and stop/disposal remain; unrelated database cells are unaffected |
+
+TASK-TEST-FAILFAST-CONTRACT (root, complete): requirements/ADR and owning source
+review. TASK-TEST-FAILFAST-NATIVE (progress_runner, implementation): only new
+AppHost Features/TestInfrastructure lifecycle helpers and ComparisonTests
+Features/TestInfrastructure native notification regressions. TASK-TEST-FAILFAST-JOIN
+(root, after helper): TestSuiteApplication and IsolatedNativeCase integration,
+CI focused selection, docs, review and exact-source validation. vector_runtime
+owns read-only lifecycle review. These tasks preserve concurrent engine changes;
+RF3 CQRS/authorization failures belong to their existing owner. Public database,
+stored data, dependencies and topology are N/A: this is private test orchestration.
+Native service tests establish notification handling, not real Docker failure
+qualification; delivered Linux tests and failure artifact review remain required.
+
 Status: implementation in progress. Owner: KeyLoad lead. Decision: [ADR-036](../ADR/ADR-036-orleans-foundation.md).
 
 REQ-TEST-008 / AC-TEST-008 (TASK-RUNTIME-EVENTSOURCE-W3) coordinates the actual

src/KeyLoad.AppHost/Features/BenchmarkComparisons/IsolatedTestFriend.cs

@@ -1,6 +1,7 @@
 using System.Runtime.CompilerServices;
 
 [assembly: InternalsVisibleTo(KeyLoad.AppHost.Features.BenchmarkComparisons.IsolatedTestFriend.AssemblyName)]
+[assembly: InternalsVisibleTo("KeyLoad.IntegrationTests")]
 
 namespace KeyLoad.AppHost.Features.BenchmarkComparisons;
 

src/KeyLoad.AppHost/Features/TestInfrastructure/AspireOwnedTaskJoin.cs

@@ -0,0 +1,70 @@
+using System.Runtime.ExceptionServices;
+
+namespace KeyLoad.AppHost.Features.TestInfrastructure;
+
+internal static class AspireOwnedTaskJoin
+{
+    private const string MultipleFailures = "Aspire execution and owned task cleanup failed.";
+
+    internal static async Task<Task> FirstFailureOrAllAsync(Task[] tasks)
+    {
+        var pending = tasks.ToList();
+        while (pending.Count != 0)
+        {
+            var settled = await Task.WhenAny(pending).ConfigureAwait(false);
+            if (!settled.IsCompletedSuccessfully)
+            {
+                return settled;
+            }
+            pending.Remove(settled);
+        }
+        return Task.CompletedTask;
+    }
+
+    internal static async Task CompleteAsync(CancellationTokenSource lifetime, Task[] tasks, Task primary)
+    {
+        await primary.ConfigureAwait(ConfigureAwaitOptions.SuppressThrowing);
+        var failures = new List<Exception>();
+        await RetainAsync(primary, failures, includeCancellation: true).ConfigureAwait(false);
+        var cancellation = lifetime.CancelAsync();
+        await cancellation.ConfigureAwait(ConfigureAwaitOptions.SuppressThrowing);
+        await RetainAsync(cancellation, failures, includeCancellation: true).ConfigureAwait(false);
+        await Task.WhenAll(tasks).ConfigureAwait(ConfigureAwaitOptions.SuppressThrowing);
+        foreach (var task in tasks.Where(task => !ReferenceEquals(task, primary)))
+        {
+            await RetainAsync(task, failures, includeCancellation: false).ConfigureAwait(false);
+        }
+        ThrowFailures(failures);
+    }
+
+    private static async Task RetainAsync(Task task, List<Exception> failures, bool includeCancellation)
+    {
+        if (task.IsFaulted)
+        {
+            failures.AddRange(task.Exception!.InnerExceptions);
+        }
+        else if (includeCancellation && task.IsCanceled)
+        {
+            try
+            {
+                await task.ConfigureAwait(false);
+            }
+            catch (OperationCanceledException error)
+            {
+                failures.Add(error);
+            }
+        }
+    }
+
+    private static void ThrowFailures(List<Exception> failures)
+    {
+        if (failures.Count == 1)
+        {
+            ExceptionDispatchInfo.Capture(failures[0]).Throw();
+        }
+        if (failures.Count > 1)
+        {
+            throw new AggregateException(MultipleFailures, failures);
+        }
+    }
+}

src/KeyLoad.AppHost/Features/TestInfrastructure/AspireRequiredResources.cs

@@ -0,0 +1,57 @@
+using Aspire.Hosting.ApplicationModel;
+using Microsoft.Extensions.DependencyInjection;
+
+namespace KeyLoad.AppHost.Features.TestInfrastructure;
+
+internal static class AspireRequiredResources
+{
+    private const string MissingRunner = "The Aspire application model has no selected runner.";
+    private const string ConflictingCompletion = "An Aspire dependency has incompatible expected completion exits.";
+
+    internal static Dictionary<string, int?> ForRunner(DistributedApplication app, string runnerName)
+    {
+        var resources = app.Services.GetRequiredService<DistributedApplicationModel>().Resources;
+        var runner = resources.SingleOrDefault(resource => string.Equals(resource.Name, runnerName, StringComparison.OrdinalIgnoreCase))
+            ?? throw new InvalidOperationException(MissingRunner);
+        var dependencies = new Dictionary<string, int?>(StringComparer.OrdinalIgnoreCase);
+        var visited = new HashSet<IResource> { runner };
+        var pending = new Queue<IResource>();
+        pending.Enqueue(runner);
+        while (pending.TryDequeue(out var resource))
+        {
+            AddDependencies(resource, runner, dependencies, visited, pending);
+        }
+        return dependencies;
+    }
+
+    private static void AddDependencies(IResource resource, IResource runner, Dictionary<string, int?> dependencies,
+        HashSet<IResource> visited, Queue<IResource> pending)
+    {
+        foreach (var wait in resource.Annotations.OfType<WaitAnnotation>())
+        {
+            if (wait.Resource is IResourceWithoutLifetime || ReferenceEquals(wait.Resource, runner))
+            {
+                continue;
+            }
+            AddContract(dependencies, wait);
+            if (visited.Add(wait.Resource))
+            {
+                pending.Enqueue(wait.Resource);
+            }
+        }
+    }
+
+    private static void AddContract(Dictionary<string, int?> dependencies, WaitAnnotation wait)
+    {
+        int? expected = wait.WaitType == WaitType.WaitForCompletion ? wait.ExitCode : null;
+        if (dependencies.TryGetValue(wait.Resource.Name, out var prior))
+        {
+            if (prior is not null && expected is not null && prior != expected)
+            {
+                throw new InvalidOperationException(ConflictingCompletion);
+            }
+            expected = prior is null || expected is null ? null : expected;
+        }
+        dependencies[wait.Resource.Name] = expected;
+    }
+}