Publish authenticated benchmark failures without blocking competitors

Author: KSemenenko

.github/workflows/AGENTS.md

@@ -52,3 +52,6 @@
 - The latest owner correction places the actual pinned TimeSeries image test and retained facts in common image preparation, with no standalone feature branch. Keep all existing workload checks and isolated native measurements; staged intensive TimeSeries measurements remain explicitly pending.
 - Build/checks, plan and images run independently; preflight, CRUD and specialized matrices share only real plan/image inputs, never each other's completion or an arbitrary max-parallel cap. Aggregate explicitly waits for all six preparation/check/matrix jobs before website checks/publication.
 - Every authored job/step in the three workflows and used composites has a concrete readable name, including checkout, SDK setup, restore/build/test and uploads. Change displayed-name validators/oracles atomically; preserve stable internal job IDs, authenticated artifacts and all historical report bytes. Explicit owner direction supersedes earlier frozen displayed labels, without relaxing any identity, workload, topology, permission, coverage or publication gate.
+
+## Owner-directed failed-cell publication, 2026-10-04
+- ADR-080 and the latest explicit owner instruction supersede the all-success cohort requirement: finish every independent workload and publish authenticated successful measurements while failed workloads retain null reports and actual failed job/workload conclusions. Keep complete planned-cell artifact accounting, successful image authority, full site qualification, exact source/run/attempt identity and least privileges. Unrelated KeyLoad build/runtime failures do not by themselves skip aggregation or website jobs; missing/corrupt evidence and failed site gates still block publication.

.github/workflows/Features/BenchmarkComparisons/IsolatedCellTeardown/AGENTS.md

@@ -17,3 +17,6 @@
 
 ## Owner-directed comparison pipeline, 2026-10-03
 - ADR-062 moves the exact-SHA native comparison cells from the historical ci.yml placement above to benchmarks.yml (`Benchmarks`). Every existing cleanup, retention, isolation and publication-failure requirement remains mandatory.
+
+## Owner-directed failed-cell publication, 2026-10-04
+- Retain original failed runner reports separately from the null-report worker envelope, plus the finalization job capture. Workload failure may be published as unavailable under ADR-080; teardown failure itself remains a failed ownership/cleanup gate and cannot validate a measured cell.

.github/workflows/Features/BenchmarkComparisons/IsolatedCellTeardown/action.yml

@@ -24,5 +24,7 @@ runs:
           ${{ runner.temp }}/keyload-cell-github/**
           **/TestResults/**
           artifacts/code-quality/**
+          artifacts/comparisons/isolated/failures/**
+          ${{ runner.temp }}/keyload-cell-github-finalize/**
         if-no-files-found: error
         retention-days: 90

.github/workflows/Features/BenchmarkComparisons/QualifySite/AGENTS.md

@@ -13,3 +13,6 @@
 
 ## Applicable skills
 - No installed skill is required for this bounded workflow composition; install none.
+
+## Owner-directed failed-cell publication, 2026-10-04
+- ADR-080 allows authenticated failed workload/null-report cells; keep every existing archive, source, TUnit, coverage, browser and freshness gate. The trusted producer/builder dependency closure now includes failed-cell finalization and its actual imports; source-bound counts change together with validators.

.github/workflows/Features/BenchmarkComparisons/QualifySite/action.yml

@@ -60,7 +60,7 @@ runs:
         git -C measured-isolated rev-parse HEAD > "$EVIDENCE_DIR/isolated-measured-revision.txt"
         closure="scripts/Features/BenchmarkComparisons/site-isolated-dependencies.txt"
         cmp "control/$closure" "website/$closure"
-        [[ "$(wc -l < "control/$closure")" == 65 ]]
+        [[ "$(wc -l < "control/$closure")" == 70 ]]
         LC_ALL=C sort -u "control/$closure" > "$EVIDENCE_DIR/isolated-dependencies.txt"
         cmp "control/$closure" "$EVIDENCE_DIR/isolated-dependencies.txt"
         sha256sum "website/$closure" >> "$EVIDENCE_DIR/executed-evidence-tools.sha256"

.github/workflows/benchmarks.yml

@@ -89,6 +89,7 @@ jobs:
       - name: Restore .NET packages
         run: dotnet restore KeyLoad.slnx
       - name: Build benchmark tests
+        id: benchmark-build
         run: dotnet build tests/KeyLoad.ComparisonTests --no-restore --configuration Release
       - name: Check TimescaleDB Docker image and startup tools
         env:
@@ -201,9 +202,25 @@ jobs:
       - name: Restore .NET packages
         run: dotnet restore KeyLoad.slnx
       - name: Build benchmark tests
+        id: benchmark-build
         run: dotnet build tests/KeyLoad.ComparisonTests --no-restore --configuration Release
       - name: Run database workload
-        run: dotnet run --project src/KeyLoad.AppHost --no-build --no-restore --configuration Release -- --KeyLoadTests:Suite=comparison '--KeyLoadTests:Filter=/*/*/IsolatedNativeComparisonTests/*' --KeyLoadTests:TimeoutMinutes=145
+        id: workload
+        if: always()
+        env:
+          PREPARATION_OUTCOME: ${{ steps.images.outcome }}
+          BUILD_OUTCOME: ${{ steps.benchmark-build.outcome }}
+        run: |
+          if [ "$PREPARATION_OUTCOME" != success ] || [ "$BUILD_OUTCOME" != success ]; then
+            echo '::error::Benchmark preparation failed; no measurement data is available.'
+            exit 1
+          fi
+          dotnet run --project src/KeyLoad.AppHost --no-build --no-restore --configuration Release -- --KeyLoadTests:Suite=comparison '--KeyLoadTests:Filter=/*/*/IsolatedNativeComparisonTests/*' --KeyLoadTests:TimeoutMinutes=140
+      - name: Record benchmark availability
+        if: always()
+        env:
+          KEYLOAD_WORKLOAD_OUTCOME: ${{ steps.workload.outcome }}
+        run: node scripts/Features/BenchmarkComparisons/finalize-worker.mjs
       - name: Save benchmark results
         if: always()
         uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a
@@ -246,9 +263,25 @@ jobs:
       - name: Restore .NET packages
         run: dotnet restore KeyLoad.slnx
       - name: Build benchmark tests
+        id: benchmark-build
         run: dotnet build tests/KeyLoad.ComparisonTests --no-restore --configuration Release
       - name: Run database workload
-        run: dotnet run --project src/KeyLoad.AppHost --no-build --no-restore --configuration Release -- --KeyLoadTests:Suite=comparison '--KeyLoadTests:Filter=/*/*/IsolatedNativeComparisonTests/*' --KeyLoadTests:TimeoutMinutes=145
+        id: workload
+        if: always()
+        env:
+          PREPARATION_OUTCOME: ${{ steps.images.outcome }}
+          BUILD_OUTCOME: ${{ steps.benchmark-build.outcome }}
+        run: |
+          if [ "$PREPARATION_OUTCOME" != success ] || [ "$BUILD_OUTCOME" != success ]; then
+            echo '::error::Benchmark preparation failed; no measurement data is available.'
+            exit 1
+          fi
+          dotnet run --project src/KeyLoad.AppHost --no-build --no-restore --configuration Release -- --KeyLoadTests:Suite=comparison '--KeyLoadTests:Filter=/*/*/IsolatedNativeComparisonTests/*' --KeyLoadTests:TimeoutMinutes=140
+      - name: Record benchmark availability
+        if: always()
+        env:
+          KEYLOAD_WORKLOAD_OUTCOME: ${{ steps.workload.outcome }}
+        run: node scripts/Features/BenchmarkComparisons/finalize-worker.mjs
       - name: Save benchmark results
         if: always()
         uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a
@@ -291,9 +324,25 @@ jobs:
       - name: Restore .NET packages
         run: dotnet restore KeyLoad.slnx
       - name: Build benchmark tests
+        id: benchmark-build
         run: dotnet build tests/KeyLoad.ComparisonTests --no-restore --configuration Release
       - name: Run database workload
-        run: dotnet run --project src/KeyLoad.AppHost --no-build --no-restore --configuration Release -- --KeyLoadTests:Suite=comparison '--KeyLoadTests:Filter=/*/*/IsolatedNativeComparisonTests/*' --KeyLoadTests:TimeoutMinutes=145
+        id: workload
+        if: always()
+        env:
+          PREPARATION_OUTCOME: ${{ steps.images.outcome }}
+          BUILD_OUTCOME: ${{ steps.benchmark-build.outcome }}
+        run: |
+          if [ "$PREPARATION_OUTCOME" != success ] || [ "$BUILD_OUTCOME" != success ]; then
+            echo '::error::Benchmark preparation failed; no measurement data is available.'
+            exit 1
+          fi
+          dotnet run --project src/KeyLoad.AppHost --no-build --no-restore --configuration Release -- --KeyLoadTests:Suite=comparison '--KeyLoadTests:Filter=/*/*/IsolatedNativeComparisonTests/*' --KeyLoadTests:TimeoutMinutes=140
+      - name: Record benchmark availability
+        if: always()
+        env:
+          KEYLOAD_WORKLOAD_OUTCOME: ${{ steps.workload.outcome }}
+        run: node scripts/Features/BenchmarkComparisons/finalize-worker.mjs
       - name: Save benchmark results
         if: always()
         uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a
@@ -311,6 +360,7 @@ jobs:
   comparison-aggregate:
     name: Combine benchmark results
     needs: [comparison-build, comparison-plan, comparison-images, comparison-preflight, comparison-crud, comparison-specialized]
+    if: ${{ always() && !cancelled() && needs.comparison-plan.result == 'success' && needs.comparison-images.result == 'success' }}
     runs-on: ubuntu-latest
     timeout-minutes: 150
     permissions: {contents: read, actions: read}
@@ -352,6 +402,7 @@ jobs:
   qualify:
     name: Check website
     needs: comparison-aggregate
+    if: ${{ always() && !cancelled() && needs.comparison-aggregate.result == 'success' }}
     runs-on: ubuntu-latest
     timeout-minutes: 180
     permissions: {contents: read, actions: read}
@@ -374,7 +425,7 @@ jobs:
   deploy:
     name: Publish website
     needs: qualify
-    if: needs.qualify.result == 'success' && needs.qualify.outputs.mode == 'publish'
+    if: ${{ always() && !cancelled() && needs.qualify.result == 'success' && needs.qualify.outputs.mode == 'publish' }}
     runs-on: ubuntu-latest
     timeout-minutes: 90
     permissions: {contents: read, actions: read, pages: write, id-token: write}

AGENTS.md

@@ -401,6 +401,7 @@ For changes outside existing owner authorization, obtain direction before changi
 ## Preferences
 
 ### Likes
+- Owner correction 2026-10-04 requires Benchmarks to finish every independent database workload and publish authenticated successful measurements even when another benchmark, including unfinished KeyLoad, fails. Record each failed workload as unavailable with no numeric data and its actual failure/job provenance; never invent results or label a failed job successful. This explicitly supersedes the all-success comparison-cohort publication restriction while preserving complete planned-cell accounting, source/run/attempt identity, native topology, fairness, site qualification and least-privilege deployment. Benchmark repair does not authorize taking over concurrent KeyLoad engine implementation.
 - Complete the coherent task's implementation before executing its runtime tests or benchmarks. Author the mapped tests with the code, then run the required validation after the implementation and self-review; do not let repeated intermediate verification replace implementation progress. This owner correction2026-10-03 supersedes test-first execution for the current work while preserving every final correctness, CI, fault and performance gate.
 - Each agent MUST follow the execution plan for its own user-assigned task and complete only that task; change its scope only when the owner explicitly redirects it (owner correction 2026-10-03).
 - Other chats' plans, progress, requests and concurrent changes MUST NOT expand an agent's task or redirect its implementation. Preserve their work and leave their tasks to their owners (owner correction 2026-10-03).

docs/ADR/ADR-080-benchmark-failure-isolation.md

@@ -0,0 +1,56 @@
+# ADR-080: Isolate benchmark failures during publication
+
+Status: Accepted; source implemented, delivered-source verification pending.
+Date: 2026-10-04. Related: REQ/AC-BC-FAIL-001..005, ADR-056/074/076.
+
+## Decision
+
+The owner requires all independent benchmarks to finish and publication to proceed
+when one database workload fails, including unfinished KeyLoad. Preserve every
+planned cell, actual native topology, bounded resources, workload contracts and
+source/run/attempt/job/artifact authentication. A failed workload emits a version4
+envelope with `disposition: failed`, a fixed safe reason and `report: null`. Its
+original job remains failed and workload step remains failure; result upload must
+succeed. Measured envelopes require successful workload jobs. Evidence mismatch,
+missing artifacts, mixed sources, failed image preparation or unsuccessful site
+qualification still fails publication. Partial success cannot establish a winner
+against an unavailable database. No engine implementation or persisted data changes.
+
+The alternative of stopping all publication discards useful independent results.
+Treating failures as zero would invent comparative performance. Retaining raw
+exception text risks exposing credentials; use a fixed public reason and link the
+original GitHub job for diagnostics. Ordinary runner failures are finalized before
+upload; cancellation/timeouts that prevent artifacts remain explicit blockers.
+
+## Implementation contract
+
+1. Root records owner policy, feature acceptance, exact baseline and shared schema.
+2. Producer worker owns aggregation/GitHub selection modules and producer regression
+   tests; preserve original step conclusions and require envelope/proof agreement.
+3. Site worker owns isolated browser contract/metadata/report/numeric/view modules
+   and dedicated TUnit projection/oracle/browser regressions. Failed cells contain
+   no numeric metrics and retain their failed job links.
+4. Root owns workflow always-finalization/aggregate condition, shared receipt
+   validation, dependency/coverage inventory, source name tests and documentation.
+   Diagnostic worker inspects native startup errors without changing KeyLoad engine.
+5. Root reviews all diffs, builds solution, runs formatter/governance and focused
+   Aspire-owned suites, then checkpoints scoped changes on current main and pushes.
+6. Genuine Linux Benchmarks run qualifies all cells, aggregate, site coverage/browser
+   and Pages publication. Local tests are development proof only.
+
+Migration is additive to version4 dispositions; deploy producer/validators/site
+atomically. Rollback reverts this coherent change and restores the conservative
+publication gate, retaining immutable original artifacts. AC-BC-FAIL-001..005 map
+to automated and actual-provider evidence in the feature specification. Root alone
+owns integration and shared contract updates; workers never commit or push.
+
+```mermaid
+flowchart TD
+  Runner[Aspire native runner] -->|success| Measured[Measurement envelope]
+  Runner -->|failure| Unavailable[Null report envelope]
+  Provider[Original GitHub job and artifact] --> Validator[Strict agreement validator]
+  Measured --> Validator
+  Unavailable --> Validator
+  Validator --> Site[Successful values and unavailable cells]
+  Site --> Gates[Full site qualification and Pages]
+```

docs/ADR/README.md

@@ -151,3 +151,9 @@ peer fences. Homogeneous rollout, real prior-binary proof and qualification pend
 ZoneTree.FullTextSearch candidate generations under a node-local physical owner,
 exact canonical ranking, persisted-policy cuts and fail-closed publication.
 Source integration and qualification pending; acceleration is not claimed.
+
+[ADR-080](ADR-080-benchmark-failure-isolation.md) accepts owner-directed independent
+benchmark failure handling: bounded transient GitHub GET retries, authenticated
+null-report failed cells and successful competitor publication. Full planned-cell
+accounting and complete site qualification remain required; delivered-source proof
+is pending.

docs/Architecture.md

@@ -947,3 +947,24 @@ flowchart LR
     Build --> Publish[Immutable tag GHCR and GitHub Release]
     Gate --> Publish
 ```
+
+## Independent benchmark failure publication
+
+[ADR-080](ADR/ADR-080-benchmark-failure-isolation.md) and
+[BenchmarkComparisons](Features/BenchmarkComparisons.md) preserve the full native
+planned-cell inventory while allowing terminal workload failures to coexist with
+successful measurements. The original GitHub job/step remains failed, its bounded
+worker envelope has a null report, and the site has no numeric result for that
+cell. The aggregate and site jobs explicitly wait for all matrices and run after
+failures; image authority, authenticated artifacts, fairness, full website
+qualification and freshness remain mandatory. KeyLoad engine repair is separate.
+
+```mermaid
+flowchart LR
+  Job[Independent native Aspire cell] --> Result[Measured or failed null report]
+  GitHub[Actual job steps and immutable artifact] --> Proof[Same source run attempt validation]
+  Result --> Proof
+  Proof --> Aggregate[Complete planned-cell aggregate]
+  Aggregate --> Qualification[Site tests browser coverage freshness]
+  Qualification --> Site[Values or no data and original job links]
+```