Skip to content
Merged
115 changes: 97 additions & 18 deletions .github/workflows/ci.yml
Original file line number Diff line number Diff line change
Expand Up @@ -7,14 +7,51 @@ on:

jobs:
test:
runs-on: macos-15
# `macos-latest`, not a pinned older image. Measured 2026-10-29: that
# label is macOS 26 Arm64 with Xcode 26.6 (17F113) as its default, and it
# is what release.yml's build job has used since that workflow was
# written (`c2453cf7`, 2026-07-27). It was never pinned to the older
# image; from the repository root, this prints nothing:
#
# git log -S'macos-15' -- .github/workflows/release.yml
#
# So the app that
# ships is already compiled there, tests and all. Pinned to `macos-15`
# this gate compiled with Xcode 16.4 (16F6) / Swift 6.1.2, a toolchain
# that neither the release builder nor the development machine (Xcode
# 27.1 / Swift 6.4) uses. That is how run 40913494898 came to fail the
# build on a file which had been green locally for weeks, and run
# 40928759131 to fail the test target on five more sites in both test
# targets: the only gate on the project was reading a compiler nobody
# here builds with.
#
# A moving label can move under us. The trade taken knowingly: both
# workflows now move together, so this gate meets a new image before a
# release does, which is the ordering that catches the move instead of
# shipping it. The toolchain step below prints `xcodebuild -version` AND
# `swift --version`, so from now on the figures travel with the run
# rather than with a comment. That is how the wrong figure survived
# from 2026-09-16, the date on the toolchain step's own measurement
# below, to 2026-10-29.
runs-on: macos-latest
# The test runner has hung intermittently since M20: the process sits at
# 0% CPU and never finishes, so the job would burn the runner's default
# six-hour budget instead of failing. Green runs take three to four
# minutes, so 20 leaves ample room for a slow runner while turning a hang
# into a red build. This is the net, not the fix -- the cause is still
# open.
timeout-minutes: 20
# six-hour budget instead of failing. This is the net, not the fix --
# the cause is still open.
#
# 35, raised from 20 on 2026-10-29 together with the Unit-Tests step
# budget below; the derivation is there. This comment also read "Green
# runs take three to four minutes, so 20 leaves ample room for a slow
# runner", which had stopped being true of the job long before the image
# moved: the last green run on `macos-15` (38225383495, 2026-10-12,
# `c0762fac`) took 9 m 52 s end to end (08:44:12Z to 08:54:04Z),
# measured with
#
# gh run view 38225383495 --repo NoiXdev/macSCP --json jobs
#
# reading the job's own startedAt and completedAt. Three to four minutes
# was the figure for the TEST STEP in September, not for the job.
timeout-minutes: 35
env:
# ---------------------------------------------------------------------
# Compiler-warning budget. Raising it accepts a warning forever; read
Expand Down Expand Up @@ -72,6 +109,11 @@ jobs:
run: |
set -euo pipefail
xcodebuild -version
# Printed because inferring it from the Xcode version is what went
# wrong: Xcode 16.4 ships Swift 6.1.2, Xcode 27.1 ships 6.4, and a
# comment claiming one while the runner had the other cost two red
# runs on 2026-10-29.
swift --version
if xcrun metal --version; then
echo "Metal toolchain already present."
exit 0
Expand Down Expand Up @@ -120,20 +162,57 @@ jobs:
# by name (run 33698102652 lost 60 suite verdicts the same way). The
# step below reads the file back; the artifact keeps it.
#
# `timeout-minutes: 8` (docs/BACKLOG.md, "A test parked on a bare
# `timeout-minutes: 16` (docs/BACKLOG.md, "A test parked on a bare
# continuation outlives its time limit"): a test parked on a bare
# continuation ignores Swift Testing's own `.timeLimit` cancellation
# and would otherwise run on to the JOB's 20-minute budget above
# instead of failing near its own stated limit. Measured from the
# last three green `develop` runs before this step timeout was
# added (`gh run list --branch develop --limit 5 --json
# databaseId,conclusion`, then `gh run view <id> --json jobs --jq
# '.jobs[].steps[] | select(.name|test("Unit-Tests")) |
# "\(.startedAt) \(.completedAt)"'`): runs 33972030162, 33959532085
# and 33955366191 took 181 s, 230 s and 184 s — a measured maximum
# of 230 s (3 m 50 s). 8 minutes (480 s) is 2.09x that maximum,
# comfortably above the "at least 2x" floor.
timeout-minutes: 8
# and would otherwise run on to the JOB's budget above instead of
# failing near its own stated limit.
#
# It read `timeout-minutes: 8` until 2026-10-29, derived -- the
# withdrawn wording -- from "the last three green `develop` runs
# before this step timeout was added ... runs 33972030162,
# 33959532085 and 33955366191 took 181 s, 230 s and 184 s — a
# measured maximum of 230 s (3 m 50 s). 8 minutes (480 s) is 2.09x
# that maximum". That derivation was correct on 2026-09-05 and is
# correct arithmetic still; what expired is the maximum it rests on.
# Two runs measure it, each read with
#
# gh run view <id> --repo NoiXdev/macSCP --log
#
# which prints the two `Build complete! (<n>s)` lines (the Build
# step's, then `swift test`'s own compile of the test targets) and,
# on a run that reaches the end, the `Test run with <n> tests passed
# after <t> seconds` line -- which the killed run below never
# printed, hence its missing run figure:
#
# 38225383495, 2026-10-12, `c0762fac`, macos-15, GREEN:
# 240.40s build, 244.42s test-target compile,
# 6821 tests in 78.397 s -- whole step 329 s, i.e. 1.46x inside
# the 480 s budget, not the 2.09x above.
# 40933631584, 2026-10-29, `9c8d3673`, macos-latest, KILLED here:
# 230.46s build, 392.46s test-target compile. The first test did
# not start until 397 s (6 m 37 s) into the step, leaving 83 s
# of the 480 s for the whole suite; the kill landed 96 s after
# the first test started. How many tests the current tree has is
# not measured -- the run never reached a total.
#
# The two compile figures are NOT a toolchain comparison and must
# not be read as one. 28,721 lines of test code and 17,144 lines of
# source landed between those two commits:
#
# git diff --shortstat c0762fac..dcb5a414 -- Tests/
# git diff --shortstat c0762fac..dcb5a414 -- Sources/
#
# Note which way that cuts on the build half: Sources grew by those
# 17,144 lines and the Build step still came back 240.40s -> 230.46s.
#
# 16 minutes (960 s) is 2.04x of 392 s + 78 s = 470 s -- this
# image's measured compile plus the OTHER image's measured run,
# because this image's run figure does not exist yet: the step was
# killed before the suite finished. The first green run on this
# image replaces that addend and this number with its own
# measurement.
timeout-minutes: 16
run: |
set -o pipefail
swift test --event-stream-output-path "$RUNNER_TEMP/test-events.jsonl" \
Expand Down
5 changes: 5 additions & 0 deletions .github/workflows/release.yml
Original file line number Diff line number Diff line change
Expand Up @@ -126,6 +126,11 @@ jobs:
run: |
set -euo pipefail
xcodebuild -version
# Printed because inferring it from the Xcode version is what went
# wrong: Xcode 16.4 ships Swift 6.1.2, Xcode 27.1 ships 6.4, and a
# comment claiming one while the runner had the other cost two red
# runs on 2026-10-29.
swift --version
if xcrun metal --version; then
echo "Metal toolchain already present."
exit 0
Expand Down
73 changes: 72 additions & 1 deletion CLAUDE.md
Original file line number Diff line number Diff line change
Expand Up @@ -200,10 +200,58 @@ cooperative pool never gave the probe's body a thread, and a body that
never began did not overrun anything. The case asserts that the deadline
settled the row, which is the property it always existed for.

**2026-10-30 update: `.timeLimit` is a wall-clock ceiling too, and on a
saturated runner it fires at random.** Two runs of the gate on
`macos-latest`, two commits apart, same image, the suites these three
cases live in still bounded at `.minutes(1)`:

| case, bound 60 s | run 40965600726 | run 41030496532 |
|---|---|---|
| `cancelAllWithParallelRunners()` | passed after 70.097 s | failed after 100.325 s |
| `cancelAllCancelsQueuedAndRunning()` | passed after 72.676 s | failed after 100.335 s |
| `sendPreservesFIFOOrderUnderVaryingLatency()` | passed after 89.332 s | failed after 100.360 s |

All three ran past their own 60 s bound in BOTH runs and reddened only in
the second. Run 41030496532 recorded 148 `Time limit was exceeded: 60.000
seconds` issues across 73 tests in 11 suites; run 40965600726, which
reached the end of the suite, recorded none. Of the 74 tests that recorded
any issue in the later run, 73 recorded that one, so there was no second
cause — the one that did not is `ConnectMainActorLivenessTests`' own
warning-severity report, and that test passed.

Why the figures read like that: Swift Testing runs the suite in parallel
with no concurrency limit, the cooperative pool is as wide as the machine
has cores (three here), and all 7975 tests are in flight at once. So
`Test X passed after N seconds` is the run's wall clock as seen from X,
not X's own work. In run 41030496532, of 6911 cases carrying a duration
line, 6843 reported over 10 s, 6335 over 30 s and 1153 over 60 s, with 45
under 1 s, against a run total of 131.530 s. A bound below the run's own
duration therefore does not measure the test at all, and whether it fires
depends on whether the timer enforcing it gets a thread — the same
starvation the 2026-09-19 update above measured on `DetachedProbe`'s
`DispatchQueue.global()` timer.

So a `.timeLimit` has to sit ABOVE the whole suite's wall clock to be a
hang bound, and below the job's step budget to be worth having at all:
past the step budget the step is killed and the test's name is lost, which
is the `docs/BACKLOG.md` row "A test parked on a bare continuation
outlives its time limit". The 110 `.minutes(1)` bounds in `Tests/` were
raised to `.minutes(5)` on 2026-10-30 — 2.3x the 131.530 s run against
0.31x the 960 s step budget.

Two limits of that fix, both unmeasured and left alone deliberately. It
couples the bound to the suite's size: a run twice as long makes five
minutes marginal again, so the figure above is a measurement with a shelf
life, not a constant. And the 27 `.minutes(2)` bounds were not touched —
no run has recorded one firing, though 120 s is 0.91x this run's total,
which makes them the next candidates rather than a cleared set.

Rule: assert the outcome and the ordering a bound is supposed to
produce, not how long producing it took. A floor (`elapsed >= …`, "this
did not return early") is fine — a slow machine cannot defeat it, only a
broken one. A ceiling always can.
broken one. A ceiling always can — including the one a `.timeLimit`
states, which is a ceiling on a clock no assertion chose, and which the
2026-10-30 update above measures firing on healthy tests.

## Architecture invariants

Expand Down Expand Up @@ -435,6 +483,29 @@ own:
the thing it scans is there at all. Without one it is not a guard, it is
a comment that runs.

**This binds a check run once at a keyboard exactly as it binds one in
the suite**, measured 2026-10-30 on the CI warning gate. The gate's own
pattern was run against a saved `gh run view --log` to predict whether
`MAX_WARNINGS=0` would pass, which needs the log's
`<step>\t<stream>\t<timestamp> ` prefix stripped first, and the strip was
written `sed -E 's/^[a-z]*\t[^\t]*\t[0-9T:.Z-]+ //'`. Outside a bracket
expression `\t` is a tab; INSIDE one it is a backslash and the letter
`t`, so `[^\t]*` excluded `t` and could not cross a step named
`Unit-Tests`. Nothing was stripped, the following `^`-anchored grep
matched nothing, and nothing-matched was reported as no-warnings. The
log held the warning 25 times. One line separates the two readings, and
it is the positive control this rule asks for:

printf 'x\tUnit\tY\n' | sed -E 's/^x\t[^\t]*\tY$/STRIPPED/'
printf 'x\tUnit\tY\n' | sed -E 's/^x\t[^[:space:]]*\tY$/STRIPPED/'

The first prints the line unchanged, the second prints `STRIPPED`. Same
family as the `grep`/ugrep `$` case under point 3 below — a pattern that
silently matches nothing turns a check into an attestation — and the
same remedy: before believing a zero, feed the pattern something it MUST
match and watch it match. The corrected strip reproduces the figure the
gate itself printed, 1 unique location, where the original reported 0.

2. **A guard that spells a symbol it could read instead is waiting for a
rename.** Walk the name back from a call site, derive a catalogue key
from the type that owns it. A literal is a second copy of a name, and
Expand Down
2 changes: 1 addition & 1 deletion Tests/macSCPAppKitTests/AlreadyOpenSessionTests.swift
Original file line number Diff line number Diff line change
Expand Up @@ -31,7 +31,7 @@ import Testing
/// these tests leave in the strip is one of their own, built around
/// `StartRecorder`, so a start that DOES go ahead is observed instead of
/// dialed for real.
@Suite("Session already open", .timeLimit(.minutes(1)))
@Suite("Session already open", .timeLimit(.minutes(5)))
@MainActor
struct AlreadyOpenSessionTests {
private func makeTempDirectory(_ label: String) -> URL {
Expand Down
2 changes: 1 addition & 1 deletion Tests/macSCPAppKitTests/ArchivePresentationTests.swift
Original file line number Diff line number Diff line change
Expand Up @@ -6,7 +6,7 @@ import Testing

/// What the pane SAYS about an archive operation: one sentence per ending,
/// one per refusal, and none of them built from a tool's output.
@Suite(.timeLimit(.minutes(1)))
@Suite(.timeLimit(.minutes(5)))
struct ArchivePresentationTests {
@Test func aCancelAndASuccessSayNothing() {
#expect(ArchivePresentation.message(for: ArchiveActivity.Ending.finished) == nil)
Expand Down
2 changes: 1 addition & 1 deletion Tests/macSCPAppKitTests/BucketObjectLockSheetTests.swift
Original file line number Diff line number Diff line change
Expand Up @@ -371,7 +371,7 @@ struct BucketObjectLockSheetTests {

/// With a hang bound, for the reason given on the per-file sheet's twin of
/// this test (`ObjectLockSheetTests.aSecondRunWhileOneIsInFlightSendsNothing`).
@MainActor @Test(.timeLimit(.minutes(1))) func aSecondSaveWhileOneIsInFlightSendsNothing() async {
@MainActor @Test(.timeLimit(.minutes(5))) func aSecondSaveWhileOneIsInFlightSendsNothing() async {
let entered = AsyncSignal()
let release = AsyncSignal()
defer { release.signal() }
Expand Down
2 changes: 1 addition & 1 deletion Tests/macSCPAppKitTests/BucketVersioningSheetTests.swift
Original file line number Diff line number Diff line change
Expand Up @@ -293,7 +293,7 @@ struct BucketVersioningSheetTests {
/// while the versioning read has already returned; in that window the sheet
/// must still be `.reading`, not a settled sheet with no button. Only when
/// the lock answer is in does the load settle, with both.
@MainActor @Test(.timeLimit(.minutes(1)))
@MainActor @Test(.timeLimit(.minutes(5)))
func theVersioningAnswerDoesNotSettleBeforeTheLockAnswer() async {
let entered = AsyncSignal()
let release = AsyncSignal()
Expand Down
10 changes: 5 additions & 5 deletions Tests/macSCPAppKitTests/ComparisonFetchTests.swift
Original file line number Diff line number Diff line change
Expand Up @@ -639,7 +639,7 @@ struct ComparisonFetchTests {
/// an arrow key. Built from a real queue and a real cancel; every
/// postcondition is read BEFORE the parked read is let go, because the
/// read ending would make the queue look tidy whatever the cancel did.
@Test(.timeLimit(.minutes(1)))
@Test(.timeLimit(.minutes(5)))
func aDiffWhoseTaskIsCancelledMidFetchLeavesNoTransferRunning() async throws {
let root = try makeTemporaryRoot()
defer { try? FileManager.default.removeItem(at: root) }
Expand Down Expand Up @@ -725,7 +725,7 @@ struct ComparisonFetchTests {
///
/// Every postcondition is read BEFORE the parked read is let go: the read
/// ending would make the queue tidy whatever the cancel did.
@Test(.timeLimit(.minutes(1)))
@Test(.timeLimit(.minutes(5)))
func aSettleInFlightIsCancelledByTheComparisonsOwnCancel() async throws {
let root = try makeTemporaryRoot()
defer { try? FileManager.default.removeItem(at: root) }
Expand Down Expand Up @@ -784,7 +784,7 @@ struct ComparisonFetchTests {
/// started the same way and NOT cancelled, settles the row and reads both
/// sides. Without this, a `startTrackedWork` that cancelled every task it
/// was handed would pass the test above.
@Test(.timeLimit(.minutes(1)))
@Test(.timeLimit(.minutes(5)))
func aSettleNobodyCancelsSettlesTheRowAndReadsBothSides() async throws {
let root = try makeTemporaryRoot()
defer { try? FileManager.default.removeItem(at: root) }
Expand Down Expand Up @@ -812,7 +812,7 @@ struct ComparisonFetchTests {
/// cannot be satisfied by the other. Here the left file arrives and
/// finishes, the right read is parked, and the task is cancelled — which
/// is what selecting a row, waiting a moment and arrowing on does.
@Test(.timeLimit(.minutes(1)))
@Test(.timeLimit(.minutes(5)))
func aDiffWhoseTaskIsCancelledDuringTheRightDownloadLeavesNoTransferRunning() async throws {
let root = try makeTemporaryRoot()
defer { try? FileManager.default.removeItem(at: root) }
Expand Down Expand Up @@ -849,7 +849,7 @@ struct ComparisonFetchTests {
}

/// The click route to the same fetch.
@Test(.timeLimit(.minutes(1)))
@Test(.timeLimit(.minutes(5)))
func aSettleWhoseTaskIsCancelledMidFetchLeavesNoTransferRunning() async throws {
let root = try makeTemporaryRoot()
defer { try? FileManager.default.removeItem(at: root) }
Expand Down
2 changes: 1 addition & 1 deletion Tests/macSCPAppKitTests/ComparisonOverviewTests.swift
Original file line number Diff line number Diff line change
Expand Up @@ -24,7 +24,7 @@ import macSCPCore
/// `en.lproj` separately, by reading the file.
///
/// No test asserts how long anything took.
@Suite("Comparison overview", .timeLimit(.minutes(1)))
@Suite("Comparison overview", .timeLimit(.minutes(5)))
@MainActor
struct ComparisonOverviewTests {
private typealias Presentation = ComparisonOverviewPresentation
Expand Down
2 changes: 1 addition & 1 deletion Tests/macSCPAppKitTests/ComparisonOwnershipTests.swift
Original file line number Diff line number Diff line change
Expand Up @@ -25,7 +25,7 @@ import Testing
/// No test here asserts how long anything took. A walk parked on a peer is
/// modelled by a call that waits until it is cancelled; the suite's hang
/// bound is `.timeLimit`.
@Suite("Comparison ownership", .timeLimit(.minutes(1)))
@Suite("Comparison ownership", .timeLimit(.minutes(5)))
@MainActor
struct ComparisonOwnershipTests {

Expand Down
2 changes: 1 addition & 1 deletion Tests/macSCPAppKitTests/ComparisonWindowModelTests.swift
Original file line number Diff line number Diff line change
Expand Up @@ -22,7 +22,7 @@ import Testing
/// Both sides are the real local file system over temporary directories, so
/// a walk finishes without a network. No test asserts how long anything took;
/// the suite's hang bound is `.timeLimit`.
@Suite("Comparison window model", .timeLimit(.minutes(1)))
@Suite("Comparison window model", .timeLimit(.minutes(5)))
@MainActor
struct ComparisonWindowModelTests {

Expand Down
2 changes: 1 addition & 1 deletion Tests/macSCPAppKitTests/ConnectAttemptHandoffTests.swift
Original file line number Diff line number Diff line change
Expand Up @@ -63,7 +63,7 @@ import Testing
/// from the byte comparison — parsed via `SessionStore`'s own real reader,
/// which is a READ, not a write, and therefore safe regardless of what a
/// broken seam did.
@Suite("Connect attempt hand-off", .timeLimit(.minutes(1)))
@Suite("Connect attempt hand-off", .timeLimit(.minutes(5)))
@MainActor
struct ConnectAttemptHandoffTests {
/// The ONE real, on-disk file this whole suite must never write —
Expand Down
2 changes: 1 addition & 1 deletion Tests/macSCPAppKitTests/DetailSurfacePlanTests.swift
Original file line number Diff line number Diff line change
Expand Up @@ -26,7 +26,7 @@ import Testing
/// raises the text is `FormFailureAlertPlanTests`' claim; that the view
/// switches on this answer and on nothing else is
/// `DetailSurfaceWiringGuardTests`'.
@Suite("Detail surface plan", .timeLimit(.minutes(1)))
@Suite("Detail surface plan", .timeLimit(.minutes(5)))
@MainActor
struct DetailSurfacePlanTests {
private static let selected = StoredSession(
Expand Down
Loading
Loading