Skip to content

Re-arm tripwire flake: onCommandComplete fires before console.read() re-arm lands, so a synchronous handler-count assert is racy #694

Closed

Description

@stalep

Summary

Quarkus CI flaked once on the aesh 3.18.6 / readline 3.18.7 bump: AeshLauncherApiTest.accumulatedOutputAcrossCommands failed with the re-arm tripwire reporting only 1 handler replacement since the previous completion (expected >= 2) — even though the REPL was alive, parked in openBlocking, and rearmedSinceLastCompletion=true.

Verdict after analysis: false positive from a race in the Quarkus test harness, not a missed re-arm and not a 3.18.6 regression. Filing here to record the analysis and to ask whether upstream wants a deterministic ordering guarantee (see below).

Where / when

java.lang.AssertionError: Command 'hello --name=Second' completed but readline was not re-armed afterwards: 1 handler replacement(s) since last completion (expected >= 2 or console close). Diagnostics (re-arm tripwire): replAlive=true, replState=WAITING, queuedSignals=0, readerDeath=null, lastArmNanos=197132876545, closeNanos=0, rearmedSinceLastCompletion=true, threads=19 [aesh-test-repl(WAITING): Unsafe#park <- ... <- StreamConnection#openBlocking <- ReadlineConsole#accept ... aesh-stream-reader(RUNNABLE): ProcessManager#drainWhilePending <- ProcessManager#requestDrain <- ProcessManager#execute <- ReadlineConsole#processLine]

The follow-up rerun error in the same fork (REPL did not start... replAlive=false TERMINATED) is cascade fallout, not a second bug.

Analysis

Completion-to-signal order in 3.18.6 (CommandJob.finish(), CommandJob.java:420-428):

  1. manager.processFinished(job) — clears activeJob, fires pipeline events, requestDrain(). The post-completion re-arm (console.read(), ProcessManager.java:461) runs inside that drain turn: inline on the worker thread if it wins the turn, or deferred to another thread.
  2. Only then executionListener.onCommandComplete(...) — the signal the Quarkus harness (AeshLauncherImpl.execute()) waits on before asserting the handler count synchronously.

So whether the current command's re-arm has landed at assert time is a thread race: healthy runs show a delta of 2 (submit stub + fast inline re-arm) or 1 (stub only, re-arm still in flight — it lands moments later, as proven here by WAITING in openBlocking plus rearmed=true at diagnose time). The >= 2 synchronous assertion is racy by construction.

Not a 3.18.6 regression: git diff v3.18.4..v3.18.6 shows no ProcessManager/readline read-path change (only context/completion/cursor-listener work), and the readline v3.18.5..v3.18.7 diff touches no handler logic. A timing shift merely exposed the race.

Proposed plan

  1. Harness-side fix in Quarkus (no upstream change required): replace the immediate assert in AeshLauncherImpl.execute() with a bounded settle wait (up to ~2 s, short poll) for armCount - baseline >= 2 or console close; fail with the existing message + diagnostics only on timeout. Keep the threshold at 2 — relaxing to >= 1 would mask a genuine Second command over piped streams never completes (3.18.1 regression) #634-style skip, since the submit stub alone already yields 1.
  2. Verify by looping AeshLauncherApiTest locally (~20x) on 3.18.6, then via CI.
  3. Optional upstream question: would you accept a guarantee that onCommandComplete fires strictly after the post-completion console.read() re-arm lands (or a dedicated re-arm hook), so downstream harnesses can assert re-arm deterministically instead of polling? Happy to implement the harness-side wait regardless; this would just remove the need for it.

Activity

  1. stalep commented on Oct 24, 2026

    @stalep
    MemberAuthor

    Verified the mechanism from this side — the analysis is correct. CommandJob.finish() calls manager.processFinished(job) (which only requests the drain turn) and then fires onCommandComplete synchronously, while the re-arm (console.read()) lands whenever the drain turn runs, possibly on another thread. So at listener time the re-arm provably may not have landed: racy by construction, not a regression. (Also checked: no scheduling/read-path changes in aesh/.../console since v3.18.5 that could have introduced this — only the #689 context selection and a test-only seam.)

    On the plan:

    1. Agree the harness-side bounded settle-wait is the right fix, including keeping the >= 2 threshold — relaxing to >= 1 would indeed mask a Second command over piped streams never completes (3.18.1 regression) #634-style skip, since the submit stub alone already yields 1. Expect recurrence on loaded runners until it's in; with a race this tight, no code change on either side is needed to flip it.
    2. Verified the "not a regression" claim as far as this repo goes (see above).

    On the upstream question (ordering guarantee): I'd advise against it, for one contractual reason and one architectural one.

    • Contractual: CommandExecutionListener documents "Called after each command finishes execution, before the prompt is redisplayed." A guarantee that the listener fires strictly after re-arm lands would contradict the documented contract — it would need a contract change, not just an implementation tweak.
    • Architectural: today failure/completion reporting is prompt and independent of readline liveness. A strict ordering couples every completion signal to re-arm landing; if re-arm ever stalls (wedged reader, console closing), completions would never report unless the rendezvous has a timeout — at which point the timeout needs a policy for "give up and fire anyway" (guarantee broken) or "swallow" (lost signal). Either way the flakiness just moves into aesh with higher stakes.

    The race is inherent: an async completion signal vs. a subsequent re-arm that may run on another thread cannot be made deterministic for a synchronous observer without a rendezvous. The bounded wait belongs in the asserter, which is exactly plan item 1.

    If a second consumer ever needs determinism (so far this is test-instrumentation-only — production consumers are all result-oriented, not re-arm-oriented), I'd rather add an explicit, opt-in re-arm signal than reorder the existing event. Not needed now.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions