feat(cli): really stop a job that is already executing [PC-4873]

[robert-ursu]

Draft — stacked on #1835. Base is feat/async-job-dispatch-and-result-push, so the diff here is only the cancellation work. Review and merge #1835 first.

Paired with UiPath/hdens#7708.

Summary

StopJob used to be able to do nothing but refuse a running job: its body runs on a thread via asyncio.to_thread, which cannot be cancelled.

That reasoning missed a fact. run/debug/eval each drive their own event loop with asyncio.run inside the worker thread (cli_run.py:333, cli_debug.py:269, cli_eval.py:531) — so a job is an event loop, and an event loop can be cancelled.

The three call sites now go through run_job_loop, which publishes that loop and its root task to a JobControl carried on a ContextVar. asyncio.to_thread propagates contextvars, so no monkeypatching is needed; outside the server (uipath run on a terminal) it is asyncio.run verbatim.

Cancellation is cooperative, not a thread kill: the runtime's context managers still unwind, so UiPathRuntimeContext.__exit__ still writes output.json and the caller's file fallback still works. A test asserts the job's finally actually ran.

The stop ladder

1. Cancel the root task only, wait STOP_GRACE_SECONDS. Cancelling every task would land a second CancelledError inside the cleanup that writes output.json and abort it mid-write.

2. If cleanup itself is stuck, sweep the loop and wait a shorter window.

3. Otherwise return False — a job wedged in a non-cancellable C call (a socket read inside an LLM request) cannot be stopped, and saying so beats claiming a stop that did not happen.

Stopped, not Faulted

The runtime records a cancelled job as FAULTED/ERROR_CancelledError, which is the wrong story for a stop the caller asked for. The result push carries an explicit stopped flag that wins over the document.

Stop is reachable now

POST /jobs/{job_key}/stop on the HTTP transport — which carries all current traffic and previously had no route to the registry at all. Stop existed only over uipath-ipc, and the pooled server is launched without --ipc-pipe.

Worth a careful look

• _invoke_command discriminates the two CancelledErrors via Task.cancelling(): 0 means the job's own loop was cancelled (an outcome — swallow it, so the lock unwinds and the result document is still read); >0 means our awaiting task was cancelled (a shutdown — re-raise, or the shutdown stalls).

• StopJob is wired to the registry. The StopJobRequest DTO and its ResumeVersion/ForceStop fields already landed on main in feat(cli): carry ResumeVersion and ForceStop on the IPC job DTOs [ROBO-5779] #1838 and were a no-op stub; this change makes them do something. A suspended job resumes under the same key, so ResumeVersion is what keeps a stop aimed at the previous run from killing the resumed one.

  (I had originally argued for a trailing optional parameter over a DTO — moot now that the .NET peer sends StopJobRequest; interop wins.)

Testing

tests/cli/test_server_cancellation.py (9 tests) uses commands shaped like the real ones — a click command whose body is run_job_loop(...) — and asserts a running job really stops, its cleanup runs, the process-wide lock is released for the next job, and an uncancellable job reports False. Full tests/cli suite green.

Known gaps

• A job blocked in a C call cannot be interrupted; reported honestly rather than papered over.

• No rung-3 hard exit: taking the process down would kill queued jobs too.

🤖 Generated with Claude Code

Review follow-ups

Two defects found reviewing this branch (Codex GPT-5.6), each fixed in its own commit with a test verified to fail beforehand.

A repeated stop aborted the cleanup. JobControl.cancel() documented that it must be called at most once, but JobRegistry.stop() called it unguarded on every request. A stop followed by a force-stop escalation is ordinary, and the second delivery lands inside the runtime's cleanup finally blocks — the ones that write output.json — and aborts them, destroying the fallback the caller relies on. Absorbing the repeat belongs in JobControl, not in every caller.

Self-cancellation was reported as a user-requested stop. _invoke_command treated any CancelledError with cancelling() == 0 as our stop landing. cancelling() reports on the awaiting server task, so a CancelledError the job's own code let escape came back as exit 143 "stopped on request" with no StopJob in sight — a fault filed as a clean stop. The classification is now gated on the control's own cancel_requested; anything else is reported as a failure.

Jira

PC-4873