RTOP-302: Stop the PLC when a task is stuck in its scan

[thiagoralves]

Problem

RTOP-302: a PLC task stuck in its scan (for example an IEC WHILE loop that never exits) could not be stopped. The dispatcher only logged overruns. A STOP joined the worker with no timeout and wedged the runtime in TRANSITIONING_TO_STOP (every command answered COMMAND:BUSY) until a 120 s fallback forced ERROR with the stuck threads still alive. On a single core the spinning SCHED_FIFO task starved everything else. Outputs also kept their last value on every stop, and IEC task priorities were inverted (IEC 0, the highest, ran at FIFO 1).

What changed

• Stuck-task watchdog (dispatcher). A task found still in the same scan on 10 consecutive due ticks (10 of its own periods) trips the dispatcher; the count resets whenever the task is found idle, and ticks replayed after a late dispatcher count as missed periods too. A task's first scan (initialisation, e.g. a Python FB starting) is not counted; the main watchdog trips it only past 10 s (PLC_FIRST_SCAN_TIMEOUT_MS). The dispatcher claims the stop, stops releasing scans, and lets each in-flight scan run until 10 periods after its release (10 s for a first scan). A task still inside IEC program code then gets SIGUSR2, whose handler siglongjmps to the existing per-thread recovery point. The unload runs afterwards on a SCHED_OTHER thread, and any stop that had to abort a task lands in ERROR.
• Outputs off on every stop. After the drain, the journal is closed so plugin writes (HMI, s7comm) can't re-energise %Q, every %Q image slot is zeroed, synchronous plugins get one last cycle_start/cycle_end, and the zeroed image is held 500 ms (PLC_OUTPUTS_OFF_SETTLE_MS) for plugins that poll from their own threads.
• Last resort: _exit(42). This happens when an aborted task does not exit within 2 s, a stop overruns its budget (max(10 × the longest task interval, 10 s) + 30.5 s), or the dispatcher stops ticking for max(10 base ticks, 1 s). The fatal path logs without taking a lock (log_emergency) and writes a marker (/run/runtime/watchdog_fault). The webserver restarts with --safe-mode --fault on exit 42, and the next boot also consumes the marker, so the runtime comes back in safe mode reporting ERROR even when the supervisor can't read the exit code. That boot, before the command socket opens, claims a stop, starts the configured plugins (including VPP board plugins), drives all outputs to 0 and stops them. If that step itself hangs, the next exit records it in the marker and the following boot skips it, relying on the hardware safe-state watchdog. Safe mode then survives later crashes until an upload clears it.
• Priorities. Watchdog FIFO 99, dispatcher FIFO 98. IEC priority is clamped to 0..48 and mapped to FIFO 49 − priority, with a warning when clamped.
• Priority inversion. Runtime mutexes are re-initialised with PTHREAD_PRIO_INHERIT, the retain store uses a PI RtMutex, and state reads are lock-free.
• Teardown hardening. Program unload is serialised against a concurrent shutdown. The Python block loader blocks the abort signal around its fork/shm/stdio work. The unused per-task heartbeat and the dead plc_io_cycle.{cpp,h} are removed.
• Docs. "Watchdog System" in docs/ARCHITECTURE.md, --fault in docs/DEVELOPMENT.md, and the lifecycle README coverage notes.

Release note

• Task scheduling changes for existing projects. A task with IEC priority 1 moved from FIFO 1 to FIFO 48.
• New trip condition. A lower-priority task delayed by higher-priority tasks for 10 of its own periods now trips the watchdog, where before it only ran at a reduced rate.

How it was verified

There is no PR-triggered C build or test in CI, so everything below was run locally.

Unit

• tests/host/run.sh (all pass):
  • test_rt_mutex: the PI protocol is set, with a negative control.
  • test_task_policy: priority mapping and clamping.
  • test_image_outputs: every %Q slot is zeroed and inputs/memory are untouched. The test fails when the zeroing call is removed.
• tests/pytest/runtimemanager/test_watchdog_exit.py: exit 42 → safe mode with --fault on the first exit; other exits keep the rapid-crash rule.
• CI pytest set: 152 passed, 7 skipped. Plugin step: 26 passed.
• pylint 10/10; black, isort and ruff clean on the changed Python.
• clang-format 21.1.0 on the new or changed .c/.h lines. Files the hook would fully rewrite were left unformatted.

Lifecycle suite (tests/lifecycle, Docker, a real compiled program): 24/24 passed, 0 build warnings.

End to end. Real projects were uploaded with openplc-cli upload to a Docker runtime built from this branch, with each fixture's source and the compiled loop checked before reading the result:

Scenario	Result
Original Opta Demo (while magicValue < 10 loop; spin confirmed in the uploaded .so)	trips 10 periods (200 ms) after it locks, program unmapped, ERROR
Two tasks on a shared global, fast one spinning (IEC −5 and 100)	ERROR every run; in one run the slow task was blocked on a global's mutex held by the aborted task, got its grace, was aborted, ERROR at 1.8 s; IEC 100 → FIFO 1 with a warning
Task taking ~8 periods per scan, 30 s	no trips
STOP while a 2 s task is stuck	ERROR at 16 s (10 periods after the stuck scan's release)
Remote Modbus TCP coil held TRUE	device received False on both a normal stop and a fault stop; stayed True on the commit before the output change (negative control)
Abort swallowed (LD_PRELOAD shim)	FATAL logged, marker written, exit 42, restarted as --safe-mode --fault, ERROR
Dispatcher hung in a plugin hook	stall detected at ~1.05 s, exit 42, safe mode, ERROR
Marker present, plain restart (exit code not visible)	boots in safe mode, ERROR
Healthy Opta Demo	runs, stops, restarts, no trips
10 ms task with a ~1 s first scan	before review fixes: ERROR at 100 ms; now runs, no trip
First scan that never ends	trips at 10 s ("first scan still running after 10000 ms"), ERROR
Python FB on a 1 ms task	runs; Python child started (the loader stayed under 10 ms on this host)
Modbus slave HMI writing the coil during the 500 ms settle (unit 1)	before: device False then True again; now stays False
Exit 42 then safe-mode boot, coil held TRUE	before: device stays True; now goes False ~1 s after the exit
Safe-mode boot outputs-off hung by a plugin	second exit records the context; next boot skips it, ERROR, no loop
Single CPU (--cpuset-cpus=0): spin, shared globals, Modbus coil, STOP while stuck	all pass; the webserver is starved while a task spins, but the dispatcher and watchdog still act

Manual (developer): tested with the Opta Demo project against a runtime built from this branch; reported working.

Not run / out of scope

• Ceedling. It fails on development as well: project.yml excludes SOEM paths that are missing, and tests/support/ethercat_stubs.c needs soem/soem.h. Not changed here.
• Mutexes that can't take priority inheritance (STruC++ GlobalVar std::mutex, the Python GIL, third-party plugin locks). A hang on one of them ends in _exit(42).
• Boot outputs-off timing. It runs before the command socket opens, so the runtime is unreachable during it (about 1 s normally, up to the 40.5 s budget if a plugin hangs).
• Overlap with open external PR fix(runtime): use progress counter for watchdog monitoring #208 (watchdog progress counter). It touches the same heartbeat code and will conflict.

No requirements document: this is a bug fix, and the Jira task holds the description and acceptance criteria. No cybersecurity risk assessment: the change touches local thread scheduling, the PLC stop path and the webserver's handling of the runtime exit code. No network exposure, authentication, cryptography, update mechanism or external file parsing is affected.

🤖 Generated with Claude Code

https://claude.ai/code/session_01FWp9HZHNzjr7A2VLviZrg5