FW16 (Ryzen AI 300 + RTX 5070): EC charge/power state machine latches a system-wide slow state; only AC re-plug clears it #1

Summary

Framework Laptop 16 (Ryzen AI 300) with the RTX 5070 GPU module intermittently drops into a whole-system slow state during combined CPU + dGPU load (gaming with video playback on a second monitor) while charging over a 240W EPR contract. The slow state persists across OS reboot — the BIOS itself runs slow — and clears only after unplugging AC for ~10 seconds (battery stays in, so the EC never loses power: an AC re-plug / renegotiation is what fixes it, not a power cycle). Separately, a USB-C port can drop into a state where a charger attach is detected but no PD contract ever forms (EC reports "AC not connected", input limit falls back to 800mA), and resetting the PD controllers does not clear it — the state lives in EC RAM and re-imposes itself. Evidence below points at the EC charge/power state machine (EPR entry/exit + throttle release paths) rather than the GPU driver, thermals, or the PD silicon.

System

• Framework Laptop 16, AMD Ryzen AI 300 ("tulip"), RTX 5070 GPU module (expansion bay reports DualInterposer / NvidiaGn22, EEPROM HW 4.0)
• BIOS 04.01 (latest stable), EC tulip-4.0.1-9b2a0b1, PD controllers 0.0.22 (all MainFw), framework_tool 0.6.5
• Charger negotiates 48V / 5A (240W EPR) — verified via EC_CMD_GET_PD_PORT_STATE (0x3E23)
• Continuous telemetry: 5s CSV (battery/CPU/NVML), 15s EC polls (--power -vv, --pdports, --thermal, host commands 0x3E22/0x3E23/0x3E18/0x3E1C, h_prochot_l GPIO), continuous EC console capture (--console follow). Full data repo (all raw telemetry, incident captures, and analysis docs) is public: https://github.com/tyeth/fw16-gpu-watchdog

Incidents captured with full telemetry

Incident A — 2026-09-03 22:01:33. Gaming at full performance (dGPU ~99W / 2.4GHz, CPU 100% max frequency, temps 85–91°C) with battery assist active (3–5.5W discharge while AC connected, battery at 99–100% "not accepting current"). Hard stop at 22:01:33 with no thermal or clock degradation visible in the preceding samples. EC host event BATT_BTP (battery trip point) fired at onset; the charge port re-sent its PD identity around the same time (30s-deferred P4: Safe mux transition follows in the EC console — the charger is on the rear/expansion-bay port). Slow BIOS after reboot; cleared by AC unplug. No display driver events (no 4101, no nvlddmkm entries) in 30 days of logs.

Incident B — 2026-09-03 23:31. Same load profile. Two EC→SMU SB-RMI mailbox timeouts mid-session (SB-SMI: Mailbox transfer timeout / SB-RMI Error: 4) at ~23:10 and ~23:17, before the 87°C-family thermal WARN at ~23:22. Soft thermal throttle asserted (0x3E22 soft mask = 0x01), hard mask 0x00, h_prochot_l never asserted, 48V/5A contract intact right up to the event. These mailbox timeouts recur on this unit even at idle; note SoftwareFirmwareIssueTracker FrameworkComputer#160 (same error strings, acknowledged as an EC↔APML communication issue).

Incident C — 2026-09-04 00:15, with a live stuck-port reproduction afterward. After this event, a left-side port entered the attach-without-contract state: Type-C attach detected (port reports Sink role), no PD contract, EC_CMD_GET_PD_PORT_STATE for the port returns all zeros, EC charger status says "AC is: not connected", input limit 800mA — while the same controller's other port kept running a DP monitor normally, and the rear port negotiated 240W fine. Diagnostics performed in the stuck state:

• --pd-reset of the affected controller: contract on the monitor port re-established; the stuck port stayed dead.
• --pd-reset of both side controllers: same result.
• Charger unplug/replug into the stuck port (fresh attach on freshly reset silicon): still no contract.
• All three controllers reported MainFw throughout — the silicon was healthy; EC RAM state re-imposed the dead state after every reset.
• Only a full AC-removal sequence cleared it. Days later the same port negotiates the full 240W EPR contract, so this is not hardware damage.

EC source correlation (branch fwk-tulip-29169)

• cypd_ccg8.c: EPR enter/exit deliberately sets charger input to ~0/500mA and asserts a hard AP throttle while renegotiating; an explicit failure path latches the port (epr_active = 0xff, comment "EPR fail, do not retry") until physical replug.
• lotus/src/cpu_power.c: safety ladder escalates on battery discharge up to a hard PROCHOT GPIO (clamps the SoC including in BIOS); ladder state and PMF event mask are EC RAM; the throttle release path depends on a successful SB-RMI limit write (clear_prochot() after set_pl_limits), i.e. on the same mailbox that produces SB-RMI Error: 4 on this unit.
• PD_PROGRESS_DISCONNECTED handling is what clears the progress/event state on a real AC disconnect — consistent with "only an AC re-plug fixes it".

Related existing reports: SoftwareFirmwareIssueTracker FrameworkComputer#160 (SB-RMI timeouts, acknowledged), FrameworkComputer#243 (AI 300 locked to minimum clocks across warm reboots until EC power event), FrameworkComputer#193 (FW16 AI 300 + RTX 5070: 240W EPR charging problems), FrameworkComputer#241 (FW13 AI 300: firmware ignores P-state requests until PSU replug).

Healthy reference trace (2026-09-10)

A charger replug on the rear port was captured end-to-end in the EC console while everything was working, for comparison with the failure mode: zero driver or device errors in Windows logs around the window (no nvlddmkm entries, no device removal), expansion bay reports "No fault", and the EC ring buffer captured a textbook-healthy 240W negotiation at the replug — including a detail worth keeping for the record: during the high-wattage (EPR) mode entry the EC asserts its hard throttle and forces input to 500mA for ~1.1 seconds, then releases (set AP throttling type 1 to on → CL: p4 s0 i500 v20000 (forced) → SPR contract → Entered EPR → CL: p4 s0 i5000 v48000 → set AP throttling type 1 to off). That is the healthy version of the exact sequence that, when it fails partway, produces the latched slow state: throttle asserted + input collapsed is the intended transient, and the failure mode is that transient becoming permanent.

Tool-side findings (this repo), happy to file separately / PR

1. FW16 sensor labels are misassigned in --thermal (verified via EC_CMD_TEMP_SENSOR_GET_INFO 0x0070): id1 charger_f75303 prints as "F75303_CPU", id2 apu_f75303 prints as "F75303_DDR", and the three dGPU F75303 labels are shifted by one (gpu_amb→"dGPU VR", gpu_vr→"dGPU VRAM", gpu_vram→"dGPU AMB").
2. --pdports only prints ports 0–3 — on FW16 the rear/expansion-bay port (the port most people charge on) is invisible; raw --host-command 0x3E23 0 4 works.
3. --smartbattery blocks forever without a console (unseal-key prompt reads stdin) — it permanently hung a hidden scheduled task until identified; a non-interactive guard would help.

Ask

• EC: make the EPR entry/exit and throttle-release paths robust to a failed step (retry instead of the epr_active = 0xff latch; a watchdog that restores input current / releases the throttle if the sequence doesn't complete; recovery that doesn't require physical AC removal).
• EC: expose the latch/ladder state read-only (the 0x3E22 throttle-source masks were invaluable; a "why is input current limited" query would complete the picture).
• Tool: items 1–3 above.