spi-hid-apple-of: internal trackpad sporadically wedges (IRQ storm or fully silent), degrades until only a cold power cycle recovers it
Environment
|
|
| Machine |
MacBook Pro 14" M1 Max 2021 (apple,j314c) |
| Distro |
Fedora Asahi Remix 44 |
| Kernel |
7.1.6-400.asahi.fc44.aarch64+16k (also reproduced on 6.19.14 before any recent upgrades) |
| Device |
spi1.0 — spi-hid-apple-of, HID 001C:05AC:0342, input via magicmouse |
| Suspend |
no suspends involved — wedges occur mid-session on an always-on machine |
Symptom
The internal trackpad sporadically stops producing input entirely (libinput debug-events shows nothing; reading /dev/input/eventN for the trackpad returns zero bytes while touching it). The internal keyboard, on the same SPI device (spi1.0) and the same controller (39b10c000.spi), keeps working throughout — so this is a per-HID-function wedge, not a controller-wide failure.
Two distinct failure modes, both observed repeatedly:
Mode A — IRQ storm with zero reports. While sliding a finger on the dead trackpad, the controller IRQ line increments at ~900–1500 interrupts/s, but no input report ever arrives. /proc/interrupts deltas over a 3 s window: 39b10c000.spi +2538, spi-hid-apple-irq (GPIO) +151, trackpad evdev bytes 0.
Mode B — fully silent. Sliding produces zero IRQ increments on both the controller line and the HID GPIO line, and zero bytes on the event node.
This is the same symptom class as the April 2023 report at https://old.reddit.com/r/AsahiLinux/comments/133fw0b/ where marcan confirmed a known bug that was fixed in the May 2023 kernel, with the then-workaround being a magicmouse reload. The behaviour looks like a recurrence/regression of that bug (or a close variant) on current kernels.
Kernel log signature
Repeated during wedges and rebind attempts (full log in the gist below):
spi-hid-apple-of spi1.0: status message mismatch: 9c 17 e5 f8
spi-hid-apple-of spi1.0: waiting for get report failed: -110
The mismatch bytes are identical every time (9c 17 e5 f8); the driver expects ac 27 68 d5. On rebind the trackpad re-probes fine (dimensions are even reported correctly on recent boots), so this is not a probe-time issue.
Recovery attempts and session degradation
Rebinding the SPI device (echo spi1.0 > .../spi-hid-apple-of/{unbind,bind}) reliably revived it at first, but recovery lifetime degraded monotonically over one session (all timestamps today, single boot):
- Wedges initially every 10–40 min; device rebind recovers for tens of minutes.
- Later, trackpad only lives for a few seconds after each rebind (verified: 5 consecutive 5 s reads of the event node, all 0 bytes, while sliding).
- Controller-level rebind (
39b10c000.spi unbind/bind on apple-spi) stops helping — the IRQ storm persists through it (measured 1095/s before → 618/s after controller rebind, still no reports).
- Full power-domain bounce (unbind controller, 10 s down, rebind, rebind device) no longer helps either — device re-probes, HID sessions increment, still zero events.
- A wedge was also once preceded by the cursor freezing pinned at the bottom edge of the screen (last report delivered was a touch at the bottom; pointer never moved again).
HID session numbering for one boot climbed from 001C:05AC:0342.0001 to .0041 (65 transport re-inits in 77 minutes; dmesg shows 32 fresh "Apple SPI Trackpad" probes plus keyboard re-probes sharing the counter). One spontaneous self-recovery was also observed once (storm stopped on its own without any rebind).
At the time of writing, only a cold shutdown (power off, SPI rails down, 30 s wait) is left to test — reboot alone is not expected to reset the touchpad controller if its power domain survives.
Full diagnostics (dmesg filtered to spi/hid, /proc/interrupts, self-healing tool log showing the degradation timeline):
https://gist.github.com/iamcheyan/c8e89f27b4e8424acaea57ca6655ae90
Notes
- Reproduced before and after today's
dnf upgrade (1848 packages, kernel packages only — m1n1/u-boot/DT untouched), across 6.19.14 and 7.1.6, so it does not look like a recent-kernel-only regression. Frequency increased noticeably after the upgrade session, but wedges predate it.
- Detection methodology (in case anyone reproduces): compare
/proc/interrupts deltas for 39b10c000.spi against bytes read from the trackpad event node while sliding. Caveat: keyd (EVIOCGRAB) holds the physical keyboard node, so reading it yields 0 bytes even while typing — use the keyd virtual node or IRQ counters as the keyboard-liveness signal, otherwise the "keyboard also dead" conclusion is a false positive.
- Happy to run additional instrumentation (debug patches, SPI tracing) on this machine — it currently reproduces within minutes.
Expected
Trackpad transport should not wedge mid-session; if the device stops responding, the driver should detect the status-mismatch/timeout pattern and self-recover (the spien GPIO power-on reset sequence in spihid_apple_of_power_on() is what makes the manual unbind/bind work, but it appears increasingly ineffective within a single boot, suggesting residual controller/firmware state that survives the 60 ms reset but not a cold power cycle).
spi-hid-apple-of: internal trackpad sporadically wedges (IRQ storm or fully silent), degrades until only a cold power cycle recovers it
Environment
apple,j314c)7.1.6-400.asahi.fc44.aarch64+16k(also reproduced on6.19.14before any recent upgrades)spi1.0—spi-hid-apple-of, HID001C:05AC:0342, input viamagicmouseSymptom
The internal trackpad sporadically stops producing input entirely (
libinput debug-eventsshows nothing; reading/dev/input/eventNfor the trackpad returns zero bytes while touching it). The internal keyboard, on the same SPI device (spi1.0) and the same controller (39b10c000.spi), keeps working throughout — so this is a per-HID-function wedge, not a controller-wide failure.Two distinct failure modes, both observed repeatedly:
Mode A — IRQ storm with zero reports. While sliding a finger on the dead trackpad, the controller IRQ line increments at ~900–1500 interrupts/s, but no input report ever arrives.
/proc/interruptsdeltas over a 3 s window:39b10c000.spi+2538,spi-hid-apple-irq(GPIO) +151, trackpad evdev bytes 0.Mode B — fully silent. Sliding produces zero IRQ increments on both the controller line and the HID GPIO line, and zero bytes on the event node.
This is the same symptom class as the April 2023 report at https://old.reddit.com/r/AsahiLinux/comments/133fw0b/ where marcan confirmed a known bug that was fixed in the May 2023 kernel, with the then-workaround being a magicmouse reload. The behaviour looks like a recurrence/regression of that bug (or a close variant) on current kernels.
Kernel log signature
Repeated during wedges and rebind attempts (full log in the gist below):
The mismatch bytes are identical every time (
9c 17 e5 f8); the driver expectsac 27 68 d5. On rebind the trackpad re-probes fine (dimensions are even reported correctly on recent boots), so this is not a probe-time issue.Recovery attempts and session degradation
Rebinding the SPI device (
echo spi1.0 > .../spi-hid-apple-of/{unbind,bind}) reliably revived it at first, but recovery lifetime degraded monotonically over one session (all timestamps today, single boot):39b10c000.spiunbind/bind onapple-spi) stops helping — the IRQ storm persists through it (measured 1095/s before → 618/s after controller rebind, still no reports).HID session numbering for one boot climbed from
001C:05AC:0342.0001to.0041(65 transport re-inits in 77 minutes; dmesg shows 32 fresh "Apple SPI Trackpad" probes plus keyboard re-probes sharing the counter). One spontaneous self-recovery was also observed once (storm stopped on its own without any rebind).At the time of writing, only a cold shutdown (power off, SPI rails down, 30 s wait) is left to test — reboot alone is not expected to reset the touchpad controller if its power domain survives.
Full diagnostics (dmesg filtered to spi/hid, /proc/interrupts, self-healing tool log showing the degradation timeline):
https://gist.github.com/iamcheyan/c8e89f27b4e8424acaea57ca6655ae90
Notes
dnf upgrade(1848 packages, kernel packages only — m1n1/u-boot/DT untouched), across 6.19.14 and 7.1.6, so it does not look like a recent-kernel-only regression. Frequency increased noticeably after the upgrade session, but wedges predate it./proc/interruptsdeltas for39b10c000.spiagainst bytes read from the trackpad event node while sliding. Caveat:keyd(EVIOCGRAB) holds the physical keyboard node, so reading it yields 0 bytes even while typing — use the keyd virtual node or IRQ counters as the keyboard-liveness signal, otherwise the "keyboard also dead" conclusion is a false positive.Expected
Trackpad transport should not wedge mid-session; if the device stops responding, the driver should detect the status-mismatch/timeout pattern and self-recover (the
spienGPIO power-on reset sequence inspihid_apple_of_power_on()is what makes the manual unbind/bind work, but it appears increasingly ineffective within a single boot, suggesting residual controller/firmware state that survives the 60 ms reset but not a cold power cycle).