Native engine port: SMP app-launch, networking, recovery + x86 interact-freeze root-cause & fixes#16
Open
squirelboy360 wants to merge 191 commits into
Open
Native engine port: SMP app-launch, networking, recovery + x86 interact-freeze root-cause & fixes#16squirelboy360 wants to merge 191 commits into
squirelboy360 wants to merge 191 commits into
Conversation
- PS/2 IntelliMouse 4-byte scroll wiring (kernel) -> EV_SCROLL -> embedder FlutterPointerEvent (signal_kind=kScroll); direction+speed configurable. - Settings screen in the shell: natural-scroll toggle + speed slider, live via config:* messages over the oscortex/shell channel. - Kernel boot spinner drawn on the framebuffer during JIT warm-up (drivers/fb.rs::draw_boot_splash, driven by compositor::tick); clears when Flutter presents its first frame. Plus a 'Launching <app>' overlay in the shell. - Adaptive embedder frame pump: gentle during warm-up, 60fps after input, ~8fps idle (was a 1ms/1000-per-sec flood) -> req:frame ratio 115:1 -> ~7:1. - Revert MAX_FRAMES to 1<<20 (RAM above 4GiB is not yet safely mappable; the cap is load-bearing) with an explanatory comment. - docs/arch.txt: document the real JIT execution model + runtime status.
Build the Flutter engine from source as a first-class OSCortex AOT target instead of shimming a Linux engine. Phases, port surface, build infra, and the hack-removal checklist in docs/native-engine-port.md; direction recorded in docs/arch.txt.
Draw the destination ahead of the work: three clean layers (kernel native ABI / shared AOT engine / self-contained AOT app bundles with own PIDs), the shared- framework property, and the hack-removal it enables. No Linux costume.
…ace map Baseline libflutter_engine.so (377MB, x64, embedder API) builds from source in the container — toolchain proven. Mapped the exact port surface: ~1,200 lines (Dart VM os_oscortex/os_thread_oscortex ~1000, fml message_loop_oscortex+paths, reuse posix) + GN glue. The fml message loop is the critical file — its emulated equivalent is what livelocks rendering today, so the native port also fixes sync.
Capture the validated Phase 0 setup as one-command scripts so the repetitive, heavy engine-build infra is reproducible, not tribal knowledge: - setup-engine-build.sh: idempotent container + depot_tools + pinned gclient sync (encodes the name='.' fix and the flags that work). - build-engine.sh: gn configure + ninja for baseline | oscortex targets. - README.md: the contributor flow, the edit->build incremental loop, and the pitfalls already solved (gclient layout, emulation, disk, prebuilt-dart). Engine checkout stays OUTSIDE the repo (22GB, never committed).
Add 'oscortex' as a --target-os. Since OSCortex has no sysroot/libc yet (runs linux-ABI via emulation), it links against linux but sets a new is_oscortex GN flag to select the OSCortex backend sources later; a true OSCortex toolchain is a later sub-phase. gn configures cleanly (1714 targets; args.gn = target_os linux + is_oscortex true). - engine-port/patches/: the two tracked diffs (gn tool, BUILDCONFIG). - engine-port/apply-port.sh: applies patches + (Phase 2) backend sources into a fresh checkout, idempotent. Temp patch files cleaned from the workspace (no remnants).
This is a public-facing repo; debugging slop and dead-path artifacts don't belong. Removed (all dead, none load-bearing for the working JIT build): - scratch/ (192 files): the entire AOT-deser debugging tree — analyze_*/disasm_*/ assemble_*/check_* scripts, .bin/.log artifacts, and vendored gen_snapshot/ shader binaries. Referenced only by build-iso.sh's dead shell-AOT step. - tools/flutter-engine/libflutter_engine.so.bak: 91MB orphaned 'pristine engine' backup, zero references. - gen_help.txt, test_size, test_size.c, .DS_Store: stray one-offs. - build-iso.sh: excised the dead [0.3/5] shell-AOT block (gen_snapshot -> libapp.so -> patch_libapp) — the shell runs JIT off kernel_blob.bin, not an AOT libapp.so; also dropped libapp.so from REQUIRED_FILES. Native AOT is done properly via the engine port (docs/native-engine-port.md). - .gitignore: ignore /scratch/ and /qemu-pipe.* so they can't return. Deliberately KEPT (still load-bearing for the current JIT build; removed in the port's Phase 4 once the native engine works): engine_patch.py, patch_libapp.py (still used by tools/build-flutter-osx.sh for the apps), the Linux-emulation shim. Verified: build-iso.sh and build-kernel-iso-fast.sh both pass bash -n; the working fast build references neither scratch/ nor libapp.so.
Add the OSCortex fml platform backend and wire is_oscortex selection. Verified:
fml.message_loop_oscortex.o + fml.paths_oscortex.o compile into libfml, the linux
message loop is excluded, and message_loop_impl.cc selects MessageLoopOscortex.
- engine-port/src/flutter/fml/platform/oscortex/: message_loop_oscortex.{cc,h}
(epoll+timerfd loop — the sync-bug-fixer, starts as a clean clone to diverge to
native primitives) + paths_oscortex.cc.
- patches/0003: build_config.h defines FML_OS_OSCORTEX (additive, gated on the
FLUTTER_OSCORTEX define), message_loop_impl.cc selects MessageLoopOscortex first,
fml/BUILD.gn swaps in the oscortex sources (excludes linux) + sets the define.
- apply-port.sh applies 0003 + copies src/.
Note: the backend uses the Linux-ABI calls (epoll/timerfd) that OSCortex natively
implements — OSCortex is a native kernel + own libc that is Linux-ABI-compatible
(like Fuchsia/Starnix), not Linux. These calls are the divergence seam if we ever
move to a fully custom ABI.
…t VM clone - Verified a release/AOT oscortex build configures and gen_snapshot is buildable from our tree -> engine+snapshot version-matched, dissolving the old AOT dead-end (the '1247 base objects' mismatch). AOT is now a build step, not a research problem. - Deferred the Dart VM os_oscortex clone: on path A os_linux.cc already works; cloning it byte-identically is busywork. Add a Dart VM backend only when a primitive must actually diverge.
libflutter_engine.so (377MB) links for the oscortex target and bakes in OUR backend: 17 MessageLoopOscortex refs, 0 MessageLoopLinux. First Flutter engine built from source FOR OSCortex. Refined the gn target (patch 0001): oscortex now configures as a HOST linux-x64 build + is_oscortex flag (identical to the proven baseline, our platform backend selected), instead of an explicit target_os that tripped ANGLE/wayland scope and embedder-constructor mismatches. Runtime still renders software via kSoftware.
…ld) flow Make the from-source build a one-time maintainer task, not something every dev repeats — exactly how Flutter distributes its own engine. - artifact.config: pins ARTIFACT_VERSION + Flutter rev + the R2 base URL. - fetch-engine.sh: CONSUMER flow — downloads the pinned prebuilt engine (libflutter_engine.so + gen_snapshot + icudtl.dat) from Cloudflare R2, verifies sha256, stages it where the OS build expects. Seconds, no checkout. - publish-engine.sh: MAINTAINER flow — packages the built artifacts and uploads to R2 (rclone or aws/S3 to the R2 endpoint). Run after build-engine.sh when the port/version changes; bump ARTIFACT_VERSION. - README: two-tier model (consumers fetch, maintainers build+publish), R2 layout, multi-arch = one tarball per ISA. Scaffolding is ready now; first publish happens once the release/AOT engine lands (Phase 3). Set ARTIFACT_BASE_URL to your R2 URL to activate fetch.
- setup-r2.sh: create the R2 bucket + enable public r2.dev URL + auto-write ARTIFACT_BASE_URL into artifact.config. Idempotent. Needs Cloudflare auth first (npx wrangler login, or CLOUDFLARE_API_TOKEN). - publish-engine.sh: upload via wrangler (npx, no install) as the primary R2 backend, rclone/aws as fallbacks. - README: one-time R2 host setup steps. Bucket creation requires the account owner's Cloudflare auth, so this is the single manual step; everything else is automated.
R2 needed dashboard activation; switched the artifact host to Google Cloud Storage (gs://dotcorr-oscortex-engine, public read, billing active). Verified end-to-end: gcloud upload -> anonymous https://storage.googleapis.com/... fetch. - artifact.config: ARTIFACT_BASE_URL -> GCS public URL + GCS_BUCKET. - publish-engine.sh: gcloud storage cp as the primary backend (wrangler/rclone/aws remain as fallbacks). fetch-engine.sh is unchanged (curl, host-agnostic).
The repo is public, so GitHub Release assets download anonymously — free, no egress fees (vs GCS ~$0.12/GB), no separate cloud account, and gh is already authed. Verified end-to-end: gh release upload -> anonymous https://github.com/DotCorr/oscortex/releases/download/... fetch. - artifact.config: ARTIFACT_BASE_URL -> GitHub releases download URL; GITHUB_REPO. - publish-engine.sh: gh release create/upload as the PRIMARY backend (GCS/R2/S3 remain as documented fallbacks). fetch-engine.sh unchanged (host-agnostic curl). - README: GitHub Releases is the host; no bucket setup needed. - Removed the unused GCS bucket (no remnants).
…erified The release oscortex engine (33MB, vs 377MB debug — no JIT) and a version-matched gen_snapshot (6.5MB) built from one tree. Published the first artifact to GitHub Releases (oscortex-engine-1) and verified the full round trip: publish-engine.sh -> gh release -> fetch-engine.sh downloads+checksums+stages. Engine + gen_snapshot from the same tree are version-matched, which dissolves the old AOT dead-end (the '1247 base objects' mismatch). Fix: publish-engine.sh derives the workspace from the container's /work mount (robust to the workspace dir name) instead of assuming a fixed name.
shell app -> frontend_server -> AOT dill (24MB) -> our version-matched
gen_snapshot -> libapp.so (4.4MB native ELF). Verified a real AOT snapshot: all 4
_kDart*Snapshot{Data,Instructions} present with REAL native instructions (T/text,
not zeroed stubs), ELF64 x86-64. The multi-session 'Snapshot expects N base
objects, provided 0' blocker is dissolved because engine + gen_snapshot are built
from one tree (version-matched). compile-app-aot.sh makes it reproducible.
The native, from-source, AOT-compiled Flutter engine RENDERS the shell on OSCortex: FlutterEngineRunsAOTCompiledDartCode -> libapp.so AOT path -> present_callback 393 frames, ZERO JIT warmup (no kernel_blob, no codegen). The UI comes up immediately, no 60-90s compile. This is the whole point of the port. Fix to get here: the release/AOT engine rejects the JIT-era GC/heap dart-flags (switches.cc:478 disallowed) — pass only the 5 AOT-safe engine args (argc=5) when is_aot, dropping the old --old_gen_heap_size etc. Known follow-up (separate, a regression from this session's IntelliMouse 4-byte scroll wiring): mouse clicks/Y are mangled + pointer events not reaching the engine; to fix next.
The 4-byte scroll-packet mode added this session corrupted pointer data: dy mis-parsed (cursor pinned to the top, y=0) and the flags byte mis-parsed (buttons stuck at 0), so clicks never registered and pointer events stopped reaching the engine. Revert to the proven 3-byte packet mode (working clicks > broken wheel). Verified: cursor Y tracks normally again (y=32 start, not pinned at 0). Scroll to be re-added later with correct 4-byte parsing + resync that doesn't regress clicks.
Input is confirmed working under AOT (122 pointer events reached Flutter, hover + clicks register, button presses detected, dropped_total=0 — the earlier '0' was the broken IntelliMouse 4-byte mode + minimal interaction, fixed by the 3-byte revert). Remove the temporary DIAG logging from the embedder + ipc_display. engine-port/compile-app-aot.sh: per-app AOT compile (frontend_server -> dill -> version-matched gen_snapshot -> libapp.so), no patch — used to give each app its own AOT bundle so the AOT engine can run them.
Previously every launched app reused the shell's /system/flutter/libapp.so (aot_va=0 JIT-era skip), so tapping a tile ran the shell snapshot in the app host and stalled. Resolve the per-app snapshot by registry lookup: build_app_libapp_path(app_id) -> /Applications/<name>.app/libapp.so, dlopen it, and point the AOT snapshot loader at that path. Shell keeps its own path. Each app is AOT-compiled to its own libapp.so and staged under its .app bundle, so a launched host loads its own native snapshot, not the shell's.
pthread_cond_signal/broadcast with no waiter parked is a no-op by contract: the predicate is mutex-protected, so a thread that hasn't yet entered cond_wait observes it under the lock and never waits. The kernel was instead recording such signals in COND_PENDING_SIGNALS and letting the next cond_wait consume one as an immediate (spurious) return. A hot mutator (pid 2) that signals its own monitor with woke=0 then re-waits would consume its own fake pending, return 0, find its predicate still false, and re-wait forever — the cond-pending-consume livelock that froze render at a fixed frame count. Remove the mechanism end to end (consume in cond_wait, posts in cond_signal/cond_broadcast, the state map, the import). cond_wait now relies solely on the seq protocol, which delivers every real signal race-free under cooperative single-core scheduling (the value-check and park cannot interleave with a signaler). No remnants.
sys::dlopen forwards path.len() to the kernel as the path length, so a NUL-terminated byte string makes the kernel read the trailing \0 as part of the filename and the open fails. The per-app and shell AOT paths were passing "...libapp.so\0"; strip the NUL at the dlopen call. The AOT snapshot itself loads via aot_snapshot_load (which maps it executable), so this only silenced a spurious failure path, but it is a real bug and removes the warning.
The Debug-level hot per-syscall traces (epoll_ctl, mprotect, cond-signal, every keypress) each block on a synchronous COM1 UART write AND re-render the framebuffer text console, with interrupts disabled. Measured: ~14000 log lines during a single boot+app-launch. Dropping to Error cut serial volume ~12x (14000 -> 1185 lines), removing tens of seconds of emulated-bare-metal warmup. Raise the one line in logger::init back to Debug for deep tracing.
Root cause of the sporadic render crashes. The user GPR snapshot lives in a PER-CPU scratch (gs:[..]) written by the syscall entry stub and shared by every thread on the core. save_full_user_gprs read it LAZILY at yield time — but the wait loops do sti;hlt and the timer ISR can switch threads mid-handler, so another thread's syscall entry overwrites the per-CPU snapshot before our save runs. We then stored the OTHER thread's callee-saved regs (rbx/rbp/r12-r15) into our context; on resume rbx was garbage. Pinpointed via addr2line/objdump: fml::MessageLoopOscortex::Run() resuming from epoll_wait wrote running_ through this=0x1400 (=1280*4, a Skia row stride leaked from another thread) -> SIGSEGV pid=3. This is the whole 0/16/140/489-frames- across-identical-boots sporadicity. Fix: capture_user_gprs_at_entry() snapshots the GPRs into the thread's own PTABLE slot at dispatch_fast entry, while the per-CPU snapshot is still fresh for this thread (before any handler/yield/interrupt window). A per-CPU 'captured' flag makes later yield-time save_full_user_gprs calls no-ops, so a clobbered shared snapshot can't leak in. Validation: 4/4 headless boots render cleanly (present 98-105) with ZERO engine SIGSEGV, vs the prior sporadic crashes. Render is now reliable.
The frame pump only called FlutterEngineScheduleFrame in the no-event branch, so a hover/click/scroll waited for the event queue to drain plus a wm_event_wait timeout (up to ~16ms) before the repaint was even requested. Request the frame in the same iteration input is received; the engine coalesces redundant requests and Flutter flushes the batched pointer packet at BeginFrame, so the scheduled frame reflects the event. Removes the scheduling-side input latency (most visible on real hardware; under cross-arch QEMU TCG the emulation dominates).
Each submitted frame did up to 5 full 1M-pixel passes: a strided re-pack into a freshly allocated 4MB Vec, a byte-indexed B<->R swap, a full-screen fill_rect clear, blit_rgba32, and swap_buffers. Measured ~65ms median (wildly variable 10-92ms) -> a ~15fps ceiling from the blit alone, which read as a low refresh rate / laggy feedback. - Fuse the strided re-pack into the swap: submit_bgra_impl reads the source stride directly and packs in ONE u32-wise pass (read BGRA as u32, swap bytes 0<->2), eliminating the intermediate buffer + the 4MB/frame allocation. - Skip the full-screen fill_rect clear when a presented surface already covers the screen (the common case: one full-screen Flutter surface). Measured after: blit median 5.4ms, steady 5.0-7.6ms (~12x faster, variance gone). Render verified correct + crash-free; colors unchanged.
next_runnable_pid_locked computed the foreground-exclusive group AFTER the input-target and embedder-baton shortcuts, so a due shell (pid 1) baton could schedule the shell engine even while an app is foreground — two heavy Flutter VMs on one cooperative core, the documented cause of the launched app's crash. Compute fg/exclusive FIRST and gate both shortcuts: suppress the pid-1 baton when an app is exclusive, and only honour the input shortcut for a target in the foreground group. No behavioural change when the shell is foreground (the common case): exclusive=false short-circuits both gates. Closes the baton concurrency hole; full app-launch validation still pending an interactive tile-tap (HMP mouse_button injection does not produce reliable clicks headless).
Replace the generic reply-null catch-all in platform_message_callback with a real channel dispatcher (match on channel name). Every channel the framework reaches for is now routed to a concrete handler that does the platform work or returns a codec-correct typed ack; the final catch-all logs the channel name ([embedder/chan] unbound: <name>) so nothing stays an invisible stub. Bound channels: - flutter/textinput (JSONMethodCodec): setClient/setEditingState/show/hide/ clearClient. Editing state (text + selection) is maintained in the embedder. PS/2 set-1 scancodes are now mapped to characters (shift/caps, backspace, enter, tab, space, arrows, home/end, delete); on a key press with an active text client the stored editing state is mutated and TextInputClient. updateEditingState is pushed back over flutter/textinput. Adds a small inline JSON reader/writer (no_std, no crates) for the editing-state maps. - flutter/mousecursor: parse activateSystemCursor kind and ack (no kernel set-cursor-shape syscall exists yet; logged + acked). - flutter/platform: Clipboard.setData/getData/hasStrings backed by an in-embedder buffer; SystemNavigator.pop; SystemSound/HapticFeedback/ SystemChrome acked. - flutter/navigation, system, accessibility, spellcheck, processtext, menu, contextmenu, scribe, restoration, keyevent, platform_views, isolate, lifecycle: explicit JSON typed-null ack.
Replace the ack-only stubs from the platform-channel contract with actual OS-provided capabilities. OSCortex is the platform under the stock engine, so where Flutter needs a platform service the kernel now implements and binds it. Mouse cursor shape (flutter/mousecursor.activateSystemCursor): - compositor: ACTIVE_CURSOR_SHAPE atomic + vector cursor sprites (arrow, I-beam, hand/link, forbidden, grab, horizontal/vertical resize, hidden). draw_software_cursor dispatches on the active shape; set_cursor_shape() repaints immediately. - new syscall SYS_CURSOR_SHAPE_SET (0x4B2). Embedder maps the Flutter cursor kind string to a CURSOR_SHAPE_* and calls it, so hovering a link shows a hand, a text field shows an I-beam, etc. Semantics / accessibility: - embedder wires update_semantics_callback2 (FlutterProjectArgs off 280) and calls FlutterEngineUpdateSemanticsEnabled(engine, true) after run. The callback receives the FlutterSemanticsUpdate2 tree and stores each node (id, label, rect, flags, actions) in a live embedder structure for a11y / automation consumers. flutter/ accessibility now replies with the correct StandardMessageCodec null (0x00), not JSON. System clipboard (flutter/platform Clipboard.*): - kernel-global clipboard buffer (embedder::clipboard) shared across every app/host, with SYS_CLIPBOARD_SET (0x4B3) / SYS_CLIPBOARD_GET (0x4B4). The embedder routes setData/getData/hasStrings to the kernel, so clipboard survives across apps. SystemNavigator.pop (flutter/platform): - SYS_APP_CLOSE_FOREGROUND (0x4B5): refocuses the shell (pid 1) and wakes it. The app embedder, when it is a launched host, flushes its reply, calls the syscall and exits so focus returns to the shell. SystemSound.play (flutter/platform): - PC-speaker beep driver (drivers::beep) via PIT channel 2 + port 0x61, exposed as SYS_BEEP (0x4B6). click vs alert play distinct short tones. Deliberate no-ops (no such hardware), acked with the correct codec: - HapticFeedback.* (no vibration motor), SystemChrome.* (single full-screen compositor surface, no system UI overlays / orientation).
Brings in the embedder channel dispatcher, text input (flutter/textinput), and real OS-backed capabilities: cursor-shape sprites + SYS_CURSOR_SHAPE_SET, live semantics, kernel-global clipboard, SystemNavigator.pop return-to-shell, and a PC-speaker beep driver. Builds green (embedder + kernel).
…e root The SMP crash-recovery teardown can free a dead app's pml4 while another core still holds a stale reference to it (a futex physaddr lookup) — the walk then dereferences a freed/reused table and hits phys_to_virt's "corrupt PTE" panic (an unrecoverable kernel page fault, deref of phys 0 / out-of-range). The OS should survive an app's crash, not die with it. Read-walks (translate_user_page / _flags) and the teardown free walk now route every table-pointer dereference through walk_table(), which returns None for a null or out-of-addressable-RAM frame instead of dereferencing it. A garbage or freed table is treated as "not mapped" (the lookup fails, the caller copes) and the kernel stays alive. The bare phys_to_virt — which still panics on garbage, a useful invariant — is unchanged on the WRITE paths. Verified: 0 panics / 0 corrupt-PTE across the SMP app-launch batch (was ~1/10 before).
packages/oscortex_ui/.dart_tool/ was tracked despite matching .gitignore (packages/*/.dart_tool/) — it predated the ignore rule. It's a regenerated build/tooling cache that should never be in version control. Removed from tracking; the existing .gitignore rule keeps it out.
… Link status Release docs for the x86 SMP app-launch milestone. Records the home-core fix, the multi-core requirement, the honest stability picture (OS crash-proof; intermittent non-fatal first-frame stall; not yet bare-metal-confirmed), and — per request — the Web Link/browser status: app shell + webview pipeline scaffold with a stub engine are done and demonstrable; the Servo web engine is NOT integrated yet.
…edger rules.md: binding rules for every agent — never present a stub as done; "done" means verified end-to-end on the real artifact; report status as exactly one of DONE/UNVERIFIED/STUB/NOT-STARTED; finish the task or name the gap precisely. CLAUDE.md: auto-loaded by agents; makes rules.md mandatory reading before starting and before reporting done. docs/FEATURE_STATUS.md: honest per-feature ledger (start of the stub audit) — flags what has NOT been re-verified rather than rubber-stamping it, and is explicit that the audit is incomplete.
…not "solved" Live HVF test 2026-06-17: tapping an app launches the 2nd engine, which stalls at FlutterEngineRunInitialized (no crash) and freezes the UI. Same single-core cooperative-scheduler root cause as x86. The prior "solved via serial-GC" note was wrong. Fix = wire the home-core SMP path for aarch64 (no aarch64 mirror of the x86 timer-ISR home-core wake-assist exists yet).
…ootstrap WIP
Progress on aarch64 home-core SMP (the ARM app-launch freeze). The prior state was
"AP comes online but stays idle (never schedules)". Now, under -smp 2 on HVF (real
parallel cores):
- The AP idle-parks (wfi) until an app is pinned to it (CPU_HAS_HOME_WORK, set in
spawn_with_bootstrap + woken by the existing reschedule IPI), so it does NOT
contend with the BSP's shell first-frame bring-up (engaging at flutter_init_ready
stalled the shell at present=1).
- On wake it engages + runs the arch-neutral home-pinned scheduler. The launched
app's host (pid 8) DOES run on the AP — main_embedder, FlutterEngineInitialize,
its worker pool all execute there (aarch64 AP EL0 entry works).
- The timer-ISR wake-assist resume is home-gated + re-enabled (multi-core only;
single-core keeps the proven set_state_try-only path, byte-unchanged).
NOT DONE (honest, per rules.md): the app's engine bootstrap DETERMINISTICALLY stalls
at FlutterEngineRunInitialized (4/4 runs) — the cooperative thread-pool handshake
does not complete on the AP, so no frame is presented. This is the deep remaining
aarch64 SMP work. Single-core ARM (the shipped run-arm.sh path, no `smp` feature) is
unaffected — all new behaviour is behind the AP/CPU_COUNT>1 gates. x86 unaffected.
…ounter access) The aarch64 AP never called enable_fpu_simd() (the x86 AP does, in ap_init). That function sets CPACR_EL1 (FP/SIMD) AND CNTKCTL_EL1.EL0VCTEN/EL0PCTEN — per-CPU registers at reset on a secondary core. Without CNTKCTL, the app's inline CNTVCT_EL0 read (liboscortex_libc monotonic clock, hit constantly by the Dart VM) trapped to EL1 with EC=0x18 → report_unhandled → the engine bootstrap deterministically wedged on the AP (4/4 runs, confirmed by an HVF register dump: AP stuck in report_unhandled, ESR EC=0x18). Adding the call clears EC=0x18. The freeze now advances to a later fault (EC=0x24 EL0 data abort) — distinct, deeper issue, still WIP; aarch64 SMP app-launch is NOT yet working. AP-only change; single-core + x86 unaffected.
…s EL0 state Re-enabling enter_user_by_pid_noreturn_try from the AP timer ISR reproduced the prior session's "pid=2 Dart corruption" exactly: a real EL0 data abort (EC=0x24) in engine code. Reverted to wake-only (set_state_try), which clears the corruption (unhandled_exc=0, 3/3). Net aarch64 SMP state with the CNTKCTL fix: the AP runs the app past the CNTVCT trap, but the engine bootstrap stalls at FlutterEngineRunInit — the cooperative hand-off alone doesn't complete it and the only pump (ISR resume) corrupts thread state. The real fix is a corruption-free per-CPU context switch (Redox-style switch_to), a multi-session rewrite. aarch64 SMP app-launch NOT working yet; single-core + x86 unaffected (all behind AP/CPU_COUNT>1 gates).
…mp vs corruption)
… bootstrap needs switch_to rework The home-gated timer-ISR wake-assist resume now fires only when the target has a valid saved FP (aarch64_fp_valid), so it can never re-enter a thread with no FP image and zero its AAPCS64 callee-saved v8–v15 (the EC=0x24 SMP corruption). Verified: 4/4 runs, 0 unhandled exceptions (corruption gone). But the app bootstrap STILL stalls (4/4, RunInitialized never completes): the threads that need re-entry during bootstrap are freshly created and have no FP yet, so the FP-gated resume skips them, and the cooperative hand-off doesn't re-enter them either. This is the catch-22 proven across 5 distinct attempts (CNTKCTL fix, resume on/off, FP-gated): a correct resume must restore ANY thread's full state without corruption — i.e. a real per-CPU switch_to context save/restore, not the build_image rebuild. That is focused multi-session architectural work (precisely scoped in [[smp-bringup]] M5), NOT a blind loop hack. aarch64 SMP app-launch remains BROKEN (honest). Single-core + x86 unaffected (multi-core/AP-gated).
…77% headless HVF)
The launched app now paints its full UI on the app core (core 1) while the
shell runs on core 0, screenshot-confirmed under `-smp 2` HVF (10/13 runs
clean-render, on par with x86 ~70-85%). aarch64 went from always-freeze/crash
to mostly-working. Single-core (shipped run-arm.sh) byte-behaviour unchanged,
verified renders; x86 compiles (the lock change is arch-neutral + more correct).
Root causes fixed (all latent SMP bugs the cooperative single-core path hid):
1. PTABLE_LOCK desync under true parallel cores. It wrapped a spin::Mutex,
recorded the owner in a SEPARATE `holder` atomic, and stashed the guard in a
single `static mut PTABLE_LOCK_GUARD`. Under HVF those three desynchronized:
windows where the Mutex was locked but `holder` read FREE, plus a raced
single guard slot — leaving inner permanently locked with holder=FREE, so
every core spun forever on acquire. Replaced with one CAS'd owner word
(acquire + owner-record are one atomic step). This was THE wedge.
2. ACTIVE_EL1_SP was a single global ("smp=1 bring-up: one slot"). Cores
clobbered each other's EL1 syscall-stack pointer; a cooperative resume on one
core reset SP_EL1 to the other core's stack → two cores on one EL1 stack →
EC=0x24 data abort, FAR=-1. Now per-CPU [MAX_CPUS].
3. The AP never armed its generic timer. CNTV_CTL/TVAL and the timer PPI's GIC
enable are per-CPU; the BSP's start_scheduler_tick only armed core 0, so the
AP took ZERO timer interrupts → no preemption, no ISR wake-assist → the app's
engine bootstrap deadlocked (worker threads need an async wake the
cooperative-only path can't deliver). Armed in ap_main once an app is pinned.
4. The wake-assist FP gate (aarch64_fp_valid) skipped FRESH workers — a freshly
spawned thread has no captured FP yet, but zeroing its FP on first entry is
CORRECT. FlutterEngineRunInitialized's isolate-launch worker is fresh, so it
was never re-entered → bootstrap stall. Replaced with aarch64_resume_fp_safe_try
(entered_once-aware): fresh ⇒ safe; already-run ⇒ needs valid FP (else logged).
5. get_group_leader walked the parent chain in an unbounded loop; a cyclic chain
hung a core inside it while holding PTABLE_LOCK, wedging the scheduler. Bounded
to MAX_PROCS with a one-shot cycle log.
Remaining: ~23% intermittent EC=0x24 shell-corruption during concurrent
shell+app execution (same class as x86's intermittent crash) — the deeper
concurrent-engine/context-switch hardening, tracked in FEATURE_STATUS.md.
…cher crash dump - dev-tools/test/arm-smp-applaunch.py: the headless HVF -smp 2 app-launch test that proved the fix, now committed (was a volatile /tmp script that got cleaned). On a clean host: 15/15 clean render, 0 crash, 0 freeze. The earlier 10/13 was an overloaded test host (30+ concurrent QEMU + builds), not an OS bug. - FEATURE_STATUS.md: x86 app-launch was dishonestly "DONE" while identical in maturity to aarch64 — both are now "WORKS ~X% headless, UNVERIFIED on bare metal, NOT DONE" (rules.md Rule 3: DONE = real-artifact-verified). aarch64 row updated to the accurate 15/15 clean-host figure + the load-induced EC=0x24 residual. - vectors.rs report_unhandled: also print focus_pid / cpu / home_cpu, so an EL0 fault dump shows whether the faulting thread was the foreground app or a backgrounded shell thread (concurrent-execution context). Runs only on a crash.
…ix + syscall correctness hardening The x86 in-app interact #GP (ip=0x140d84cdb, dart:io EventHandler::Poll) is the engine reading the epoll_wait syscall number (0x47B=1147) as an event count, overrunning its 16-slot events[] array, and dereferencing a garbage DescriptorInfo*. Kernel root: epoll_wait yields are re-execution- saved; a yield saved in RETURN mode (rip left PAST the syscall) with rax=the nr returns the nr instead of re-executing the syscall. Fixes / hardening: - perform_cooperative_yield: save_return_context -> save_return_context_reexec. It was the only re-exec helper using RETURN mode (the de-starve/block helpers already use _reexec) -- a real bug. Fixes the cooperative-resume slice of the leak. - x86 entry_user_rip arch-parity: capture the user rip at the IRQ-masked syscall-entry point and prefer it in save_return_context_inner (mirrors aarch64, which already does this); also corrects cross-thread wake rip. - syscall correctness: epoll/timerfd/eventfd close-cleanup (poll::on_fd_closed from sys_close), pipe write atomicity (block, don't tear, for <=PIPE_BUF), SYSCALL-entry FMASK DF clear, MAP_FIXED honors hint only when fixed, translate_user_page non-canonical-VA guard (recoverable, not a panic). - app crash-recovery unwedge (defocus a dead foreground group so the shell can drain relaunches). - flag-gated Phase-3 per-CPU scheduler groundwork (percpu-sched feature, default OFF -> shipping ISO byte-identical). - headless multi-app harnesses (dev-tools/test) + foundation specs (docs). HONEST STATUS -- NOT a complete freeze fix on its own: The bulk of the leak is a rarer race (~0.004% of epoll_waits, but ~50/run at the engine's ~1.2M waits/run) that is NOT yet root-caused kernel-side; the unmodified (fetched) engine still #GPs in ~2/3 interact runs with just these kernel changes. The reliable crash-free fix is an engine-side size-validation -- reject epoll_wait counts > kMaxEvents (16) in HandleEvents -- verified GPF=0 across runs. That lives in the engine artifact (fetched via fetch-engine.sh) and must be re-published to take effect. Diagnostic localizer logs remain in poll.rs (gated / ERROR-level, inert in the shipping config) -- to be pruned in a follow-up.
…the interact-freeze crash-free defense The reliable crash-free fix for the x86 interact #GP (the engine reads the epoll_wait syscall number 0x47B=1147 as an event count and overruns its 16-slot events[] array) is an engine-side validation in dart:io's HandleEvents: reject bogus epoll_wait counts > kMaxEvents=16, plus skip a non-canonical DescriptorInfo cookie. Verified GPF=0 across runs; the unmodified engine #GPs in ~2/3 interact runs. The engine .so is gitignored/fetched, so this cannot land as a binary in this repo. Added as a reproducible engine patch instead: - engine-port/patches/0005-epoll-size-validation.patch - engine-port/apply-port.sh: wire 0005 in (rooted at the dart dep) - engine-port/REPUBLISH-epoll-fix.md: apply -> build -> bump artifact.config -> publish -> fetch -> verify. The publish step uploads to the GitHub release / R2 and needs maintainer credentials. Companion to the kernel-side partial fix (perform_cooperative_yield -> save_return_context_reexec) in the prior commit. Together they make the build crash-free; the kernel rare-race residual remains a follow-up.
…ser-resume paths (kernel-only crash-free interact) The x86 in-app interact #GP (ip=0x140d84cdb, dart:io EventHandler::HandleEvents → di->Mask() on a non-canonical DescriptorInfo*) is the kernel intermittently delivering rax = the epoll_wait syscall number (0x47B=1147) as epoll_wait's return value. The engine reads 1147 as the event count, overruns its 16-slot events[] array, and #GPs. ~50 leaks/run. The prior cooperative-yield re-exec fix (save_return_context_reexec) closed only the cooperative-resume slice (~28/run). The remaining bulk is delivered via the two timer-ISR direct-iretq preempt resumes (apic_resched_handler, apic_timer_handler) and the wake-assist enter_user_by_pid_noreturn_try path — not the cooperative path. The crash register/syscall trace proved every nr=0x47B epoll_wait is issued from the syscall-stub trampoline at ~0x7fffeefa, NOT the engine .text. Fix: a delivery-point guard (epoll_nr_leak_guard) at ALL FOUR user-resume paths. After the CR3 switch (so the read sees the resuming thread's mapping), if rax == the epoll_wait nr AND the resume rip is inside the syscall-stub trampoline [TRAMPOLINE_PAGES_VA, +0x2000) but is NOT the `syscall` instruction (0F 05) — the thread is RETURNING the nr rather than re-executing — neutralize rax to 0. epoll is level-triggered, so the ready fd re-fires on the next wait. Scoped to the trampoline pages, which are always mapped (the thread was executing there), so the [rip] read can never fault. This is the kernel-only crash-free fix: no engine rebuild/republish needed. The engine size-validation patch (0005) remains as defense-in-depth. Verified on the real ISO with the UNMODIFIED (fetched) engine, SMP=1 interact: - pre-fix (engine-.text range): guard-fired=0, GPF=2 every real run (leak crashes). - this fix: GPF=0 across 6 runs; guard-fired=36-57/run (every leak caught). The engine stays alive (schedule_frame loop, input processed to end-of-run); remaining "FROZEN" is the static-UI present-flatline false-positive. Adds a lock-free guard-fired counter dumped (rate-limited, ERROR level) from the safe epoll_wait syscall path — never the boot-fragile resume path — for regression visibility.
Boots the ISO headless, waits for the shell, launches Files via QMP clicks, and screendumps shell + app + post-interact frames to PNG. Used to confirm the epoll-nr-leak fix (55189d5): shell renders, Files paints its full UI, and in-app clicks navigate (/Applications -> /Applications/Files.app/libapp.so, 30->0 items) with no #GP. Complements x86-multiapp.py (which only samples present_callback).
…ash-free The interact #GP is fixed kernel-only (55189d5); single core now renders Files + navigates on click with GPF=0 (8 runs, unmodified engine). Supersedes the stale 'doesn't render / SMP is the fix' row. Engine republish no longer required.
…IVE + crash-free (3/3 HVF) Both arches now verified crash-free on interact (single core). aarch64 needs no guard — nr in x8, return in x0 are separate registers, so the x86 nr-aliases-return leak can't occur. EC=0x24 is a -smp 2 concurrent issue only, not single core.
… clock-independent SMP wait Two real-hardware failures found by booting on bare metal (HP ProBook i3-470 + Intel MacBook), neither reproducible under QEMU: 1. NO INPUT on the ProBook. Input was xHCI-only (no xHCI on 2010 EHCI/UHCI-era hardware) AND PS/2 was deliberately skipped on bare metal — a "safe mode" guard in platform.rs that was stale (it dated from a since-fixed unbounded i8042 flush that hung; ps2::init is now fully bounded). On a laptop the keyboard IS the i8042, so the box had no input at all. Fix: run the bounded/fail-safe ps2::init on all x86, not only under a detected hypervisor; register the PS/2 native whenever it binds (drop the qemu_like gate). USB-HID still binds via the xHCI probe where present. 2. BOOT HANG at `cortex::smp` on the MacBook. The AP-online wait was bounded only by a wall-clock deadline from rdtsc_ns() (raw TSC / fixed rate). Mac firmware doesn't wire the legacy PIT, so that clock never advances as assumed -> the deadline is never reached -> the BSP spins forever (observed: frozen at "cortex::smp 0.5s"). Fix: add a clock-INDEPENDENT spin cap alongside the deadline so the wait ALWAYS terminates and the BSP continues, advertising only the APs that actually came online (boots single-core-safe). QEMU unaffected: APs come online normally and the cap is never reached (verified -smp 2: 2/2 online, 0 timeouts, GPF=0, no regression). Both verified non-regressing in QEMU; bare-metal validation pending on the user's machines. First instances of the broader hardware-generality framework: fail-safe boot (no wait bounded only by a firmware-variable clock) + probe-don't-assume input (no hypervisor-gated drivers).
…ng (fixes landed, UNVERIFIED on metal) First real bare-metal boots (HP ProBook i3-470 + Intel MacBook) exposed: input non-functional (xHCI-only USB + PS/2 skipped on bare metal) and smp-on hangs the Mac at cortex::smp (clock-dependent AP-wait). Both fixed (9543368); only the user's hardware can confirm. Single-core is the safe bare-metal config.
… fault (bare-metal input) On bare metal (HP ProBook) a working PS/2 mouse streams ~100+ move events/sec. Hovering an interactive card then froze the OS: the prebuilt engine cannot run hover hit-tests that fast on a single core, backs up internally, and faults (#GP at an engine address, then a fault-loop). Reproduced headless: a 40/s move-flood on a card faults after ~600 events; the SAME ~600 events at 10/s sail through -> rate-bound, not a leak. Fix: deliver pointer MOVES at the engine's real throughput, not the raw mouse rate. A move is delivered only once the engine has caught up (presented the frame the last move caused -- hooked at sys_surface_present / sys_gpu_submit), with a hard 10/s floor for moves that cause no repaint (jitter inside an already-hovered card -- the exact trigger). The latest position is held and flushed on the next present. Button transitions (clicks/releases) bypass the throttle and are never dropped/delayed; the software cursor is drawn from CURSOR_X/Y (updated on every event) so the visible pointer stays smooth -- only hover hit-testing is rate-limited. Generic for any high-rate pointer source. Verified on the real ISO (single-core, unmodified engine): the flood that froze at 601 moves now survives 28104 moves (1874/s), present still advancing, zero faults; a normal interact still lands clicks (Files launches + navigates). No regression.
…l stall (uncached fb) On a 2016 Intel MacBook boot reached cortex::spawn_shell and stalled 115s+ (counter climbing) with no shell, but booted fine on the ProBook's small panel. Root cause: the bootloader maps the framebuffer UNCACHED (it is in the PCI MMIO hole, phys 0xfd000000; no PAT/MTRR is set up anywhere). On a 2880x1800 Retina panel (~20 MB) every full-frame paint is tens of MB of uncached MMIO writes, so once spawn_shell starts the engine the present path livelocks. The ProBook's small panel makes the same writes cheap. Fix: remap the framebuffer write-combining. - setup_pat_wc(): reprogram IA32_PAT entry PA1 (PWT=1) from WT to WC. Nothing uses WT; mmio() uses PA3=UC, so this is safe. - map_write_combining(): map ONLY the fb's physical pages at a fresh kernel-half VA with PWT set (-> PA1=WC), in an empty PML4 slot, leaving the bootloader's original uncached mapping (and any neighbouring MMIO in the same huge page) untouched. fb writes go from ~tens-of-MB/s to GB/s, scaling away with panel size. - leaf_entry_of(): read-only cache-attribute probe (diagnostic). x86-only (gated); aarch64 (ramfb) unaffected. Verified on the real ISO: WC engages (FB_ADDR=0xffffc00000000000 -- the WC VA, not the uncached fallback), shell + Files render correctly with no garbling, RESPONSIVE, panic=0, GPF=0. The fb is confirmed uncached (Limine logs phys 0xfd000000). The speed win is only observable on real uncached-MMIO hardware (a large panel) -- QEMU's fb is already fast -- so bare metal confirms the Mac boot; QEMU confirms correctness (engaged + renders). PAT is per-CPU: setup_pat_wc runs on the BSP (single-core shipping config); an SMP build writing the fb from APs would need it in ap_init too.
…el (foundation, behavior-identical) First step of the per-CPU preemptive scheduler rework — the generic fix for the freeze class (app-launch / install-demo / Mac stall all = the cooperative scheduler can't reclaim the core from a busy/stuck/GC-safepointed thread). A0: timer_preempt_switch_try (the single funnel that switches away a running thread, both arches + BSP/AP) now also bails when the interrupted thread is non-preemptable -- preempting + migrating a thread mid-Dart-isolate-critical-section is the EC=0x24/#GP isolate-confinement class. is_preemptable (the dead Redox primitive) is set by the engine via SYS_SET_PREEMPTABLE (A1, not yet wired), so the field stays always-true (init) and this is a BEHAVIOR-IDENTICAL no-op today. Verified: SMP=2 multiapp RESPONSIVE, GPF=0, no regression; the single-core shipping path is unaffected (the bail never fires while true). Also lands the re-verified, architecturally-corrected execution roadmap in docs/spec-phase3-percpu-scheduler.md (EXECUTION LOG): the single bail site (NOT scattered in the ISRs), the SYS_SET_PREEMPTABLE ABI + slice-expiry safety valve (A1 -- a non-preemptable thread can delay a discretionary switch but NEVER freeze a core), the 8 exact run-queue enqueue hooks (B), and the highest-risk two-layer switch_to (step 4). aarch64 EL0 preemption is hard-off -- untouched.
…t freeze bug) + foundation blueprint The 12-agent research synthesis (Redox/Linux/seL4/Zircon/Dart-engine) pinned the freeze class to ONE root cause: the kernel cannot involuntarily reclaim a CPU from a busy thread, because the mandatory time-slice preempt SILENTLY never fires -- account_tick_try did PTABLE_LOCK.try_lock() and returned None on contention, so while threads churn the lock the quantum is never decremented and a CPU-bound thread (install loop, Dart GC) is never forced off -> the whole OS (incl. the kernel-drawn cursor) freezes. Step 1a: a LOCK-FREE per-CPU quantum (PER_CPU_QUANTUM, advanced by the timer ISR with no lock) is now the slice verdict, so a spent quantum ALWAYS forces the switch regardless of lock contention. PCB cpu_ticks/slice_left stays best-effort under the lock and no longer gates preemption. Per-CPU single-writer -> race-free without a lock. Verified on the real ISO: SMP=2 x2 + SMP=1, GPF=0, panic=0, no regression (app-launch + static-UI plateau unchanged). The load-bearing backstop the rest of the rework builds on (the forced-quantum valve + is_preemptable counter come next). Also lands docs/foundation-blueprint.md: the unified diagnosis + the ordered 7-step, arch-neutral, lowest-risk-first plan (forced valve -> engine is_preemptable signal -> atomic futex WAIT -> address-space futex key -> correct cond_var -> two-layer switch_to -> delete the crutches) with exact files, verification gates, risks, and the Redox/Linux pattern per step.
…eup window) Blueprint Step 3. The FUTEX_WAIT value-check and the waiter enqueue were in SEPARATE FUTEX_WAITERS lock acquisitions, so a FUTEX_WAKE landing in the gap found no waiter and was LOST -- the thread then parked forever (a face of the post-app-launch render freeze). Now value-check + enqueue + the Running->Blocked transition all happen under ONE FUTEX_WAITERS critical section: lock once, read_volatile(uaddr) UNDER the lock, compare (EAGAIN if changed), push the waiter, then try_block (Running->Blocked ONLY, returns false if a wake already flipped us -> skip sleeping). The lock is dropped BEFORE any cooperative hand-off, never held across the yield (no cross-core strand). The wake side uses try_unblock (Blocked->Running, idempotent) after dropping FUTEX_WAITERS. Same ordering on FUTEX_WAIT_BITSET (op 9); its bitset/timeout semantics are unchanged (Step 5). New process primitives: try_block (Running->Blocked-only, bool) and try_unblock (Blocked->Running-only, idempotent, resched IPI). Lock order is FUTEX_WAITERS -> PTABLE_LOCK in the WAIT path; audited every FUTEX_WAITERS site -- no reverse nest exists (no ABBA). Verified on the real ISO: SMP=2 (files, files+weblink) + SMP=1, GPF=0, panic=0, deadlock=0, sustained rendering (present_callback -> n=395 in the long run), no hang. Single-core behavior preserved (a wake can't interleave a held lock there -> try_block always succeeds -> identical block-then-halt). Crutches (S6/S9, cond_miss_bridge, 0x47B) untouched -- they retire in Step 7. The futex KEY is still raw VA -- Step 4 fixes the cross-process collision next.
…l the cross-process collision)
Blueprint Step 4. FUTEX_WAITERS was keyed by raw VA -- process-local. The shell (pid1) and a
launched app load the same libc/Flutter .so at IDENTICAL VAs, so their pthread mutex/cond words
collided in one bucket and a wake could pop the WRONG process's waiter -> park forever (the
post-app-launch render freeze). The previous physical-address overlay (futex_phys_of) translated
the VA on EVERY op -> double-faulted (Attempt-1).
Fix: key by FutexKey::Private{root_phys, va}, root_phys = the process's page-table root (pml4_phys)
-- a single field read, NO page-table walk. Same-process threads share pml4_phys
(spawn_thread/clone_thread copy the parent's) -> correctly collide; different processes -> different
roots -> correctly separated. MAP_SHARED is impossible (sys_mmap has no flags param -> all mappings
private), so every futex is private and this key is provably exact; Shared{frame_phys} is reserved.
futex_phys_of + its per-op translate are DELETED (the #DF cause is gone).
Key computed ONCE per op from current_user_root() BEFORE FUTEX_WAITERS (preserves the Step-3
FUTEX_WAITERS->PTABLE_LOCK order). CondWaitState carries a pre-computed cond_key so the timer-ISR
cond-timeout path removes waiters WITHOUT touching page tables under the IRQ mask. caller==0
force-wake drains all VA-matching buckets (bring-up path intact).
Verified on the real ISO: SMP=2 (files, files+weblink) + SMP=1. DF=0 across all runs (the Attempt-1
double-fault is structurally eliminated), GPF=0, panic=0, strong sustained rendering (present_callback
-> n=448). KNOWN: an intermittent SOFT freeze (present flatlines after rendering, no crash) persists
-- the cooperative cond-timing / scheduler gap, addressed by Steps 5-6, not a collision/#DF issue.
…g — kill the "Poll failed" soft freeze The intermittent post-app-launch soft freeze (app renders, then present flatlines, no crash) was the epoll de-starve's 0x47B re-exec leaking: on resume the _reexec rip occasionally advances PAST the syscall, so the epoll_wait nr (0x47B) reaches the engine as the return value. The guard caught it but NEUTRALIZED to 0 -- a spurious epoll_wait=0 the engine's I/O loop logged as "Poll failed" and degraded into the freeze (correlated 1:1: fired=8 <-> frozen, fired=1 <-> responsive). Fix: the guard now RE-EXECUTES instead of faking 0. On a leak it rewinds rip to the `syscall` instruction (rip-2, verified 0F 05) and keeps rax=0x47B, so epoll_wait re-issues cleanly and the thread re-blocks in-kernel -- exactly the de-starve's intent. The de-starve (load-bearing for focus-prioritized render -- removing it stalls the shell at the event loop, confirmed) is untouched; only the leak HANDLING changes. Guard returns (rax, rip); the 4 resume sites (idt.rs direct-iretq x2, enter_user_by_pid_noreturn + _try) apply both. Verified on the real ISO: SMP=1 x3 with guard fired=8 (the exact frozen frequency before) all RESPONSIVE, present_callback -> n=394-401, PollFailed=0; SMP=2 files RESPONSIVE, present n=416. PollFailed=0 across all runs. The underlying frame-rewrite resume race still fires the guard; re-executing makes it harmless (the full structural elimination is the Step-6 two-layer switch).
…UI, cursor moves" degradation After extended heavy use the Flutter UI goes unresponsive while the kernel cursor still moves (ProBook, user-reported). Root cause: process::exit() + the reapers free page tables but touch NONE of the per-pid kernel tables (COND_WAIT_STATE, FUTEX_WAITERS, EPOLL_BLOCKED, malloc bookkeeping). The Dart VM spawns+joins short-lived worker threads continuously, so every dead thread ORPHANS its entries forever. Those tables are scanned O(n) on the timer ISR + every cooperative yield, so they bloat until the scheduler burns its time on phantom wakes to dead/recycled pids and the engine's real frame threads starve -- a one-way ratchet, permanent, with the kernel (cursor/timer) still alive. A short test never repros it; it needs long heavy thread churn. Fix: cleanup_dead_pid(pid, pml4, is_thread) at the single exit chokepoint (covers sys_exit / kill / kill_group / crash recovery), run AFTER PTABLE_LOCK is dropped (the futex/cond locks order before it). Drains the pid from COND_WAIT_STATE + every FUTEX_WAITERS bucket + EPOLL_BLOCKED, and -- standalone process only (threads share the parent's pml4) -- purges its malloc blocks by pml4. Helpers where the tables live: syscall::cleanup_dead_pid orchestrates, poll::cleanup_dead_pid, posix::cleanup_dead_pml4. Verified: compiles + boots + renders on the real ISO (present->474), no regression (single-core). The accumulation is now bounded -- every thread death reclaims its own entries.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What & why
This branch (
feat/native-engine-port, 174 commits ahead ofdevelop) runs a custom prebuilt Flutter engine on the OSCortex microkernel (x86_64 + aarch64) as the shell/app runtime — SMP app-launch, app networking (TCP/DNS), crash auto-recovery, and a signed package pipeline. This session's focus was the x86 in-app interact-freeze.The interact
#GP(ip=0x140d84cdb, dart:ioEventHandler::Poll) is the engine reading the epoll_wait syscall number (0x47B= 1147) as an event count, overrunning its 16-slotevents[]array and dereferencing a garbageDescriptorInfo*. Kernel root cause: epoll_wait yields are re-execution-saved; a yield saved in RETURN mode (rip left past thesyscall) withrax=the nr returns the nr instead of re-executing the syscall.Fixes / hardening this session:
perform_cooperative_yield:save_return_context→save_return_context_reexec— it was the only re-exec helper using RETURN mode (the de-starve/block helpers already use_reexec). A real bug; fixes the cooperative-resume slice of the leak.entry_user_riparch-parity: capture the user rip at the IRQ-masked syscall-entry point and prefer it insave_return_context_inner(mirrors aarch64); also corrects cross-thread wake rip.poll::on_fd_closed), pipe write atomicity, SYSCALL-entryFMASKDF-clear,MAP_FIXEDhonors hint only when fixed,translate_user_pagenon-canonical-VA guard (recoverable, not a panic); app crash-recovery unwedge.percpu-sched, default OFF → shipping ISO byte-identical) + foundation specs + headless harnesses.#GPs in ~2/3 interact runs with just these kernel changes. The reliable crash-free fix is an engine-side size-validation (rejectepoll_waitcounts >kMaxEvents=16 inHandleEvents) — verified GPF=0 across runs. That lives in the engine artifact (fetched viafetch-engine.sh) and must be re-published to take effect. Diagnostic localizer logs remain inpoll.rs(gated / ERROR-level, inert in shipping) — to prune in a follow-up.Type
Architecture compliance
tools/flutter-engine/engine_patch.pyonly — N/A this session: no engine source is committed here (the engine.sois gitignored/fetched); the engine-side size-validation, if adopted, is a separate engine-artifact rebuild.landing/Verification
-smp 2): boots, renders the shell, launches the Files app (rendered=True),panic=0, runs the full 110s interact test. The#GPwas localized to the kernel re-exec path and the cooperative-yield slice fixed. Harness:dev-tools/test/x86-multiapp.py(SMP=1 APPS=files INTERACT=1 DUR=110).bash tests/run_all.sh— not run this sessioncargo fmt --all -- --check— not run this sessionflutter analyze— N/A (no Dart/shell change this session)