Demo stability (lockup + leak) and continuous Pages deployment (#20) - #25
Merged
Conversation
… stuck (#19, #20) A live Dropbox run failed with 'fetch: send: NetworkError' after ~20 s of a single 'configuring storage…' line — a message that names neither the operation nor the host, following a window with no progress at all. Three fixes, all reproduced and verified in the browser: - every provider request names itself in transport errors (PUT host/path: transport failed after 3 attempts: …), and transport failures retry up to 3x. Statuses are never retried — 429/5xx belong to the caller — and every provider call in the engine is idempotent by construction, so the retry is safe. - storage setup announces each of its ~20 sequential steps, so a failure says which one died and the advice is actionable. - a duplicate 'Save & connect' no longer re-runs the whole setup underneath the first one. The guard is claimed synchronously at call time: the background chain serializes work, so a flag checked inside the job always finds the previous run finished. Also: unversioned assets were serving returning visitors a cached bundle against fresh components (it cost an hour of chasing an already-deployed fix). The build now stamps a mutable root and artifacts inherit it — NOTES §Release integrity's bootloader shape in miniature, and what makes a Pages republish take effect. docs/spike-demo is rebuilt on it. Verified after the changes: the tasks-engine MinIO+relay scenario still passes; in-browser, the S3 leg revokes with the cooperative note and bob's pull goes dark (kp missing 404), and the Dropbox leg reports expired tokens legibly (401 expired_access_token) instead of a bare network error.
) The lockup-then-crash was backpressure, not (only) a leak: every periodic driver appended to an unbounded promise chain unconditionally — app poll (400ms x3), reconciliation pulls (2.5s), auto bucket-sync (4s), stats (4s) — while the work behind them routinely outlives the period (storage runs 1-3s/op on consumer APIs). Fixed-rate scheduling plus slower-than-period work diverges: the queue itself becomes the leak, and user input ends up behind hundreds of pending jobs. Periodic work now skips a tick whose predecessor is still running, which is the correct semantics anyway — a reconciliation pull is a refresh, not a transaction. Measured against a 1.5s/op delay proxy in front of MinIO: 180 ticks skipped in 3 minutes (all of which the old code would have queued), background depth bounded at 3-4, and a UI-path add still completing in 3ms while storage churns. __demo.health() exposes queue depth and per-timer skip counts. A residual browser-only leak remains and is now bounded away from our code: 500 driver/tasks calls leak nothing, 10 sync pulls leak nothing, boot without a wire is flat, and host/leak-probe.ts — the same engine composite under Deno with the same live subscriptions, idling 90s — is flat (-1.8MB). Growth needs a live wire in a browser, so it belongs to the deltic browser ports or the embedder's browser glue, not to the engine. Recorded with the isolation steps for a follow-up profile.
…real-browser measurement (#20) Two outcomes from chasing the reported lockup/crash. A real leak, fixed: driver.sync-start inserted a result into the syncs map for every sync and nothing ever removed it — sync-status only read it — so the table grew without bound at the demo's ~48 syncs/minute. Statuses are one-shot by contract, so the entry is now removed as it is read. stats() additionally publishes the guest's table sizes, because a growth bug inside the component is otherwise invisible from outside it. The rest was not a leak, and the previous README claim (blaming the deltic browser ports) was wrong. Bisected: driver/tasks calls flat, app polls flat, reconciliation pulls appeared to grow — then two independent checks cleared the stack. table-probe.ts runs 400 pulls headless: every guest table flat, RSS plateaus at ~300MB. cdp-heap.ts drives the identical page in a real headless Chromium and forces a GC at the end: over 300s the heap sawtooths 8-40MB and returns to 7.5MB, net -3.1MB. So the unbounded growth was the queue divergence fixed in the previous commit (one retained closure per queued job) plus paseo's instrumented webview retaining objects. Memory gets measured in a real browser from now on; the probes are committed so the next person starts from a reproduction.
The site was built on a laptop and committed: every rebuild added an ~11 MB engine composite to git history (four of them already landed), and 'keep Pages updated' was a manual step that could silently drift from the source it claims to demo. .github/workflows/pages.yml builds the site from source on every push and deploys docs/ from main; PRs build it too but do not deploy, so a broken demo fails the PR rather than the site. scripts/setup.sh is the shared entry point — the same script developers run — pinning the sibling deltic ports by commit (they carry embedder conventions that have broken this demo before), the toolchain via rust-toolchain.toml, and wasm-tools/wac/just by version. It delegates to polymorph-iroh's own setup for its vendored deps, verified from a cold clone. docs/spike-demo/ is now generated rather than committed, so the wasm artifacts stop accruing in history. NOTE: the Pages source must be switched from 'branch: main /docs' to 'GitHub Actions' for this to take effect; until then the last committed build keeps serving. Doing that before this lands would take the demo offline, so it is a deliberate post-merge step.
Restores docs/spike-demo/ and drops the .gitignore entry from the previous commit. The workflow still builds the site from source, but deleting the artifacts that currently SERVE the site in the same change that introduces an unproven deploy path would take the demo offline for the length of the gap — and this repo has never run a user-defined workflow, so the path is unproven by definition. The cutover order now lives in the workflow header: green build on main, switch the Pages source to GitHub Actions and confirm the deploy, then delete the committed artifacts.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Rebased onto main after #24 merged — these five commits were pushed to that branch minutes after it landed, so they never made it in. Content is unchanged apart from the rebase.
Demo stability: the reported lockup/crash
Reported as "memory leak; app locks up and crashes after several interactions and a couple of minutes". Two real causes, both fixed:
poll(400 ms x3), reconciliation pulls (2.5 s), auto bucket-sync (4 s), stats (4 s) — while the work behind them routinely outlives the period (storage runs 1-3 s/op on consumer APIs). Fixed-rate scheduling plus slower-than-period work diverges: the queue becomes the leak, and user input lands behind hundreds of pending jobs. Periodic work now skips a tick whose predecessor is still running. Measured against a 1.5 s/op delay proxy: 180 ticks skipped in 3 minutes (all of which the old code would have queued), queue depth bounded at 3-4, and a UI task add still completing in 3 ms while storage churns.__demo.health()exposes depth and per-timer skip counts.sync-startinserted a result intosyncsfor every sync and nothing removed it (sync-statusonly read it) — unbounded at ~48 syncs/minute. Statuses are one-shot by contract, so the entry is removed as it is read;stats()now publishes guest table sizes, because a growth bug inside a component is invisible from outside it.The rest was a measurement artifact, and my earlier claim was wrong. I had blamed the deltic browser ports in the previous PR's README; two independent checks cleared them:
host/table-probe.tsruns 400 pulls headless (every guest table flat, RSS plateaus ~300 MB), andhost/cdp-heap.tsdrives the identical page in real headless Chromium with a forced GC — over 300 s the heap sawtooths 8-40 MB and returns to 7.5 MB, net -3.1 MB. The unbounded growth was the queue divergence plus paseo's instrumented webview retaining objects. All three probes are committed.Also in here: transport errors now name their request (
PUT host/path: transport failed after 3 attempts) and retry transient failures 3x (statuses are never retried); storage setup announces each of its ~20 steps so a failure says which one died; a duplicate "Save & connect" is refused synchronously; and unversioned assets no longer serve returning visitors a cached bundle (build-stamped root, artifacts inherit it).Continuous Pages deployment
.github/workflows/pages.ymlbuilds the site from source on every push and deploysdocs/from main; PRs build but do not deploy, so a broken demo fails the PR rather than the site.scripts/setup.shis the shared entry point — the same script developers run — pinning the sibling deltic ports by commit, the toolchain viarust-toolchain.toml, andwasm-tools/wac/justby version; it delegates to polymorph-iroh's own setup for its vendored deps. Verified from a cold clone: siblings pinned, endpoint component built, engine composite composed against it.Cutover is deliberately staged (order in the workflow header): land the workflow and confirm a green build on main → switch the Pages source from
branch: main /docsto GitHub Actions and confirm the deploy → then deletedocs/spike-demo/from git and gitignore it. I did the deletion first, then reverted it: removing the artifacts that currently serve the site, in the same change that introduces an unproven deploy path, would take the demo offline for the length of the gap.Note: no PR-triggered run appeared for the previous branch, and this repo has never run a user-defined workflow — I could not determine why from the API (org Actions policy is not readable by this account). If this PR shows no
Pages / buildcheck, that is the thing to look at.Gates
spikes/tasks-engine:just checkclean; full MinIO+relay scenarioSPIKE PASSED.spikes/demo:deno checkclean;just site/just pagesbuild; demo beats re-driven in-browser (both providers, revocation on each).scripts/setup.sh: cold-clone run verified.Refs #20, #19.