Explorer downloads ~74 MB cold: DuckDB-WASM 1.24.0's HEAD+Range probe expects 206; our (correct) 200 defeats range requests

@rdhyee

🤖 rbotyee (Claude, operated by @rdhyee) — Raymond's intent: overnight work on making isamples.org more robust at scale. What I did: measured the Explorer across connection speeds and browsers, root-caused a large cold-load cost, and proved the cause with a controlled A/B. Filing because the fix lives in the Cloudflare Worker, not this repo.

Summary

A cold load of the Interactive Explorer transfers ~74 MB before the facet panel appears. Our own docs claim "typically less than 1 MB for initial exploration" (index.qmd) and "only the bytes you need are transferred" (explorer.qmd). Both are wrong by roughly two orders of magnitude.

The whole thing comes down to one status code.

DuckDB-WASM decides whether a server supports partial downloads by sending a HEAD request with a Range header. data.isamples.org answers 200 instead of 206, so DuckDB concludes the server can't do partial reads and downloads every file whole — including samples_map_lite_v3.parquet (62.9 MB) when it actually needs about 1.5 MB of it.

Proposed fix

In the Cloudflare Worker fronting data.isamples.org: when a HEAD carries a Range header and the object supports ranges, respond 206 with Content-Range (no body, per HEAD semantics) rather than 200.

Content-Range is already in Access-Control-Expose-Headers, so nothing else needs to change.

Evidence — controlled A/B, not inference

A transparent reverse proxy forwarded every request to the real host unchanged, altering exactly one thing: HEAD+Range → 206. Same build, same cold cache, one variable.

Unthrottled

	Control (`200`)	Treatment (`206`)
Bytes from data host	74,202,598	3,341,812
`full HTTP read` fallbacks	8	0
`samples_map_lite_v3.parquet`	62,924,115 B (whole file)	1,467,731 B (8 ranged reads)
`sample_facet_masks.parquet`	10,138,648 B (whole file)	767,000 B (48 ranged reads)

3G fast (1.6 Mbps) — the user-visible result

	Control	Treatment
Globe drawn	54.0 s	54.1 s
Facet panel	440.6 s	94.2 s
Bytes	74.2 MB	3.0 MB

4.7× faster to a usable filter panel, 24× less data. Globe and table are unchanged — they read the small H3 summary, not the big files.

Why this went unnoticed

I cleared the server three times before finding it, because I kept testing with GET:

Probe	Result
`GET` + `Range`	206 + `Content-Range` ✅
`HEAD` + `Range`	200, no `Content-Range` ❌ ← what DuckDB actually sends
`OPTIONS` preflight	204, allows `Range` ✅
CORS exposure of `Accept-Ranges`	correct ✅

Everything was right except the one verb that mattered. Please verify any fix with HEAD, not GET.

Worth noting this is not an RFC violation — raw.githubusercontent.com answers 200 to HEAD+Range too. It's about interoperating with DuckDB-WASM's capability probe, which is by far this bucket's most important client.

Also answers #313

The same harness answers @akthom's #313, open since June and never done.

Production, cold cache, desktop:

Connection	Globe	Facet panel
Unthrottled	2.2 s	10.8 s
4G	15.4 s	168.7 s
3G fast	38.7 s	423.1 s
3G slow	156.6 s	never (>600 s)

On a slow connection the facet panel never appears at all.

Cross-browser (good news): works in Chromium, Firefox and WebKit/Safari, desktop and mobile viewports, zero uncaught page errors, comparable timings. The 74 MB is identical in all three, so this is not a browser quirk.

Reproducing

Both instruments are committed in perf/313-bandwidth-findings on the rdhyee fork:

tests/playwright/bandwidth_matrix.py — the measurement harness
tests/playwright/range_fix_proxy.py — the A/B proxy that isolates the cause

Full write-up with method notes and limitations: PERF_BANDWIDTH_FINDINGS_2026-08-06.md on that branch.

After the fix

index.qmd's "less than 1 MB" and explorer.qmd's "only the bytes you need" should be updated to the measured figure (~3.3 MB cold) — still a good story, and an honest one.

Caveat on the numbers: single runs on one machine and uplink, DevTools throttling rather than a real cellular link, cold-cache first visits only. Order-of-magnitude, not benchmarks. The A/B comparison is the reliable part, since both arms ran under identical conditions.

@rdhyee

🤖 rbotyee (Claude, operated by @rdhyee) — Raymond's intent: he asked "is this a slam dunk?", so I stress-tested my own finding with Codex instead of defending it. What I did: three corrections to the issue above, one of them to a claim that was simply wrong, plus the warm-cache measurement that was missing.

Short version: the mechanism holds and the effect is large, but the framing above is wrong in two important ways, and I owe a correction to a colleague's prior work.

Correction 1 — I was wrong that #313 had never been investigated

ISSUE_313_FINDINGS_2026-06-26.md is a real prior investigation. It raised this exact hypothesis (its "H2 — DuckDB-WASM full-HTTP-read fallback") two months ago, and dismissed it with:

"The server does correctly respond 206 for range requests (curl confirmed)"

That is the same GET-versus-HEAD mistake I made, and it sent that investigation off toward Firefox HTTP/2 connection-pooling theories instead. So this finding isn't new ground — it's the resolution of a hypothesis that was correctly identified in June and mis-cleared by a probe that used the wrong verb. That prior work deserves the credit for finding the right suspect.

Correction 2 — the proposed fix is NOT standards-correct, and I should not have called it a fix

RFC 9110 §14.2 is unambiguous: Range is defined only for GET, and a server MUST ignore Range on other methods, including HEAD.

So the Worker's current HEAD+Range → 200 is correct. DuckDB-WASM 1.24.0's capability probe is the nonconforming party. What I proposed is an interoperability workaround, and the issue title/body above frame it as a bug fix. That's wrong and I'd have misled anyone who acted on it.

Consequences for the proposal, all adopted:

Scope the shim as narrowly as possible. Answer 206 only to the exact probe observed (Range: bytes=0-). Every other ranged HEAD stays standards-correct at 200, so the divergence cannot leak to other clients.
Ship it with a removal path, not as a permanent contract.
Say "DuckDB-WASM 1.24.0's probe", not DuckDB-WASM generally.

Correction 3 — the warm-cache trade-off, which the issue omitted entirely

The 74 MB figure is a cold load. Today those files carry Cache-Control: max-age=1yr, immutable, so repeat visitors already pay very little. Measured properly (fresh profile per arm, full browser close + relaunch between visits, passive instrumentation — Playwright's route() disables the HTTP cache and would have destroyed the thing under test):

	Control (200 / full reads)	Treatment (206 / ranged)
Cold bytes	74,202,598	3,341,812
Warm bytes (relaunch)	0	1,467,731
Warm time-to-facets	4.2 s	4.2 s

So there is a real warm-cache cost: ~1.47 MB per repeat visit, against ~71 MB saved on the first. The illustrative byte crossover is around 48 revisits. Treat that as a crossover under the measured visit-2 behaviour, not a user break-even — visits 3+ weren't measured, and the byte counts come from Playwright rather than the proxy's own network accounting.

No warm time regression (4.2 s either way).

A hypothesis we tested and killed

Codex predicted from Chromium source that a repeat HEAD+Range might be served from the sparse cache entry and normalised 206→200 by FixHeadersForHead(), re-triggering the full-read fallback and making the treatment much worse warm. Falsified: on the warm relaunch every HEAD+Range was observed as 206, with zero fallback logs. Worth recording because it was the most plausible way this could have backfired.

What else changed

Reproduced against the real Worker, not just my proxy: wrangler dev --local with a seeded local R2 and a contract test (workers/data-isamples-org/test/range_contract.sh). Unfixed Worker: 17 pass / 3 fail, all three being HEAD+Range. Caching, CORS, ETag stability and ranged GET all pass.
The standards-correct client-side alternative does not work on our pinned version. DuckDBClient.of(sources, config) does pass through to db.open(config), so it was a one-line change. On Quarto's pinned duckdb-wasm 1.24.0: reliableHeadRequests: false had no effect (still 73.6 MB, 8 fallbacks); adding allowFullHTTPReads: false dropped bytes but broke the app — facets never render, because DuckDB can neither range nor full-read.
Workers Caching appears disabled. Production returns no CF-Cache-Status, and the Worker demonstrably still sees Range (it computes content-range itself). If it is ever enabled, Cloudflare strips Range, caches a full 200 and slices 206s itself — which would replace/bypass the Worker's manual range path, and the nonstandard HEAD probe would need re-testing under that configuration.

Where this leaves it

Option	Works?	Standards-correct?
Narrow Worker shim (`Range: bytes=0-` → 206)	✅ proven	❌ knowing divergence
Client `reliableHeadRequests` on 1.24.0	❌ no effect / breaks app	✅
Upgrade duckdb-wasm (Explorer does its own init)	untested	✅ — the removal path

Recommendation: ship the narrow shim as a documented, reversible compatibility measure, with the duckdb-wasm upgrade as the removal path. ~71 MB saved on first visit and a usable filter panel 4.7× sooner on 3G is worth 1.47 MB per revisit and no warm time cost — but it should be labelled honestly as a shim for one client's nonconforming probe, not as fixing a server bug.

Before deploying, Codex recommends a real-Cloudflare-edge canary: everything above was measured on headless Chromium over http://localhost through a Python proxy, and the local wrangler dev Worker test — not the real Worker on the real edge.

Excluded from the headline: one control run reported 143.6 MB cold rather than 74.2 MB, because sample_facets_v4 (69 MB) crossed the measurement boundary asymmetrically between arms. That weakens that single run, not the mechanism — but I'm not using it affirmatively either.

CI: cheap HTTP contract test for the data origin (would have caught i…

isamplesorg#345: canary config + one-command deploy/verify runbook

isamplesorg#345 canary: poll for workers.dev route propagation before…

isamplesorg#345: harden the deploy path (Codex review of the canary b…

isamplesorg#345 deploy path, Codex round 2: fail-closed version parsi…

isamplesorg#345: state the win honestly — ~74 MB → ~3.5 MB is bytes-t…

isamplesorg#345 CI/deploy, Codex round 3: --runxfail so the contract …

isamplesorg#345 is live (data.isamples.org Worker 6f8170a7, 2026-08-2…

@rdhyee

🤖 rbotyee (Claude, operated by @rdhyee) — Raymond's intent: ship this. What happened: Raymond deployed the shim to data.isamples.org on 2026-08-27 at 17:48 ET (./deploy-prod.sh; Worker version 6f8170a7-0bdd-4590-879e-6c1e66660246, previous b61d3bbc-… kept as the rollback target). CC verified afterwards.

Verification

deploy-prod.sh --verify: 14/14 — probe HEAD + Range: bytes=0- → 206 with correct Content-Range and Content-Length; every other ranged HEAD still 200 (the shim has not widened); ranged GET 206; plain GET/HEAD 200; immutable Cache-Control, CORS, /current/ alias intact.
tests/test_data_origin_contract.py against production: 5 passed with --runxfail; the xfail marker is removed in #345: narrow HEAD+Range shim so DuckDB-WASM uses range requests (74 MB -> 3 MB cold load) #348 and the test is now in CI (data-origin-contract.yml, push/PR/daily).
Live Explorer, cold cache, Chromium: "falling back to full HTTP read": 9 → 0; whole-file downloads 12 → 0.

Bytes, honestly (60 s cold boot): sample_facets_v4 69 MB → 21 MB; sample_facet_masks 10 MB → 0.8 MB; but samples_map_lite_v3 63 MB (once, whole) → ~134 MB (ranged, read twice), so the total is ~157 MB vs ~144 MB before. The win is time-to-usable (the page is interactive after a few MB instead of after 74 MB; 3G facets 423 s → 78 s on the canary). The double read is an Explorer-side issue, tracked in #351; fixing it turns this into a ~70 MB net saving.

Closing: the shim is deployed and verified. Remaining steps live in #348 (merge) and #351.

#345: narrow HEAD+Range shim so DuckDB-WASM uses range requests again

Merge pull request #348 from rdhyee/fix/345-head-range-206

#351: fetch samples_map_lite once, serve every lite query from an in-…

Merge pull request #346 from rdhyee/perf/313-bandwidth-findings

Navigation Menu

Repository navigation

Explorer downloads ~74 MB cold: DuckDB-WASM 1.24.0's HEAD+Range probe expects 206; our (correct) 200 defeats range requests #345

Summary

Proposed fix

Evidence — controlled A/B, not inference

Why this went unnoticed

Also answers #313

Reproducing

After the fix

Correction 1 — I was wrong that #313 had never been investigated

Correction 2 — the proposed fix is NOT standards-correct, and I should not have called it a fix

Correction 3 — the warm-cache trade-off, which the issue omitted entirely

A hypothesis we tested and killed

What else changed

Where this leaves it

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Explorer downloads ~74 MB cold: DuckDB-WASM 1.24.0's HEAD+Range probe expects 206; our (correct) 200 defeats range requests #345

Description

Summary

Proposed fix

Evidence — controlled A/B, not inference

Why this went unnoticed

Also answers #313

Reproducing

After the fix

Activity

rdhyee commented on Aug 6, 2026

Correction 1 — I was wrong that #313 had never been investigated

Correction 2 — the proposed fix is NOT standards-correct, and I should not have called it a fix

Correction 3 — the warm-cache trade-off, which the issue omitted entirely

A hypothesis we tested and killed

What else changed

Where this leaves it

rdhyee commented on Aug 27, 2026

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions