fix(builders): shim __dirname/__filename in fully-bundled ESM output - #3876
Conversation
esbuild leaves the CJS globals __dirname/__filename as free identifiers when inlining CJS modules into ESM output, so dependencies that reference them at module scope (google-gax via @google-cloud/pubsub, Prisma's runtime) crash the deployed Vercel function at init with 'ReferenceError: __dirname is not defined in ES module scope' before any workflow code runs. v4 was immune because the Build Output API function was CJS; #1562 switched it to ESM with a banner that shimmed only require(). Extend the ESM banner to define __filename/__dirname from import.meta.url, matching the shim verified live in #2770.
🦋 Changeset detectedLatest commit: 0e44ce9 The changes in this PR will be included in the next version bump. This PR includes changesets to release 16 packages
Not sure what this means? Click here to learn what changesets are. Click here if you're a maintainer who wants to add another changeset to this PR |
🧪 E2E Test Results❌ Some tests failed ❌ Failed E2E Tests▲ Vercel Production (8 failed)python-node (8 failed):
🌐 Cross-language Conformance (9 failed)python (9 failed):
|
| Passed | Failed | Skipped | Total | |
|---|---|---|---|---|
| ❌ ▲ Vercel Production | 3570 | 8 | 742 | 4320 |
| ✅ 💻 Local Development | 3922 | 0 | 558 | 4480 |
| ✅ 📦 Local Production | 3922 | 0 | 558 | 4480 |
| ✅ 🐘 Local Postgres | 3922 | 0 | 558 | 4480 |
| ✅ 🪟 Windows | 320 | 0 | 0 | 320 |
| ❌ 🌐 Cross-language Conformance | 0 | 9 | 132 | 141 |
| ✅ vercel-http-transport | 817 | 0 | 143 | 960 |
| ✅ vercel-multi-region | 27 | 0 | 0 | 27 |
| ✅ vercel-ws-transport | 553 | 0 | 87 | 640 |
| Total | 17053 | 17 | 2778 | 19848 |
Details by Category
❌ ▲ Vercel Production
| App | Passed | Failed | Skipped |
|---|---|---|---|
| ✅ astro-node | 132 | 0 | 28 |
| ✅ astro-quickjs | 132 | 0 | 28 |
| ✅ example-node | 132 | 0 | 28 |
| ✅ example-quickjs | 132 | 0 | 28 |
| ✅ express-node | 132 | 0 | 28 |
| ✅ express-quickjs | 132 | 0 | 28 |
| ✅ fastify-node | 132 | 0 | 28 |
| ✅ fastify-quickjs | 132 | 0 | 28 |
| ✅ hono-node | 132 | 0 | 28 |
| ✅ hono-quickjs | 132 | 0 | 28 |
| ✅ nest-node | 132 | 0 | 28 |
| ✅ nest-quickjs | 132 | 0 | 28 |
| ✅ nextjs-turbopack-node | 157 | 0 | 3 |
| ✅ nextjs-turbopack-quickjs | 157 | 0 | 3 |
| ✅ nextjs-webpack-node | 157 | 0 | 3 |
| ✅ nextjs-webpack-quickjs | 157 | 0 | 3 |
| ✅ nitro-node | 132 | 0 | 28 |
| ✅ nitro-quickjs | 132 | 0 | 28 |
| ✅ nuxt-node | 132 | 0 | 28 |
| ✅ nuxt-quickjs | 132 | 0 | 28 |
| ❌ python-node | 0 | 8 | 152 |
| ✅ sveltekit-node | 151 | 0 | 9 |
| ✅ sveltekit-quickjs | 151 | 0 | 9 |
| ✅ tanstack-start-node | 132 | 0 | 28 |
| ✅ tanstack-start-quickjs | 132 | 0 | 28 |
| ✅ vite-node | 132 | 0 | 28 |
| ✅ vite-quickjs | 132 | 0 | 28 |
✅ 💻 Local Development
| App | Passed | Failed | Skipped |
|---|---|---|---|
| ✅ astro-stable-node | 134 | 0 | 26 |
| ✅ astro-stable-quickjs | 134 | 0 | 26 |
| ✅ express-stable-node | 134 | 0 | 26 |
| ✅ express-stable-quickjs | 134 | 0 | 26 |
| ✅ fastify-stable-node | 134 | 0 | 26 |
| ✅ fastify-stable-quickjs | 134 | 0 | 26 |
| ✅ hono-stable-node | 134 | 0 | 26 |
| ✅ hono-stable-quickjs | 134 | 0 | 26 |
| ✅ nest-stable-node | 134 | 0 | 26 |
| ✅ nest-stable-quickjs | 134 | 0 | 26 |
| ✅ nextjs-turbopack-canary-node | 141 | 0 | 19 |
| ✅ nextjs-turbopack-canary-quickjs | 141 | 0 | 19 |
| ✅ nextjs-turbopack-stable-node | 160 | 0 | 0 |
| ✅ nextjs-turbopack-stable-quickjs | 160 | 0 | 0 |
| ✅ nextjs-webpack-canary-node | 141 | 0 | 19 |
| ✅ nextjs-webpack-canary-quickjs | 141 | 0 | 19 |
| ✅ nextjs-webpack-stable-node | 160 | 0 | 0 |
| ✅ nextjs-webpack-stable-quickjs | 160 | 0 | 0 |
| ✅ nitro-stable-node | 134 | 0 | 26 |
| ✅ nitro-stable-quickjs | 134 | 0 | 26 |
| ✅ nuxt-stable-node | 134 | 0 | 26 |
| ✅ nuxt-stable-quickjs | 134 | 0 | 26 |
| ✅ sveltekit-stable-node | 153 | 0 | 7 |
| ✅ sveltekit-stable-quickjs | 153 | 0 | 7 |
| ✅ tanstack-start-node | 134 | 0 | 26 |
| ✅ tanstack-start-quickjs | 134 | 0 | 26 |
| ✅ vite-stable-node | 134 | 0 | 26 |
| ✅ vite-stable-quickjs | 134 | 0 | 26 |
✅ 📦 Local Production
| App | Passed | Failed | Skipped |
|---|---|---|---|
| ✅ astro-stable-node | 134 | 0 | 26 |
| ✅ astro-stable-quickjs | 134 | 0 | 26 |
| ✅ express-stable-node | 134 | 0 | 26 |
| ✅ express-stable-quickjs | 134 | 0 | 26 |
| ✅ fastify-stable-node | 134 | 0 | 26 |
| ✅ fastify-stable-quickjs | 134 | 0 | 26 |
| ✅ hono-stable-node | 134 | 0 | 26 |
| ✅ hono-stable-quickjs | 134 | 0 | 26 |
| ✅ nest-stable-node | 134 | 0 | 26 |
| ✅ nest-stable-quickjs | 134 | 0 | 26 |
| ✅ nextjs-turbopack-canary-node | 141 | 0 | 19 |
| ✅ nextjs-turbopack-canary-quickjs | 141 | 0 | 19 |
| ✅ nextjs-turbopack-stable-node | 160 | 0 | 0 |
| ✅ nextjs-turbopack-stable-quickjs | 160 | 0 | 0 |
| ✅ nextjs-webpack-canary-node | 141 | 0 | 19 |
| ✅ nextjs-webpack-canary-quickjs | 141 | 0 | 19 |
| ✅ nextjs-webpack-stable-node | 160 | 0 | 0 |
| ✅ nextjs-webpack-stable-quickjs | 160 | 0 | 0 |
| ✅ nitro-stable-node | 134 | 0 | 26 |
| ✅ nitro-stable-quickjs | 134 | 0 | 26 |
| ✅ nuxt-stable-node | 134 | 0 | 26 |
| ✅ nuxt-stable-quickjs | 134 | 0 | 26 |
| ✅ sveltekit-stable-node | 153 | 0 | 7 |
| ✅ sveltekit-stable-quickjs | 153 | 0 | 7 |
| ✅ tanstack-start-node | 134 | 0 | 26 |
| ✅ tanstack-start-quickjs | 134 | 0 | 26 |
| ✅ vite-stable-node | 134 | 0 | 26 |
| ✅ vite-stable-quickjs | 134 | 0 | 26 |
✅ 🐘 Local Postgres
| App | Passed | Failed | Skipped |
|---|---|---|---|
| ✅ astro-stable-node | 134 | 0 | 26 |
| ✅ astro-stable-quickjs | 134 | 0 | 26 |
| ✅ express-stable-node | 134 | 0 | 26 |
| ✅ express-stable-quickjs | 134 | 0 | 26 |
| ✅ fastify-stable-node | 134 | 0 | 26 |
| ✅ fastify-stable-quickjs | 134 | 0 | 26 |
| ✅ hono-stable-node | 134 | 0 | 26 |
| ✅ hono-stable-quickjs | 134 | 0 | 26 |
| ✅ nest-stable-node | 134 | 0 | 26 |
| ✅ nest-stable-quickjs | 134 | 0 | 26 |
| ✅ nextjs-turbopack-canary-node | 141 | 0 | 19 |
| ✅ nextjs-turbopack-canary-quickjs | 141 | 0 | 19 |
| ✅ nextjs-turbopack-stable-node | 160 | 0 | 0 |
| ✅ nextjs-turbopack-stable-quickjs | 160 | 0 | 0 |
| ✅ nextjs-webpack-canary-node | 141 | 0 | 19 |
| ✅ nextjs-webpack-canary-quickjs | 141 | 0 | 19 |
| ✅ nextjs-webpack-stable-node | 160 | 0 | 0 |
| ✅ nextjs-webpack-stable-quickjs | 160 | 0 | 0 |
| ✅ nitro-stable-node | 134 | 0 | 26 |
| ✅ nitro-stable-quickjs | 134 | 0 | 26 |
| ✅ nuxt-stable-node | 134 | 0 | 26 |
| ✅ nuxt-stable-quickjs | 134 | 0 | 26 |
| ✅ sveltekit-stable-node | 153 | 0 | 7 |
| ✅ sveltekit-stable-quickjs | 153 | 0 | 7 |
| ✅ tanstack-start-node | 134 | 0 | 26 |
| ✅ tanstack-start-quickjs | 134 | 0 | 26 |
| ✅ vite-stable-node | 134 | 0 | 26 |
| ✅ vite-stable-quickjs | 134 | 0 | 26 |
✅ 🪟 Windows
| App | Passed | Failed | Skipped |
|---|---|---|---|
| ✅ nextjs-turbopack-node | 160 | 0 | 0 |
| ✅ nextjs-turbopack-quickjs | 160 | 0 | 0 |
❌ 🌐 Cross-language Conformance
| App | Passed | Failed | Skipped |
|---|---|---|---|
| ❌ python | 0 | 9 | 132 |
✅ vercel-http-transport
| App | Passed | Failed | Skipped |
|---|---|---|---|
| ✅ example | 132 | 0 | 28 |
| ✅ express | 132 | 0 | 28 |
| ✅ hono | 132 | 0 | 28 |
| ✅ nextjs-turbopack | 157 | 0 | 3 |
| ✅ nitro | 132 | 0 | 28 |
| ✅ vite | 132 | 0 | 28 |
✅ vercel-multi-region
| App | Passed | Failed | Skipped |
|---|---|---|---|
| ✅ nextjs-turbopack | 27 | 0 | 0 |
✅ vercel-ws-transport
| App | Passed | Failed | Skipped |
|---|---|---|---|
| ✅ example | 132 | 0 | 28 |
| ✅ express | 132 | 0 | 28 |
| ✅ nextjs-turbopack | 157 | 0 | 3 |
| ✅ vite | 132 | 0 | 28 |
📊 Workflow Benchmarkscommit Backend:
Streams
📈 STSO distribution vs main (inline / queue-hop histograms)1020 steps (inline) Cumulative STSO time: main 138419ms → this run 140590ms (Δ +2171ms, +2%) 📈 CRTT drill-down vs main (RTT distributions & profiles)RTT over stream progress (avg per tenth of stream, bars scaled min→max): RTT by chunk size (avg per log size bin, ~160B → ~12KB serialized, bars scaled min→max): Delivery jitter over stream progress (avg positive CDV per tenth of stream, bars scaled min→max): 📜 Previous results (2)4bb2883Fri, 28 Aug 2026 03:57:47 GMT · run logs
Streams
b7e64beThu, 27 Aug 2026 23:46:20 GMT · run logs
Streams
ℹ️ Metric definitions & methodologyStreams: first-chunk RTT (the stream-open path, before any buffering/backpressure), CRTT percentiles, and worst delivery stall (CDV max). Cells are medians across iterations; per-run values in the artifacts. No 🔴/🟢 marks until targets attach. The collapsed STSO distribution section above buckets every step gap, split inline (same warm process — pure framework overhead) vs queue-hop (fresh process — dispatch, reinit, replay). The collapsed CRTT drill-down: per-variant RTT histograms (fixed log bins, Best/P75/P90/P99 deltas compare against the most recent benchmark run on Metrics — TTFS: time to first step body (in-deployment start() → first step body) · Fan-out TTFS: fan-out time to first step (in-deployment start() → first of the parallel step bodies to complete) · Fan-out TTLS: fan-out time to last step (in-deployment start() → last of the parallel step bodies to complete, i.e. when the Promise.all resolves) · STSO: step-to-step overhead (gap between consecutive step bodies) · WO: workflow overhead (whole-run time outside step bodies, in-deployment anchored) · CRTT: chunk round-trip time (per-chunk write → read latency, one clock domain: deployment → stream backend → same deployment) · CDV: chunk delay variation / delivery jitter (inter-arrival gap minus inter-write gap per seq-adjacent pair; skew-free; the row is each run's MAX positive value, so one stall moves it) Scenarios — step: one trivial no-op step, no stream; no hooks, so the run stays in turbo mode (in-process fast path) · stream: one streaming step; no hooks, so the run stays in turbo mode (in-process fast path) · hook + stream: registers a hook before one step, which exits turbo mode (dispatch path) · 1020 steps: 1020 trivial sequential steps; STSO is measured between consecutive steps in the given step ranges, and WO is the whole-run overhead outside step bodies · Promise.all(100 steps): 100 trivial no-op steps started together in a single Promise.all; Fan-out TTFS is the first of them to complete and Fan-out TTLS the last, both from the in-deployment clientStart, so their gap is the spread the runtime adds across the fan-out · paced control (100/s, 60B): the control: 300 tiny (~60B) deltas metronome-paced at 100/s — zero workload structure, so it reads the transport floor and flush cadence, and disambiguates transport-wide vs workload-specific when a replay row moves · size sweep (100/s, 160B-12KB): same pacing as the control with deltas padded in rotation across seven log-spaced sizes (~160B–12KB) — rotation decouples size from stream position, so it isolates whether chunk size causes latency · replay gateway-gpt-5.4-nano-2000t (1x): raw provider SSE cadence captured at the AI gateway boundary (gpt-5.4-nano, the most popular gateway model; per-token deltas p50 208B = the modal production chunk size), replayed exactly as measured — the typical customer's workload; its CDV is the typical customer's real delivery jitter · replay eve-gpt-5.6-sol-2000t (1x): a captured eve turn (gpt-5.6-sol, the most-used demanding eve model; ~2000 output tokens = production p50 turn length) replayed exactly as measured — eve's envelope protocol re-ships the cumulative message so sizes ramp 142B→13KB; the demanding outlier tenant's reality · replay eve-gpt-5.6-sol-2000t (2x): the same eve capture at 2x — the headroom/stress row; real fast-tier models emit the same chunk sizes at proportionally higher rate, so time compression is a faithful speed model · first chunk (pooled): every run's seq-0 RTT pooled across all stream scenarios — the first chunk precedes any workload differentiation, so pooling samples one shared stream-open path with exact percentiles Replay cadences (semantic sha256) — eve-gpt-5.6-sol-2000t 🔴 marks a percentile over its target (within target is left unmarked). Targets (p75/p90/p99, ms) — TTFS 200/300/600 All timestamps are deployment-side; runs are triggered in-deployment, so the CI runner and api.vercel.com sit outside every measured window. TTFS = Cold starts stay in the numbers (real bursty-workload latency, inflates P75+); Best is the warm floor. |
Sim WorldSimulated world deterministic testing for races. Traces 🟠 world-sim scenario book — 1 fail of 41 total
Full trace: |
About these numbersSizes are gzip; parentheses show the change against
|
pranaygp
left a comment
pranaygp
left a comment
There was a problem hiding this comment.
Reviewed the diff and verified the fix end-to-end locally. The diagnosis is right and the fix is the standard interop shim — no blocking concerns. One forward-looking naming issue inline, plus a test nit.
What I verified
- The test is a genuine regression test: reverting only the banner change makes it fail with exactly
ReferenceError: __dirname is not defined in ES module scope; restoring it passes. Fullpackages/builderssuite is 250/251, and the one failure (workflow-bundle-boundary.test.ts, zod in the workflow bundle) reproduces onmain. - The shim cannot leak into the workflow sandbox. The interim VM bundle is
format: 'cjs'/platform: 'neutral', sogetEsmRequireBannerreturns''for it. That matters:__dirnamebeing absent inside the VM is intentional (#374), and this PR correctly leaves that alone. - All four
esmbanner sites areplatform: 'node', so the newnode:url/node:pathimports always resolve. - No collision hazard from
var __dirnameat module scope. I specifically probed the case that worried me — a bundled ESM dep using the ubiquitousconst __dirname = dirname(fileURLToPath(import.meta.url))idiom. esbuild reserves__dirnamewhenever any module references it free and renames the declaration to__dirname2, so the CJS dep still reads the banner's value. Confirmed with a standalone esbuild repro; noSyntaxError, no clobbering. tsc --noEmitandbiome checkclean on the changed files.- The
python-node/pythonE2E failures are not from this PR — the same 8/9 failures appear on #3877 and #3873.
On linking issues to close
I searched issue bodies and comments; there is no existing report of this exact bug, so I'd avoid Fixes on anything:
- #1956 (Prisma engine binary) and #1003 (sharp) are the tempting ones, and #2770 claims both — but they need that PR's runtime asset tracing. This shim only removes the earlier
__dirnamecrash that #2770's own description mentions hitting first. Link as "unblocks the init crash in", not as closing. - #3778 ([nest] steps bundle declares
requiretwice) is the closest live relative — same banner, same dedup flag. See the inline note; it's a trap for whoever fixes it, not a break here. - #374 and #1161 match by title but are the VM-sandbox class (
__dirname/requireinsidenode:vm), which this deliberately does not change. Don't link them as fixed.
| 'import { dirname as __pathDirname } from "node:path";\n' + | ||
| 'var require = __createRequire(import.meta.url);\n' + | ||
| 'var __filename = __fileURLToPath(import.meta.url);\n' + | ||
| 'var __dirname = __pathDirname(__filename);\n' |
There was a problem hiding this comment.
skipEsmRequireBanner now gates three globals — worth renaming before #3778 gets fixed
This banner is now the sole provider of require, __filename, and __dirname for every fully-bundled ESM output, but the flag that suppresses it is still skipEsmRequireBanner, and its JSDoc (L923-928) still describes it purely in terms of __createRequire being declared twice.
That is a live trap rather than a cosmetic one. #3778 ([nest] steps bundle declares require twice) is open, and the issue itself identifies skipEsmRequireBanner as the mechanism that should have prevented it: NestLocalBuilder calls createCombinedBundle({ bundleFinalOutput: false }), so the flag resolves to false, the steps bundle keeps the banner, and it collides with the const require = ... that #rewriteStepsBundleForCjs() prepends.
To be clear, this PR does not make #3778 worse — nest's shim only declares require, so no new duplicate identifier appears. But the obvious fix for #3778 (pass skipEsmRequireBanner: true from NestLocalBuilder) would now also strip __filename/__dirname from the nest steps bundle, silently reintroducing exactly the crash this PR fixes — for nest CJS users only, with no test covering it.
Suggest renaming to something like skipEsmInteropBanner and updating the JSDoc at L923-928 to say it gates all three globals, so whoever fixes #3778 has to reason about the dirname shim rather than discover it in production.
| ); | ||
| expect( | ||
| bundle.match(/var __dirname = __pathDirname\(__filename\);/g) | ||
| ).toHaveLength(1); |
There was a problem hiding this comment.
Two small things, neither blocking:
-
Other emit sites are uncovered. This exercises the combined flow route (
createCombinedBundle+bundleFinalOutput: true), which is the reported crash — but the shim is also emitted fromcreateWorkflowsBundle's final wrapper (L1550) and the webhook bundle (L2031, which hardcodesgetEsmRequireBanner('esm')). The webhook route is a separately deployed function, so a CJS dep with module-scope__dirnamereachable from it would have failed identically. Worth at least noting in the comment that those paths are untested. -
The assertion guards the second-order symptom. A double banner's first failure is duplicate
import { fileURLToPath as __fileURLToPath }bindings — a parse-timeSyntaxError— which theexecuteStepimport on L150 already catches. Matching one of the import lines instead of thevar __dirname = ...line would fail closer to the real cause. Belt-and-braces as written, so take it or leave it.
There was a problem hiding this comment.
Both taken: the import-binding assertion landed in 4bb2883, and rather than only noting the uncovered emit sites, 0e44ce9 asserts the webhook function bundle (its own esbuild pass, separately deployed) carries the shim too. createWorkflowsBundle's final wrapper remains covered only via the combined bundle, which the comment now says.
…terop shim The webhook route is a separately deployed function built through its own esbuild pass (createWebhookBundle); a CJS dependency referencing __dirname reachable from it would have crashed identically, so cover that emit site rather than only noting it as untested.
|
No backport to This is a genuine crash fix, but the defect it fixes does not exist on To override, re-run the Backport to stable workflow manually via |
Problem
A customer upgrading from workflow v4 to v5 (
5.0.0-beta.46) hit a hard init crash on every flow-route invocation:The function dies with
Runtime.ExitErrorbefore any workflow code runs, so the flow route returns 500 and runs sit inpendingforever.Root cause
#1562 switched the Vercel Build Output API function from CJS (
index.js) to a fully-bundled ESMindex.mjs. esbuild wraps inlined CJS modules in__commonJSclosures and our banner providesrequireviacreateRequire, but esbuild deliberately leaves the CJS globals__dirname/__filenameas free identifiers in ESM output (evanw/esbuild#1921).google-gaxis CJS-only (no ESM build exists) and computes a path from__dirnameat module scope (build/src/grpc.js:66), so the whole bundle throws at import. v4 never hit this because the function was real CJS.Fix
Extend
getEsmRequireBannerto also define__filename/__dirnamefromimport.meta.url— the standard interop shim (tsup injects the same one forformat: esm+platform: node). This is the shim already written and verified live (with@prisma/client, which fails the same way) in #2770; it's extracted here so the regression fix isn't blocked on that PR's larger runtime-asset-tracing work. The existingskipEsmRequireBannerdedup keeps it declared exactly once in the combined bundle.The shimmed
__dirnamepoints at the function root, not each dep's original package directory — that's sufficient for pubsub (protos load viarequired JSON, which esbuild inlines) and per-module-accurate asset paths remain #2770's territory.Test
New regression test builds a temp app through
VercelBuildOutputAPIBuilderwith a google-gax-shaped CJS dependency (module-scopepath.join(__dirname, ...)), then imports the emittedindex.mjsunder plain Node and executes the step. Onmainit fails with exactly the production error; with the fix it passes. It also asserts the shim appears exactly once in the bundle.Related: #2770, #1956.