"Unable to deserialize cloned data…" / "internal error; reference = …" intermittently on dynamic worker facets

[jonastemplestein]

Message from human: This is a real but infrequent platform issue we've seen in prod and can't quite pin down so I thought this report might be useful.

But feel free to ignore if not, I won't be offended

If somebody does look at this, it'd be good to know what the error actually means / how we're meant to respond to it

AI-written issue

Repro: https://github.com/iterate/alarm-loader-facet-repro (one 96-line Worker, deploy and open its URL once)

What the program does

Thirty Durable Objects (six per region via location hints) each wake on their own alarm every 60 s. Inside alarm() each one starts 20 facets whose class comes from env.LOADER.get(id, code).getDurableObjectClass("Facet", { props }) and calls ping() on each: ten from a plain dynamic worker, ten from one whose env carries a stub of the worker's own WorkerEntrypoint (with props). Every alarm wakes a freshly constructed object (a random instance id per construction shows this). ping() returns "pong".

Expected

Every alarm line: plain ok | env ok.

Observed

Sometimes an object fails every facet call in an alarm, and keeps failing on its following alarms, each in a fresh instance. The plain facets reject with internal error; reference = <id> (a new id per call), the facets whose dynamic worker has an env stub reject with V8's Unable to deserialize cloned data due to invalid or unsupported version. The facet's constructor never runs, so the rejection comes from facet start. Real lines:

2026-09-22T12:47:59.958Z alarm 80 instance db03292e: plain ok | env ok
2026-09-22T12:48:59.964Z alarm 81 instance ea61ba3e: plain 10/10 FAILED: internal error; reference = u13k521kb6gnhu0et… | env 10/10 FAILED: Unable to deserialize cloned data due to invalid or unsupported version.
… (17 alarms in a row, 12:48:59 to 13:05:00 UTC) …
2026-09-22T13:06:00.085Z alarm 98 instance ec26cb2f: plain ok | env ok

Two catches so far on the same deployment shape: 2026-09-18 16:04–16:07 UTC (object weur-2, 4 alarms, the first of them 2 min 34 s late) and 2026-09-22 12:48–13:05 UTC (object weur-3, 17 alarms, no timing anomaly). Each time exactly one object out of sixty across two accounts; the other 29 objects in the same region were fine at the same minutes; the streak ended on its own. First failing line came 80 to 130 minutes after start.

How we observed it, and how often

Protocol: deploy the program to an account, open its URL once (that starts the objects), then leave it alone. Each object records one line per alarm in its own storage; a client fetched the URL every 10 minutes and looked for lines with FAILED. Nothing else touches the objects, so every alarm is the first act of a freshly constructed object. Two accounts were used to cover two machine pools; location hints spread the objects over five regions.

deployment	objects	alarm every	observed	alarms	failing alarms	windows
this program, account A (dev)	30 (6 × 5 regions)	60 s	2026-09-18 13:57 → 09-19 13:57 UTC (24 h)	43,200	4	1: weur-2, 16:04–16:07 (alarm 125 arrived 2 min 34 s late, then 4 in a row)
this program, account B (prod)	30	60 s	same 24 h	43,200	0	0
this program, account A, fresh redeploy	30	60 s	2026-09-22 11:28 → 09-23 09:21 UTC (22 h so far)	39,369	17	1: weur-3, 12:48:59–13:05:00, 17 in a row, alarms exactly 60 s apart
this program, account B, fresh redeploy	30	60 s	same 22 h	39,375	0	0
earlier shape, account A: 5 objects, one loader entry, 20 plain facets per alarm	5	90 s	2026-09-17 18:53 → 09-18 08:47 UTC (14 h)	2,586	22	6: 20:17–20:23, 20:20, 20:48, 21:29, 21:58, 01:43 (several objects in the same second each time)
earlier shape, account A: 10 objects, facets with an env stub + callbacks into the parent	10	75 s	2026-09-17 20:16 → 09-18 08:47 UTC (12.5 h)	5,526	3	1: 20:36:15, three objects in the same second
earlier shape, account A: 10 objects, nine variants per alarm (leave-one-out)	10	75 s	2026-09-17 20:50 → 09-18 08:47 UTC (12 h)	5,678	1	1: 01:42:47, one object, all nine variants at once

So: with objects spread over regions, one window per roughly 1,400 object-hours, hitting one object and lasting 4 to 17 alarms; with five objects that happened to share a machine (the first shape), six windows in the first four hours and then hours of quiet. The windows are per machine: objects that fail together fail in the same second, objects elsewhere are untouched, and the two programs on the same account hit different windows. Every window ended on its own with the object's next alarm succeeding. The prod-account copy never caught one in 46 hours, which we read as luck of placement rather than a difference between accounts, since our production worker on that account sees the failure on its own facets (support ticket).

Timeline and identifiers

Everything below is on account 376ef7ed81b0573f93524de763666c15, Worker alarm-loader-facet-repro
(https://alarm-loader-facet-repro.iterate-dev-preview.workers.dev). An alarm invocation carries no
cf-ray (there is no HTTP request), so the ids are the requestId / traceId Workers Logs records
for each invocation. Times are UTC, to the millisecond, as the Durable Object itself recorded them;
they match the Workers Logs rows exactly.

Every failing invocation reports outcome: ok — the program catches each rejection and writes a
line — so they do not appear as errors anywhere. What does stand out is duration: while failing, the
whole pass (20 facet starts, all rejecting) finishes in 14–20 ms, against 33–92 ms for the
same object's healthy passes either side.

Catch 2 — 2026-09-22, Worker version f9567990-9986-41f7-a300-d61c7dd7eead

Deployed and started at 11:28 UTC. Durable Object b6e409df153c1ec25641d620101b2346a7a5d5de3d2677ca520db60325e8342a
(idFromName("weur-3"), locationHint: "weur"). Healthy either side: alarm 79 at 12:46:59.580 (46 / 3 ms,
requestId d75640fa184a3798989567c4fbb1d06c), alarm 80 at 12:47:59.958 (90 / 3 ms, d099d0839b1e7a1c081b1770a3a846b1);
then alarm 98 at 13:06:00.085 (92 / 2 ms, 6deca7929490587b319270dc0ee24bac) and alarm 99 at 13:07:00.093
(33 / 2 ms, 954309563dfddaee5029ab1d32160539). Seventeen consecutive failing passes between them:

alarm	UTC	wall / CPU ms	requestId	traceId	first reference = of that pass
81	12:48:59.964	20 / 3	d333f7d1b1333233311016a90822f86b	0baf6d449b8178968d22d1f66f2cc59c	u13k521kb6gnhu0et2vmmsfq
82	12:49:59.972	18 / 2	796dbdd7512e6c124eeab25f900da5f6	e83db0efdebb9b86ade657f3a9f6fbbc	2glqcpjf1k8te9n3tis841rj
83	12:50:59.976	18 / 2	8a32c061255d5cd094d76f28f7dd6663	3112ffbdddf0a5fb2ded547b6d6aaa07	e8lroai7kvb1j2dd13kqri4l
84	12:51:59.981	19 / 2	2adf36603260b70d0e266cff9f5b158b	5e16875c39b126320980cb5ad49ba944	dp5ffcpe58qrfn7q236j6ap0
85	12:52:59.989	19 / 2	aec432381e13e352edc2fe2a6dbe0cf4	25a1b2d9e6b606422c009796538a9abf	8ktp4lul0lf2jg7g1lsqng88
86	12:53:59.992	14 / 1	c799d156fc8a733b994b1671afc1ceb8	638848281cf9b7a74eae2dd59b2e3cfc	1p9g1a58pg0vt89qrm43k631
87	12:54:59.996	17 / 2	eaae20dd8b86f2d2634067a013b3ac58	4db11d550fdffcb8366221eaa10946ba	4o7tt35vo9fhvudl3dm9gdqf
88	12:56:00.001	19 / 3	13abbbf5d4608ebc1b01b31e27cdd390	785b77a2023a429107957d3d44125737	ojklam2k2t7ioh8d226bhhie
89	12:57:00.004	18 / 2	41397664b74f655ec7503a690e1ed76d	ed1c22871326796b43fd35ac9ca05d20	gq154uaaj0qirhp3ooiu6qh3
90	12:58:00.011	54 / 3	2edd92ab7322caeeab8bf5e84cb8621c	4e5c04427ec97758eab9204d686c6b64	s6alk9tmmdv9rpvudue2h5d7
91	12:59:00.014	15 / 2	075bdbfe9e31b8c621f002617c5b060a	a439e8d7c43b39bbc170493ea8d08e85	711qgk7bkkm0efnth0l1ct51
92	13:00:00.017	15 / 2	e195dada0715b5340706911659607d13	74d0f2b96130bcc71c91089341404c7f	66hlbi5r6a2s5k3ke7a9mnsr
93	13:01:00.021	19 / 2	845350873d7d369d2d22d267cbc4442c	bddea9cd013ebd618bcdd515ba8705db	oimttjir10euo2fsev8o4fjv
94	13:02:00.025	18 / 2	0227e6fe571530bcc102c047b135a6ad	c5d2cf0b551589240b9eb520fea96061	r6voet4ukrsgj3rtvmst0t4l
95	13:03:00.030	20 / 3	87b31a584398d6d53c4ec5020a644066	2a6547ca5f8f6982284531dfd0f671ca	efgfrrlsm49sf5i8mmus7ei6
96	13:04:00.035	20 / 3	1ee8871fde0675c425388ef92301de41	6bf7760d8a94e3abd5e10f5952f2c638	i0dil1egrlcunco3p6knoh59
97	13:05:00.039	17 / 2	ee8132d6034e8ef569cdf07a273df36d	094e537c3c8eb6940dd143b04cee937f	ootr5td7hu7hrtslji09b5qa

Each pass also produced nine further internal error; reference = … ids (one per plain facet call) and ten
identical Unable to deserialize cloned data due to invalid or unsupported version. rejections; the table lists
the first reference of each pass. The alarms stayed exactly 60 s apart throughout.

Catch 1 — 2026-09-18, Worker version 1ee1a204-e321-4557-9fab-dc657068e49e

Deployed and started at 13:57 UTC. Durable Object e74fdadf7c10ba1699c98b8f02c154d9ce33bca4bdbcb1e0fd5025e9554f69d2
(idFromName("weur-2"), locationHint: "weur"). Its previous alarm 124 ran clean at 16:01:29.422; the next one
arrived 2 min 34 s later instead of 60 s, and the object's alarm phase shifted from :29.4 to :03.5:

alarm	UTC	wall / CPU ms	requestId	traceId	first reference = of that pass
125	16:04:03.573	184 / 6	069d3db712bedf455b405a6cbbe078ca	05b507f28bc7a50c5877fd51664b9c0e	4f4r7cgj5qomq11vmhb2gc1f
126	16:05:03.584	31 / 6	6ebae68c2a369f9b77ac99cb375e6bfb	3e3deaa16afdc29353fbc6ffea78a67b	9k40hh2hja7falg3ai80asal
127	16:06:03.591	40 / 4	16a5e7f808fc2e9d516673b695d471fc	5eba37860f7e12c4df0b4c096e7e119b	b3032qc6mup298jn6er90rlh
128	16:07:03.600	32 / 6	861c9d0092fe94e912e6bee7812146f4	06dd4132ac7f9472d9b1f00a14fb3c08	gr2lk2omb10d395npi3jqtc5

Alarm 129 at 16:08:03.819 (237 / 12 ms, requestId 400faca8c9600525f0fbac0c7b48b815) was clean again, and so was
everything after. One oddity in this window: an extra alarm invocation at 16:05:04.603, 1.02 s after alarm 126,
67 ms wall and 0 ms CPU, requestId e30193820ebb81a19992af373f0b18aa — it recorded no line, so the handler
did not run to completion.

Other identifiers

• The two catches ran on different Worker versions of the same code, so this is not one bad deployment.
• The second account used as a control (30 more objects, same code, same hours) is 04b3b57291ef2626c6a8daa9d47065a7,
  Worker alarm-loader-facet-repro at alarm-loader-facet-repro.iterate.workers.dev; it logged 39,375 alarms across
  2026-09-22 11:28 → 09-23 09:21 UTC with zero failures.
• Our production worker, where we first hit this: account 04b3b57291ef2626c6a8daa9d47065a7, Worker os-next-prd,
  Durable Object class IterateContextDurableObject. Example failing objects (origin alarm), 2026-09-17:
  9a039f21b19e6a4aa8ebd49da95b7b6ced984b8cf580de6d0a0287adbdd0d14e,
  f69352aaa92b1a110afe67b28f254351a0554ecbc89cdc31fa74e74a5c24ef11,
  8e039d4cc730e10b8a102dfe80038c9d51293db1293b947629e95e23504c4888 (six consecutive failing alarm passes,
  16:16:29 → 16:25:25). Storage-reset references from the same window: s793o3b71ksr5t1o0k15th08 (16:46:05, object
  afdcf6e48f9a04466b5f99beef2a88c32b383d0a74b5230d93306e874a59aca8), 3uqpflu4a0kmeekll4j9gkjc (16:41:42).

What was ruled out (variants run inside the same alarm, so they share the machine and the window)

Module size, ctx.props reads, facet SQLite use, structured arguments, minting the class before ctx.facets.get, an async code callback, compatibility flags, one loader entry per facet or per object, twelve objects waking in the same second. When a window hits, all variants fail together; none fails otherwise. The only thing that changes anything is the env stub, and it only changes the message.

What our production worker adds

Same call path there (os-next-prd, account 04b3b57291ef2626c6a8daa9d47065a7, facets always with the env stub): alarm-only in 120 controlled trials (23 of 60 alarm-woken objects failed, 0 of 60 request-woken); once it has hit, every facet minted from the same loader entry keeps failing until the loader id changes or the object is evicted, while a facet from a different loader id works at once in the same object; many-facet alarm passes also log Internal error in Durable Object storage caused object to be reset; reference = …. A support ticket with the reference ids, object ids and timestamps is filed separately.