Skip to content

Lambda extension stops polling after a 300s invocation, hanging every later invocation on that execution environment #24218

Description

@LuccaRebelloToledo

Which SDK are you using?

@sentry/aws-serverless

SDK Version

10.36.0 (Lambda layer SentryNodeServerlessSDKv10:49). Also present on develop (10.73.0,
layer :88) — the extension source is unchanged, so this is not fixed by upgrading.

Framework Version

AWS Lambda, nodejs22.x

Steps to Reproduce

No AWS account needed — the official runtime image runs external extensions.

cd packages/aws-serverless && yarn build:extension

mkdir -p /tmp/repro/task /tmp/repro/opt/extensions /tmp/repro/opt/sentry-extension
cat > /tmp/repro/task/index.js <<'EOF'
exports.handler = async event => {
  await new Promise(r => setTimeout(r, Number(event.sleepMs || 0)));
  return { ok: true };
};
EOF
cp build/lambda-extension/sentry-extension /tmp/repro/opt/extensions/
cp build/lambda-extension/index.mjs      /tmp/repro/opt/sentry-extension/

# AWS_LAMBDA_FUNCTION_TIMEOUT matters: the emulator defaults to 300s, the same number
# under test, and a function timeout at 300s masks the bug entirely.
docker run -d --name repro -p 9000:8080 \
  -e AWS_LAMBDA_FUNCTION_TIMEOUT=900 \
  -v /tmp/repro/task:/var/task:ro -v /tmp/repro/opt:/opt:ro \
  public.ecr.aws/lambda/nodejs:22 index.handler

URL=http://localhost:9000/2015-03-31/functions/function/invocations
curl -s -XPOST $URL -d '{"sleepMs":310000}'              # {"ok":true} after ~310s
curl -s --max-time 60 -XPOST $URL -d '{"sleepMs":100}'   # never returns
docker logs repro

Expected Result

Both invocations return.

Actual Result

The second invocation's handler finishes in ~110ms and the response never comes:

handler: done
INVOKE RTDONE(status: success, produced bytes: 0, duration: 107.806000ms)
                                       <- nothing after this

External agent sentry-extension ... registered appears exactly once, so both invocations ran
on the same execution environment — the first one poisoned it. On real Lambda this ends as
Status: timeout with the handler long finished, and PostRuntimeExtensionsDuration equal to
the full function timeout.

Production numbers from one SQS worker (600s timeout, 155,083 invocations over 7 days):

statistic value
PostRuntimeExtensionsDuration p50 / p90 / p99.9 0 / 0 / 0.04 ms
maximum 599,949.55 ms

37 invocations hit Status: timeout in that window, against 33 that ran past 300s and did not
— about one poisoned environment each, which is what the mechanism predicts.

Additional Context

Mechanism. /event/next both acknowledges the previous event and waits for the next one,
so the poll issued after each event stays open for the whole of the following invocation.
Node's fetch applies undici's headersTimeout, default 300,000 ms — measured against a
server that accepts the connection and never sends headers:
rejected after 301.0 s -> UND_ERR_HEADERS_TIMEOUT. Frozen time does not count against it
(undici's clock is tick-driven, fastNow += TICK_MS), so only a genuinely long invocation
triggers this, which is why the cliff sits exactly at 300s.

The rejection then escapes the loop, which has no try/catch:

while (true) { await extension.next(); }
main().catch(err => { DEBUG_BUILD && debug.error('Error in Lambda Extension', err); });

Why it is silent. Lambda ends an invocation only once the runtime and every registered
extension
have asked for the next event, so the long invocation itself completes normally and
later ones hang with nothing logged. And debug is only enabled from Sentry.init, which this
separate process never calls — debug.error here cannot print under any option or env var.

Scope. The extension is enabled by default for layer users, and useLayerExtension: false
does not help: the layer ships the binary in /opt/extensions/, which Lambda starts regardless
of SDK options. Reproduced identically with the published SentryNodeServerlessSDKv10:88 layer,
and does not reproduce with a patched extension.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

Projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions