Skip to content

fix: retry nodejs/python uprobe attach on transient failure - #342

Open
ybucci wants to merge 1 commit into
coroot:mainfrom
ybucci:fix/nodejs-python-instrumentation-retry
Open

fix: retry nodejs/python uprobe attach on transient failure#342
ybucci wants to merge 1 commit into
coroot:mainfrom
ybucci:fix/nodejs-python-instrumentation-retry

Conversation

@ybucci

@ybucci ybucci commented Aug 2, 2026

Copy link
Copy Markdown

What

instrumentNodejs/instrumentPython mark nodejsChecked/pythonGilChecked as done unconditionally, even when the eBPF uprobe attach fails. Since instrument() calls them exactly once per process, a single failed attach permanently disables Node.js/Python instrumentation for that process — silently, with no retry and no error surfaced anywhere but the log line.

Why it matters

This fires often right after the agent restarts (or on any container-restart burst): the registry rescans every already-running container at once, so many processes race to attach uprobes against the same shared *ebpf.Program (t.uprobes[...]) at the same moment. Observed on a real cluster:

E nodejs.go:32] pid=X ...: failed to attach nodejs uprobes: prog cannot be nil: invalid input
E nodejs.go:32] pid=Y ...: failed to attach nodejs uprobes: symbol uv__io_poll not found

--instrumentation-delay doesn't help here — it only protects processes that just started; already-running processes rescanned on agent restart are already past the delay, so they all attach concurrently with zero spacing.

Net effect: whether a given container ends up with working Node.js/Python metrics is effectively random per restart, with no operator-visible signal that it happened (the Node.js/Python inspection tab in the UI just silently doesn't show up).

Fix

instrumentNodejs/instrumentPython now return bool (done or not) and only set checked = true on success, on a definitive "not this language" result, or after maxInstrumentAttachAttempts (5) tries. instrument()'s existing backoff loop (already used for the "process not ready yet" case) now keeps retrying until both are resolved, instead of unconditionally returning after the first pass.

Verification

Built and deployed to a real k3s cluster (patched v1.35.2). Confirmed via logs:

  • Processes that failed with prog cannot be nil (the concurrency-race case) succeeded on retry within ~1-2s.
  • Processes that failed with symbol uv__io_poll not found (a separate, deterministic issue — worth its own investigation, not addressed here) correctly exhausted retries and gave up after 5 attempts, same as before.

Happy to adjust the retry count/backoff or split this differently if you'd prefer a different shape.

instrumentNodejs and instrumentPython marked pythonGil/nodejs state as
done unconditionally, even when AttachNodejsProbes /
AttachPythonThreadLockProbes failed. Since instrument() only called
them once per process, a single failed attach permanently disabled
Node.js/Python instrumentation for that process's lifetime, with no
retry and no user-visible error (only a log line).

In practice this fails often right after the agent restarts: it
rescans every already-running container at once, and many processes
race to attach uprobes against the same shared *ebpf.Program at the
same time, producing sporadic errors such as:

  failed to attach nodejs uprobes: prog cannot be nil: invalid input
  failed to attach nodejs uprobes: symbol uv__io_poll not found

--instrumentation-delay only protects newly-started processes; it
does nothing for this rescan-on-restart case since already-running
processes are already past the delay.

instrument() is split into two phases: wait until the process is
ready (exe/cmdline resolved) and run the one-shot .NET detection, then
retry the Node.js/Python attach via the existing backoff loop until
both resolve. instrumentNodejs/instrumentPython return a bool (done or
not) and only mark done on success, a definitive "not this language"
result, or after maxInstrumentAttachAttempts (5) tries. The shared
retry bookkeeping (done flag + attempt count) is factored into a small
retryState type instead of being duplicated per language.

Verified against a real cluster: processes that failed with "prog
cannot be nil" (the race case) succeeded on retry within ~1-2s;
processes that failed with "symbol not found" (a different,
deterministic issue: a stripped binary with no uv__io_poll symbol
exported, not something a retry can fix) correctly exhausted retries
and gave up, as expected.
@ybucci
ybucci force-pushed the fix/nodejs-python-instrumentation-retry branch from e7858e0 to a9d9610 Compare August 2, 2026 20:11
@def

def commented Aug 3, 2026

Copy link
Copy Markdown
Member

The overall idea is that if we couldn't instrument the app on the first attempt, there's almost no chance that a second attempt will succeed. You mentioned that prog cannot be nil is related to "the concurrency race" case. Could you elaborate on that?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants