fix: retry nodejs/python uprobe attach on transient failure - #342
Open
ybucci wants to merge 1 commit into
Open
Conversation
instrumentNodejs and instrumentPython marked pythonGil/nodejs state as done unconditionally, even when AttachNodejsProbes / AttachPythonThreadLockProbes failed. Since instrument() only called them once per process, a single failed attach permanently disabled Node.js/Python instrumentation for that process's lifetime, with no retry and no user-visible error (only a log line). In practice this fails often right after the agent restarts: it rescans every already-running container at once, and many processes race to attach uprobes against the same shared *ebpf.Program at the same time, producing sporadic errors such as: failed to attach nodejs uprobes: prog cannot be nil: invalid input failed to attach nodejs uprobes: symbol uv__io_poll not found --instrumentation-delay only protects newly-started processes; it does nothing for this rescan-on-restart case since already-running processes are already past the delay. instrument() is split into two phases: wait until the process is ready (exe/cmdline resolved) and run the one-shot .NET detection, then retry the Node.js/Python attach via the existing backoff loop until both resolve. instrumentNodejs/instrumentPython return a bool (done or not) and only mark done on success, a definitive "not this language" result, or after maxInstrumentAttachAttempts (5) tries. The shared retry bookkeeping (done flag + attempt count) is factored into a small retryState type instead of being duplicated per language. Verified against a real cluster: processes that failed with "prog cannot be nil" (the race case) succeeded on retry within ~1-2s; processes that failed with "symbol not found" (a different, deterministic issue: a stripped binary with no uv__io_poll symbol exported, not something a retry can fix) correctly exhausted retries and gave up, as expected.
ybucci
force-pushed
the
fix/nodejs-python-instrumentation-retry
branch
from
August 2, 2026 20:11
e7858e0 to
a9d9610
Compare
Member
|
The overall idea is that if we couldn't instrument the app on the first attempt, there's almost no chance that a second attempt will succeed. You mentioned that prog cannot be nil is related to "the concurrency race" case. Could you elaborate on that? |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
instrumentNodejs/instrumentPythonmarknodejsChecked/pythonGilCheckedas done unconditionally, even when the eBPF uprobe attach fails. Sinceinstrument()calls them exactly once per process, a single failed attach permanently disables Node.js/Python instrumentation for that process — silently, with no retry and no error surfaced anywhere but the log line.Why it matters
This fires often right after the agent restarts (or on any container-restart burst): the registry rescans every already-running container at once, so many processes race to attach uprobes against the same shared
*ebpf.Program(t.uprobes[...]) at the same moment. Observed on a real cluster:--instrumentation-delaydoesn't help here — it only protects processes that just started; already-running processes rescanned on agent restart are already past the delay, so they all attach concurrently with zero spacing.Net effect: whether a given container ends up with working Node.js/Python metrics is effectively random per restart, with no operator-visible signal that it happened (the
Node.js/Pythoninspection tab in the UI just silently doesn't show up).Fix
instrumentNodejs/instrumentPythonnow returnbool(done or not) and only setchecked = trueon success, on a definitive "not this language" result, or aftermaxInstrumentAttachAttempts(5) tries.instrument()'s existing backoff loop (already used for the "process not ready yet" case) now keeps retrying until both are resolved, instead of unconditionally returning after the first pass.Verification
Built and deployed to a real k3s cluster (patched
v1.35.2). Confirmed via logs:prog cannot be nil(the concurrency-race case) succeeded on retry within ~1-2s.symbol uv__io_poll not found(a separate, deterministic issue — worth its own investigation, not addressed here) correctly exhausted retries and gave up after 5 attempts, same as before.Happy to adjust the retry count/backoff or split this differently if you'd prefer a different shape.