fix: segments/logs lost when two processes first report for the same service concurrently - #65
Merged
Merged
Conversation
… first time addSegmentItem and addLogItem look the service name up and then put it back as two separate operations. When two agent processes report their FIRST data for the same service name concurrently, both threads observe null, both construct an item, and the second put overwrites the first — the segments already appended to the discarded item are lost while both reporters receive HTTP 200. Once the key exists there is no loss, since SegmentItem/LogItem hold CopyOnWriteArrayList, so only the very first insert per service name is affected. This surfaced in apache/skywalking-python#409, where a plugin test forks a worker: parent and child share one service name and post their first segments milliseconds apart, so a segment vanished in roughly 2 of 5 CI runs. Reproduced directly against this collector with barrier- synchronized first-time POSTs: 15 of 200 rounds lost a segment, and 0 of 200 once the key had been seeded. Use computeIfAbsent, which performs the whole get-or-create atomically — the pattern MeterItems.addMeter already uses in this package. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
wu-sheng
added a commit
to apache/skywalking-python
that referenced
this pull request
Aug 1, 2026
…aio compatibility, websockets >= 13 support (#409) fix: never create a gRPC channel in the Gunicorn prefork master (#409) With grpcio >= 1.80 (EventEngine) a gRPC channel that lives across fork() breaks: gRPC's at-fork handlers self-skip while its own threads are inside gRPC, leaving stale poll-engine eventfds behind. Under `sw-python run -p gunicorn` the agent started fully in the master, so every worker inherited such a channel — producing the continuous `Kick Failure (eventfd_write: Bad file descriptor)` spam and, less visibly, a racy deadlock in which a forked worker never boots and Gunicorn never notices. grpcio 1.83.0 removed the last legacy-poller opt-out (grpc/grpc#42828); upstream grpc/grpc#43055 and grpc/grpc#43062 are still open. The failure is entirely client-side and unrelated to the OAP version. The master now only installs instrumentation and arms the fork hooks; the queues, reporter threads and gRPC channel are created in each forked worker, which is gRPC's supported model and what the uWSGI path has always done. The master therefore no longer registers as a service instance. A new agent.started() guard turns instrumented code into a no-op wherever the reporters are inactive, keeping `gunicorn --preload` app imports safe in the master. GRPC_POLL_STRATEGY is no longer set, the grpcio floor moves to >= 1.83 (the generated stubs already require it at import) and codegen grpcio-tools is pinned in lockstep. Gunicorn prefork combined with SW_AGENT_ASYNCIO_ENHANCEMENT is now rejected instead of silently starting an unsafe pre-fork agent, and explicit os.fork() over gRPC is documented — and warned about at runtime — as unreliable, directing those users to the HTTP or Kafka reporter. Also repairs two unrelated cases that had been failing on master CI: - sw_grpc: grpcio 1.83.0 validates `isinstance(x, Channel)` against the grpc.aio._channel module global, which the plugin had rebound to a factory function, so every aio stub creation raised TypeError. It is a Channel subclass now; the supported range stays grpcio 1.*. - sw_websockets: uvicorn 0.50.0 dropped its legacy websockets fallback and unconditionally imports websockets.server.ServerProtocol (websockets >= 11 only), crashing the test provider. The harness pins uvicorn < 0.50; websockets 10.3/10.4 remain tested. The websockets plugin additionally instruments the new websockets.asyncio client (websockets >= 13, the default since 14) alongside the legacy one — applications on the modern API previously produced no spans at all. The support matrix gains 13.1 everywhere and 17.0.1 on Python >= 3.11. New plugin tests cover both fork paths end to end: sw_gunicorn validates a complete consumer -> prefork-provider trace under `--preload`, asserting exact worker boot counts, instrumentation-only behaviour in the master and the absence of the gRPC fork errors; sw_fork_support validates a continuous trace across an explicit os.fork() over the HTTP reporter. The latter seeds its service name first, working around a first-insert race in the mock collector (fixed by apache/skywalking-agent-test-tool#65). Fixes apache/skywalking#13958
CI-on-MacOS has failed since GitHub moved macos-latest to arm64: with actions/setup-java@v1, JAVA_HOME points at a hosted-toolcache path that lacks the macOS Contents/Home layout, so $JAVA_HOME/bin/java does not exist and mvnw aborts with "JAVA_HOME is not defined correctly" before compiling anything. setup-java@v1 also runs on a Node runtime GitHub has already removed. Move both jobs to actions/setup-java@v4 (and checkout@v4). The distribution must be zulu rather than the more common temurin: JDK 8 has no macOS aarch64 temurin build, while zulu publishes one. Both actions are in the actions/* namespace, which the ASF allow-list permits without a SHA pin. Java 8 remains the build JDK; verified locally with `./mvnw -pl mock-collector -am compile` under Zulu 8 (BUILD SUCCESS). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
wankai123
approved these changes
Aug 1, 2026
wu-sheng
added a commit
that referenced
this pull request
Aug 1, 2026
…66) The workflow referenced docker/login-action, docker/setup-qemu-action and docker/setup-buildx-action by floating tag. The ASF GitHub Actions allow-list only approves specific commit SHAs for third-party actions, so the workflow is rejected at startup: the run for #65 ended in startup_failure and no image was published for that commit. Since publish-docker only runs on push-to-master, the rejection never surfaces in PR CI — it is only visible after a merge. Pin all three to the approved SHAs already used across the sibling ASF SkyWalking repositories (verified against apache/infrastructure-actions approved_patterns.yml): - docker/login-action@650006c6... # v4.2.0 - docker/setup-qemu-action@06116385... # v4.1.0 - docker/setup-buildx-action@d7f5e7f5... # v4.1.0 Merging this triggers a push-to-master build, which publishes an image containing the #65 concurrency fix.
wu-sheng
added a commit
to apache/skywalking-python
that referenced
this pull request
Aug 1, 2026
The pinned mock-collector image dated from October 2022, 13 commits behind the tool's master. Move it to c6b91a0e, which carries the fix for the first-insert race in SegmentItems/LogItems (apache/skywalking-agent-test-tool#65): when two processes reported their first segment for the same service name concurrently, the collector silently dropped one while both reporters got HTTP 200. sw_fork_support worked around that race by seeding the service name with a parent-only /ping request and waiting for the collector to register it before triggering the concurrent parent/child reports. The collector no longer needs the help, so the endpoint, the seed step and the extra expected segment are gone and the test is back to asserting exactly the cross-fork trace it is about. The upgrade also picks up the validator changes made since 2022, one of which affected us. LogAssert now sorts both the expected and the actual logs by their body text before comparing them pairwise (apache/skywalking-agent-test-tool#59), and the sort uses the raw expected string, so a matcher such as `text: not null` participates as the literal "not null". sw_loguru matches its two logging-module records that way while the records themselves led with the default layout's timestamp, so they sorted first and their placeholders last, inverting the pairing. Pin a layout that leads with the logger name, which sorts after the placeholders and orders the two records deterministically, and reorder the expected entries to match; SWFormatter is still exercised. Its expected.data.yml is the only one in the tree with a non-empty logItems block, so no other test is affected. Also refresh CLAUDE.md: supported Python and grpcio floor, the current plugin list, the agent's fork/prefork lifecycle, and the plugin-test validation notes.
wu-sheng
added a commit
to apache/skywalking-python
that referenced
this pull request
Aug 2, 2026
…410) The pinned mock-collector image dated from October 2022, 13 commits behind the tool's master. Move it to c6b91a0e, which carries the fix for the first-insert race in SegmentItems/LogItems (apache/skywalking-agent-test-tool#65): when two processes reported their first segment for the same service name concurrently, the collector silently dropped one while both reporters got HTTP 200. sw_fork_support worked around that race by seeding the service name with a parent-only /ping request and waiting for the collector to register it before triggering the concurrent parent/child reports. The collector no longer needs the help, so the endpoint, the seed step and the extra expected segment are gone and the test is back to asserting exactly the cross-fork trace it is about. The upgrade also picks up the validator changes made since 2022, one of which affected us. LogAssert now sorts both the expected and the actual logs by their body text before comparing them pairwise (apache/skywalking-agent-test-tool#59), and the sort uses the raw expected string, so a matcher such as `text: not null` participates as the literal "not null". sw_loguru matches its two logging-module records that way while the records themselves led with the default layout's timestamp, so they sorted first and their placeholders last, inverting the pairing. Pin a layout that leads with the logger name, which sorts after the placeholders and orders the two records deterministically, and reorder the expected entries to match; SWFormatter is still exercised. Its expected.data.yml is the only one in the tree with a non-empty logItems block, so no other test is affected. Also refresh CLAUDE.md: supported Python and grpcio floor, the current plugin list, the agent's fork/prefork lifecycle, and the plugin-test validation notes.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
SegmentItems.addSegmentItemandLogItems.addLogItemdo a check-then-act on the map:When two agent processes report their first data for the same service name concurrently, both handler threads see
null, both create an item, and the secondputoverwrites the first — the data appended to the discarded item is silently lost while both reporters receive HTTP 200.Only the first insert per service name is affected:
SegmentItem/LogItemhold aCopyOnWriteArrayList, so appends are safe once the key exists. The map is aConcurrentHashMap, so the intent was clearly thread-safety — the get/put pair just defeats it. Every receiver funnels into these methods (gRPC and both HTTP servlets), so the loss is transport-independent.Impact
Found while debugging a flaky test in apache/skywalking-python#409: a plugin test forks a worker, so parent and child share one service name and post their first segments milliseconds apart. A segment disappeared in roughly 2 of 5 CI runs — sometimes the parent's, sometimes the child's — which looked like an agent bug for quite a while.
Reproduced directly against this collector image, from inside its network:
segmentSize=1)Fix
Use
computeIfAbsent, which performs the get-or-create atomically — the patternMeterItems.addMeteralready uses in this same package:Once a new image is published, the workaround in apache/skywalking-python#409 (seeding the service name with a warm-up request before the concurrent pair) can be removed.
Also: repair the macOS CI job
CI-on-MacOShas been failing since GitHub movedmacos-latestto arm64 — unrelated to this change, but it blocks every PR. Withactions/setup-java@v1,JAVA_HOMEpoints at a hosted-toolcache path lacking the macOSContents/Homelayout, so$JAVA_HOME/bin/javadoes not exist andmvnwaborts withJAVA_HOME is not defined correctlybefore compiling anything (the log hands out an x64 JDK 8 path on an arm64 runner).setup-java@v1also runs on a Node runtime GitHub has already removed.Both jobs move to
actions/setup-java@v4andactions/checkout@v4. The distribution must be zulu, not temurin: JDK 8 has no macOS aarch64 temurin build, while zulu publishes one. Both actions are in theactions/*namespace, which the ASF allow-list permits without a SHA pin.Verification
Java 8 remains the build JDK —
./mvnw -pl mock-collector -am compilepasses under Zulu 8 locally (BUILD SUCCESS), and both CI jobs are green on this PR.