Skip to content

fix: segments/logs lost when two processes first report for the same service concurrently - #65

Merged
wu-sheng merged 2 commits into
masterfrom
fix/concurrent-first-insert
Aug 1, 2026
Merged

fix: segments/logs lost when two processes first report for the same service concurrently#65
wu-sheng merged 2 commits into
masterfrom
fix/concurrent-first-insert

Conversation

@wu-sheng

@wu-sheng wu-sheng commented Aug 1, 2026

Copy link
Copy Markdown
Member

Problem

SegmentItems.addSegmentItem and LogItems.addLogItem do a check-then-act on the map:

SegmentItem segmentItem = segmentItems.get(serviceName);
if (segmentItem == null) {
    segmentItem = new SegmentItem(serviceName);
    segmentItems.put(serviceName, segmentItem);   // not atomic with the get
}
segmentItem.addSegments(segment);

When two agent processes report their first data for the same service name concurrently, both handler threads see null, both create an item, and the second put overwrites the first — the data appended to the discarded item is silently lost while both reporters receive HTTP 200.
Only the first insert per service name is affected: SegmentItem/LogItem hold a CopyOnWriteArrayList, so appends are safe once the key exists. The map is a ConcurrentHashMap, so the intent was clearly thread-safety — the get/put pair just defeats it. Every receiver funnels into these methods (gRPC and both HTTP servlets), so the loss is transport-independent.

Impact

Found while debugging a flaky test in apache/skywalking-python#409: a plugin test forks a worker, so parent and child share one service name and post their first segments milliseconds apart. A segment disappeared in roughly 2 of 5 CI runs — sometimes the parent's, sometimes the child's — which looked like an agent bug for quite a while.
Reproduced directly against this collector image, from inside its network:

scenario result
200 rounds of two barrier-synchronized first-time POSTs per fresh service name 15/200 rounds silently lost a segment (segmentSize=1)
same 200 rounds, key pre-seeded by one sequential POST first 0/200 lost
Tests where each process uses a distinct service name (the common case) are unaffected, which is why this stayed hidden.

Fix

Use computeIfAbsent, which performs the get-or-create atomically — the pattern MeterItems.addMeter already uses in this same package:

final SegmentItem segmentItem = segmentItems.computeIfAbsent(serviceName, SegmentItem::new);
segmentItem.addSegments(segment);

Once a new image is published, the workaround in apache/skywalking-python#409 (seeding the service name with a warm-up request before the concurrent pair) can be removed.

Also: repair the macOS CI job

CI-on-MacOS has been failing since GitHub moved macos-latest to arm64 — unrelated to this change, but it blocks every PR. With actions/setup-java@v1, JAVA_HOME points at a hosted-toolcache path lacking the macOS Contents/Home layout, so $JAVA_HOME/bin/java does not exist and mvnw aborts with JAVA_HOME is not defined correctly before compiling anything (the log hands out an x64 JDK 8 path on an arm64 runner). setup-java@v1 also runs on a Node runtime GitHub has already removed.
Both jobs move to actions/setup-java@v4 and actions/checkout@v4. The distribution must be zulu, not temurin: JDK 8 has no macOS aarch64 temurin build, while zulu publishes one. Both actions are in the actions/* namespace, which the ASF allow-list permits without a SHA pin.

Verification

Java 8 remains the build JDK — ./mvnw -pl mock-collector -am compile passes under Zulu 8 locally (BUILD SUCCESS), and both CI jobs are green on this PR.

… first time

addSegmentItem and addLogItem look the service name up and then put it
back as two separate operations. When two agent processes report their
FIRST data for the same service name concurrently, both threads observe
null, both construct an item, and the second put overwrites the first —
the segments already appended to the discarded item are lost while both
reporters receive HTTP 200. Once the key exists there is no loss, since
SegmentItem/LogItem hold CopyOnWriteArrayList, so only the very first
insert per service name is affected.

This surfaced in apache/skywalking-python#409, where a plugin test forks
a worker: parent and child share one service name and post their first
segments milliseconds apart, so a segment vanished in roughly 2 of 5 CI
runs. Reproduced directly against this collector with barrier-
synchronized first-time POSTs: 15 of 200 rounds lost a segment, and 0 of
200 once the key had been seeded.

Use computeIfAbsent, which performs the whole get-or-create atomically —
the pattern MeterItems.addMeter already uses in this package.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
wu-sheng added a commit to apache/skywalking-python that referenced this pull request Aug 1, 2026
…aio compatibility, websockets >= 13 support (#409)

fix: never create a gRPC channel in the Gunicorn prefork master (#409)

With grpcio >= 1.80 (EventEngine) a gRPC channel that lives across fork()
breaks: gRPC's at-fork handlers self-skip while its own threads are inside
gRPC, leaving stale poll-engine eventfds behind. Under `sw-python run -p
gunicorn` the agent started fully in the master, so every worker inherited
such a channel — producing the continuous `Kick Failure (eventfd_write:
Bad file descriptor)` spam and, less visibly, a racy deadlock in which a
forked worker never boots and Gunicorn never notices. grpcio 1.83.0 removed
the last legacy-poller opt-out (grpc/grpc#42828); upstream grpc/grpc#43055
and grpc/grpc#43062 are still open. The failure is entirely client-side and
unrelated to the OAP version.

The master now only installs instrumentation and arms the fork hooks; the
queues, reporter threads and gRPC channel are created in each forked worker,
which is gRPC's supported model and what the uWSGI path has always done. The
master therefore no longer registers as a service instance. A new
agent.started() guard turns instrumented code into a no-op wherever the
reporters are inactive, keeping `gunicorn --preload` app imports safe in the
master. GRPC_POLL_STRATEGY is no longer set, the grpcio floor moves to
>= 1.83 (the generated stubs already require it at import) and codegen
grpcio-tools is pinned in lockstep. Gunicorn prefork combined with
SW_AGENT_ASYNCIO_ENHANCEMENT is now rejected instead of silently starting an
unsafe pre-fork agent, and explicit os.fork() over gRPC is documented — and
warned about at runtime — as unreliable, directing those users to the HTTP or
Kafka reporter.

Also repairs two unrelated cases that had been failing on master CI:

- sw_grpc: grpcio 1.83.0 validates `isinstance(x, Channel)` against the
  grpc.aio._channel module global, which the plugin had rebound to a factory
  function, so every aio stub creation raised TypeError. It is a Channel
  subclass now; the supported range stays grpcio 1.*.
- sw_websockets: uvicorn 0.50.0 dropped its legacy websockets fallback and
  unconditionally imports websockets.server.ServerProtocol (websockets >= 11
  only), crashing the test provider. The harness pins uvicorn < 0.50;
  websockets 10.3/10.4 remain tested.

The websockets plugin additionally instruments the new websockets.asyncio
client (websockets >= 13, the default since 14) alongside the legacy one —
applications on the modern API previously produced no spans at all. The
support matrix gains 13.1 everywhere and 17.0.1 on Python >= 3.11.

New plugin tests cover both fork paths end to end: sw_gunicorn validates a
complete consumer -> prefork-provider trace under `--preload`, asserting
exact worker boot counts, instrumentation-only behaviour in the master and
the absence of the gRPC fork errors; sw_fork_support validates a continuous
trace across an explicit os.fork() over the HTTP reporter. The latter seeds
its service name first, working around a first-insert race in the mock
collector (fixed by apache/skywalking-agent-test-tool#65).

Fixes apache/skywalking#13958
CI-on-MacOS has failed since GitHub moved macos-latest to arm64: with
actions/setup-java@v1, JAVA_HOME points at a hosted-toolcache path that
lacks the macOS Contents/Home layout, so $JAVA_HOME/bin/java does not
exist and mvnw aborts with "JAVA_HOME is not defined correctly" before
compiling anything. setup-java@v1 also runs on a Node runtime GitHub has
already removed.

Move both jobs to actions/setup-java@v4 (and checkout@v4). The
distribution must be zulu rather than the more common temurin: JDK 8 has
no macOS aarch64 temurin build, while zulu publishes one. Both actions
are in the actions/* namespace, which the ASF allow-list permits without
a SHA pin.

Java 8 remains the build JDK; verified locally with `./mvnw -pl
mock-collector -am compile` under Zulu 8 (BUILD SUCCESS).

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@wu-sheng wu-sheng added the bug Something isn't working label Aug 1, 2026
@wu-sheng
wu-sheng merged commit d49fd41 into master Aug 1, 2026
2 checks passed
@wu-sheng
wu-sheng deleted the fix/concurrent-first-insert branch August 1, 2026 14:19
wu-sheng added a commit that referenced this pull request Aug 1, 2026
…66)

The workflow referenced docker/login-action, docker/setup-qemu-action and
docker/setup-buildx-action by floating tag. The ASF GitHub Actions
allow-list only approves specific commit SHAs for third-party actions, so
the workflow is rejected at startup: the run for #65 ended in
startup_failure and no image was published for that commit.

Since publish-docker only runs on push-to-master, the rejection never
surfaces in PR CI — it is only visible after a merge.

Pin all three to the approved SHAs already used across the sibling ASF
SkyWalking repositories (verified against apache/infrastructure-actions
approved_patterns.yml):
  - docker/login-action@650006c6...       # v4.2.0
  - docker/setup-qemu-action@06116385...  # v4.1.0
  - docker/setup-buildx-action@d7f5e7f5... # v4.1.0

Merging this triggers a push-to-master build, which publishes an image
containing the #65 concurrency fix.
wu-sheng added a commit to apache/skywalking-python that referenced this pull request Aug 1, 2026
The pinned mock-collector image dated from October 2022, 13 commits behind
the tool's master. Move it to c6b91a0e, which carries the fix for the
first-insert race in SegmentItems/LogItems
(apache/skywalking-agent-test-tool#65): when two processes reported their
first segment for the same service name concurrently, the collector
silently dropped one while both reporters got HTTP 200.

sw_fork_support worked around that race by seeding the service name with a
parent-only /ping request and waiting for the collector to register it
before triggering the concurrent parent/child reports. The collector no
longer needs the help, so the endpoint, the seed step and the extra
expected segment are gone and the test is back to asserting exactly the
cross-fork trace it is about.

The upgrade also picks up the validator changes made since 2022, one of
which affected us. LogAssert now sorts both the expected and the actual
logs by their body text before comparing them pairwise
(apache/skywalking-agent-test-tool#59), and the sort uses the raw expected
string, so a matcher such as `text: not null` participates as the literal
"not null". sw_loguru matches its two logging-module records that way
while the records themselves led with the default layout's timestamp, so
they sorted first and their placeholders last, inverting the pairing.
Pin a layout that leads with the logger name, which sorts after the
placeholders and orders the two records deterministically, and reorder the
expected entries to match; SWFormatter is still exercised. Its
expected.data.yml is the only one in the tree with a non-empty logItems
block, so no other test is affected.

Also refresh CLAUDE.md: supported Python and grpcio floor, the current
plugin list, the agent's fork/prefork lifecycle, and the plugin-test
validation notes.
wu-sheng added a commit to apache/skywalking-python that referenced this pull request Aug 2, 2026
…410)

The pinned mock-collector image dated from October 2022, 13 commits behind
the tool's master. Move it to c6b91a0e, which carries the fix for the
first-insert race in SegmentItems/LogItems
(apache/skywalking-agent-test-tool#65): when two processes reported their
first segment for the same service name concurrently, the collector
silently dropped one while both reporters got HTTP 200.

sw_fork_support worked around that race by seeding the service name with a
parent-only /ping request and waiting for the collector to register it
before triggering the concurrent parent/child reports. The collector no
longer needs the help, so the endpoint, the seed step and the extra
expected segment are gone and the test is back to asserting exactly the
cross-fork trace it is about.

The upgrade also picks up the validator changes made since 2022, one of
which affected us. LogAssert now sorts both the expected and the actual
logs by their body text before comparing them pairwise
(apache/skywalking-agent-test-tool#59), and the sort uses the raw expected
string, so a matcher such as `text: not null` participates as the literal
"not null". sw_loguru matches its two logging-module records that way
while the records themselves led with the default layout's timestamp, so
they sorted first and their placeholders last, inverting the pairing.
Pin a layout that leads with the logger name, which sorts after the
placeholders and orders the two records deterministically, and reorder the
expected entries to match; SWFormatter is still exercised. Its
expected.data.yml is the only one in the tree with a non-empty logItems
block, so no other test is affected.

Also refresh CLAUDE.md: supported Python and grpcio floor, the current
plugin list, the agent's fork/prefork lifecycle, and the plugin-test
validation notes.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

bug Something isn't working

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants