Give every span of an OTLP trace response a status - #14100
Merged
wankai123 merged 1 commit intoSep 22, 2026
Merged
Conversation
Grafana's Tempo datasource reads a span's status without checking that it is there (int64(span.Status.Code) in the backend's trace_transform.go), so a span served without one crashes its trace view with a nil pointer dereference and the request fails with 500. The field is optional in OTLP and senders do leave it out: APISIX's opentelemetry plugin sets a status only on an upstream 5xx, and until now such a span reached the OAP with the field absent whenever no OpenTelemetry Collector normalized it on the way. OTLPTraceAssembler.assemble already rebuilds every span while it regroups them, so it now fills in an empty status when the stored span has none. An absent status and an empty one both mean STATUS_CODE_UNSET, so nothing a client reads changes, and because this happens on read the traces already stored render too. The SkyWalking and Zipkin datasources set a status on every span they convert, so only the native OTLP path needed this.
wu-sheng
approved these changes
Sep 22, 2026
wankai123
added a commit
to apache/skywalking-showcase
that referenced
this pull request
Sep 22, 2026
…lector (#307) The OAP serves OTLP on its own ports, gRPC on 11800 and OTLP/HTTP on 12800, so the second copy of the music application no longer needs a collector to relay its spans. Each sender uses the transport its instrumentation has: the Java and Node.js agents speak gRPC, while the Python auto-instrumentation image ships no gRPC exporter and the APISIX plugin has no gRPC client at all, so those two use OTLP/HTTP. On Kubernetes the collector keeps only the Prometheus pipelines the monitoring features need, and feature-otlp-trace no longer turns it on; on Docker Compose the copy's own collector is gone and the OAP joins the copy's network instead, which keeps the two copies isolated because their services answer to the same hostnames. The Zipkin store becomes the mesh's alone, fed by Istio's sidecars, so /zipkin shows the mesh and /otlp this copy rather than both showing the same spans under two sets of service names. The Docker layout runs no mesh, so its Zipkin store stays empty; the receiver and the query stay on there only so the provisioned ZipkinTraceQL datasource and its dashboard answer instead of 404. The OAP image moves to apache/skywalking#14100, which this design needs: a collector used to mask the fact that APISIX sends successful spans without a status, and Grafana's trace view crashes on such a span until the OAP fills the field in.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fix the
/otlpTraceQL trace view failing in Grafana withruntime error: invalid memory address or nil pointer dereferenceWhy the bug exists.
statusis an optional field of an OTLP span, and senders do leave it out. APISIX'sopentelemetryplugin, for one, sets a status only when the upstream answered 5xx:Grafana's Tempo datasource reads the field without checking that it is there, in
pkg/tempo/trace_transform.go:So a trace containing one such span crashes Grafana's trace view: the plugin panics,
/api/ds/queryanswers 500, and the user sees "Internal Server Error - please inspect Grafana server log for details". The panic stack runstempo.go→trace.go→trace_transform.go, the trace-by-id path only; search is unaffected because its response carries no span status. The code is unguarded in Grafana 12.4.2, in the latest 12.4 patch, and in the extractedgrafana/grafana-tempo-datasourcethat 13.x uses, so upgrading Grafana does not avoid it.This surfaced now because an OpenTelemetry Collector used to hide it. Its internal model holds the status as a value rather than a pointer, so every span it re-serialized carried at least an empty status. A span stored while a collector was in the path reads back as
status={}; the same span sent straight to the OAP has the field absent.The fix.
OTLPTraceAssembler.assemblealready rebuilds every span while it regroups them by resource and scope, so it now fills in an empty status when the stored span has none. An absent status and an empty one both meanSTATUS_CODE_UNSET, so nothing an API client reads changes. Because this happens on read, traces already in storage render too. The SkyWalking and Zipkin datasources set a status on every span they convert, so only the native OTLP path needed the guard.Tests. Two new cases in
OTLPTraceAssemblerTest: a span stored without a status comes back withSTATUS_CODE_UNSETand the field present in the JSON encoding as well, and a span stored with an error status keeps it untouched. One assertion inshouldRegroupByResourceThenScopeAndOrderSpansByStartcompared an assembled span byte for byte against its stored form and now expects the filled-in status; the whole module passes at 58 tests.Verified on a Kubernetes deployment of the showcase whose OpenTelemetry-instrumented services export straight to the OAP. A trace whose APISIX span had no status went from
/api/ds/queryreturning 500 with a panic in Grafana's log, to 200 with all 27 span rows and no panic, without re-ingesting anything.CHANGESlog.