Skip to content

Give every span of an OTLP trace response a status - #14100

Merged
wankai123 merged 1 commit into
apache:masterfrom
wankai123:fix/otlp-trace-view-span-status
Sep 22, 2026
Merged

wankai123 merged 1 commit into
apache:masterfrom
wankai123:fix/otlp-trace-view-span-status

Conversation

@wankai123

Copy link
Copy Markdown
Member

Fix the /otlp TraceQL trace view failing in Grafana with runtime error: invalid memory address or nil pointer dereference

  • Add a unit test to verify that the fix works.
  • Explain briefly why the bug exists and how to fix it.

Why the bug exists. status is an optional field of an OTLP span, and senders do leave it out. APISIX's opentelemetry plugin, for one, sets a status only when the upstream answered 5xx:

if upstream_status and upstream_status >= 500 then
    span:set_status(span_status.error, "upstream response status: " .. upstream_status)

Grafana's Tempo datasource reads the field without checking that it is there, in pkg/tempo/trace_transform.go:

status := span.Status
statusCode := int64(status.Code)

So a trace containing one such span crashes Grafana's trace view: the plugin panics, /api/ds/query answers 500, and the user sees "Internal Server Error - please inspect Grafana server log for details". The panic stack runs tempo.gotrace.gotrace_transform.go, the trace-by-id path only; search is unaffected because its response carries no span status. The code is unguarded in Grafana 12.4.2, in the latest 12.4 patch, and in the extracted grafana/grafana-tempo-datasource that 13.x uses, so upgrading Grafana does not avoid it.

This surfaced now because an OpenTelemetry Collector used to hide it. Its internal model holds the status as a value rather than a pointer, so every span it re-serialized carried at least an empty status. A span stored while a collector was in the path reads back as status={}; the same span sent straight to the OAP has the field absent.

The fix. OTLPTraceAssembler.assemble already rebuilds every span while it regroups them by resource and scope, so it now fills in an empty status when the stored span has none. An absent status and an empty one both mean STATUS_CODE_UNSET, so nothing an API client reads changes. Because this happens on read, traces already in storage render too. The SkyWalking and Zipkin datasources set a status on every span they convert, so only the native OTLP path needed the guard.

Tests. Two new cases in OTLPTraceAssemblerTest: a span stored without a status comes back with STATUS_CODE_UNSET and the field present in the JSON encoding as well, and a span stored with an error status keeps it untouched. One assertion in shouldRegroupByResourceThenScopeAndOrderSpansByStart compared an assembled span byte for byte against its stored form and now expects the filled-in status; the whole module passes at 58 tests.

Verified on a Kubernetes deployment of the showcase whose OpenTelemetry-instrumented services export straight to the OAP. A trace whose APISIX span had no status went from /api/ds/query returning 500 with a panic in Grafana's log, to 200 with all 27 span rows and no panic, without re-ingesting anything.

  • If this pull request closes/resolves/fixes an existing issue, replace the issue number. No issue is filed; found while running the showcase without a collector in the trace path.
  • Update the CHANGES log.

Grafana's Tempo datasource reads a span's status without checking that it is there
(int64(span.Status.Code) in the backend's trace_transform.go), so a span served without
one crashes its trace view with a nil pointer dereference and the request fails with 500.
The field is optional in OTLP and senders do leave it out: APISIX's opentelemetry plugin
sets a status only on an upstream 5xx, and until now such a span reached the OAP with the
field absent whenever no OpenTelemetry Collector normalized it on the way.

OTLPTraceAssembler.assemble already rebuilds every span while it regroups them, so it now
fills in an empty status when the stored span has none. An absent status and an empty one
both mean STATUS_CODE_UNSET, so nothing a client reads changes, and because this happens
on read the traces already stored render too. The SkyWalking and Zipkin datasources set a
status on every span they convert, so only the native OTLP path needed this.
@wankai123 wankai123 added the bug Something isn't working and you are sure it's a bug! label Sep 22, 2026
@wankai123 wankai123 added this to the 11.1.0 milestone Sep 22, 2026
@wankai123
wankai123 merged commit 03ecf41 into apache:master Sep 22, 2026
475 of 478 checks passed
@wankai123
wankai123 deleted the fix/otlp-trace-view-span-status branch September 22, 2026 03:12
wankai123 added a commit to apache/skywalking-showcase that referenced this pull request Sep 22, 2026
…lector (#307)

The OAP serves OTLP on its own ports, gRPC on 11800 and OTLP/HTTP on 12800, so the second
copy of the music application no longer needs a collector to relay its spans. Each sender
uses the transport its instrumentation has: the Java and Node.js agents speak gRPC, while
the Python auto-instrumentation image ships no gRPC exporter and the APISIX plugin has no
gRPC client at all, so those two use OTLP/HTTP. On Kubernetes the collector keeps only the
Prometheus pipelines the monitoring features need, and feature-otlp-trace no longer turns
it on; on Docker Compose the copy's own collector is gone and the OAP joins the copy's
network instead, which keeps the two copies isolated because their services answer to the
same hostnames.

The Zipkin store becomes the mesh's alone, fed by Istio's sidecars, so /zipkin shows the
mesh and /otlp this copy rather than both showing the same spans under two sets of service
names. The Docker layout runs no mesh, so its Zipkin store stays empty; the receiver and
the query stay on there only so the provisioned ZipkinTraceQL datasource and its dashboard
answer instead of 404.

The OAP image moves to apache/skywalking#14100, which this design needs: a collector used
to mask the fact that APISIX sends successful spans without a status, and Grafana's trace
view crashes on such a span until the OAP fills the field in.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

bug Something isn't working and you are sure it's a bug!

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants