S3 HeadObject returns 403 when sentry_sdk.start_span and Datadog botocore instrumentation are both active

[andrey-berenda]

Environment

• Python 3.14
• Linux aarch64
• sentry-sdk==2.66.0
• ddtrace==4.12.1 (Datadog admission-controller library injection)
• FastAPI
• boto3/botocore with AWS SigV4

Description

An S3 HeadObject request against an existing, accessible object returns 403 Forbidden only when it executes inside sentry_sdk.start_span(...) while Datadog's botocore integration is active. The same pod identity, bucket, key, client configuration, and credentials return 200 OK when either the Sentry span is removed or Datadog's botocore integration is disabled.

This also reproduces through our real FastAPI endpoint. The example below is read-only and reproduces on its first request.

Minimal reproducer

import boto3
import sentry_sdk
from botocore.config import Config
from fastapi import FastAPI
from fastapi.testclient import TestClient
from sentry_sdk.integrations.fastapi import FastApiIntegration

sentry_sdk.init(
    dsn=None,
    traces_sample_rate=1.0,
    integrations=[FastApiIntegration()],
)

s3 = boto3.client(
    "s3",
    config=Config(
        signature_version="s3v4",
        s3={
            "addressing_style": "virtual",
            "us_east_1_regional_endpoint": "regional",
        },
    ),
)

app = FastAPI()

@app.post("/reproduce")
def reproduce():
    with sentry_sdk.start_span(op="dependency.resolve", name="FileStorage"):
        response = s3.head_object(
            Bucket="YOUR_BUCKET",
            Key="YOUR_EXISTING_KEY",
        )
    return {"status": response["ResponseMetadata"]["HTTPStatusCode"]}

response = TestClient(app).post("/reproduce")
print(response.status_code, response.text)

Run under Datadog auto-instrumentation, for example:

DD_TRACE_BOTOCORE_ENABLED=true ddtrace-run python reproducer.py

Actual result

botocore.exceptions.ClientError: An error occurred (403) when calling the HeadObject operation: Forbidden

Control matrix

Datadog botocore integration	Nested sentry_sdk.start_span	Result
enabled	present	403 Forbidden
enabled	absent	200 OK
disabled	present	200 OK

The confirmed workaround is:

DD_TRACE_BOTOCORE_ENABLED=false

A corresponding Datadog report exists at DataDog/dd-trace-py#19477. This appears to be an interoperability issue; we are reporting it to both projects because the trigger requires both instrumentations.

[linear-code]

PY-2667

[sentrivana]

Thanks @andrey-berenda, we'll take a look.

[jian-wu]

I had Claude code verify whether ddtrace 4.12.0 exposed this preexisting issue and it was confirmed.

Confirmed it — checked ddtrace/propagation/http.py directly. The _BaggageHeader class (lines ~1000-1109) extracts the inbound baggage header into Context._baggage verbatim, and on inject, serializes whatever's in span_context._baggage straight back out to a baggage header (line 1043: headers[_HTTP_HEADER_BAGGAGE] = header_value) — no filtering by vendor prefix, since W3C baggage is meant to be opaque/pass-through.

That closes the loop on your question whether it was a preexisting sentry issue. Here's the mechanism:

Yes — this was a preexisting Sentry issue that ddtrace 4.12.0 exposed, not a ddtrace-introduced bug in isolation. Sentry's config (unrestricted trace_propagation_targets) has presumably always been willing to inject a baggage header on outgoing AWS calls via its httplib patch. But that injection happens at the wire layer, strictly after SigV4 signing — so pre-4.12.0, it was harmless. AWS never validated it because it was never part of SignedHeaders.

4.12.0 is what changed: it added the first-ever pre-sign header injection point in this dependency chain (the before-sign botocore hook). That handler, using _BaggageHeader.inject, faithfully carries forward whatever's in the ddtrace span's _baggage dict — which itself was populated by extracting the inbound request's baggage header (the sentry-* keys your upstream service wrote, extracted generically since ddtrace treats baggage as opaque pass-through, not Sentry-specific). So for the first time, a baggage header — containing inherited Sentry-authored content — got baked into the pre-sign headers and thus into SignedHeaders.

Sentry's unrelated, unchanged, post-sign injection then appends its own freshly-serialized copy (with a newly computed sample_rand) on the same request. Two baggage: header instances hit the wire; AWS comma-joins same-named headers before validating, producing exactly the duplicated content you saw — which no longer matches what was actually signed.

So: latent Sentry misconfiguration (present indefinitely) + a genuinely new ddtrace 4.12.0 mechanism (first pre-sign injection point, incidentally forwarding inherited baggage) = signature-breaking duplication. Neither side alone explains it; the version bump is the trigger, Sentry's targets config is the enabling condition. Your Sentry trace_propagation_targets fix is the right primary remediation since it removes the enabling condition regardless of ddtrace version; a ddtrace cap is a reasonable secondary belt-and-suspenders.