.NET: [Bug]: Hosted-agent runtime never activates OTLP export to a third-party destination, despite docs describing it as automatic

[cristofima]

Description

A .NET Foundry hosted agent (AgentHost.CreateBuilder + Microsoft.Agents.AI.Foundry.Hosting, zero manual OpenTelemetry instrumentation code, per the "zero-code telemetry" model this package advertises) never delivers OTLP data to a third-party destination via the documented OTEL_EXPORTER_OTLP_* environment variables, even though the exact same variables work correctly outside the hosted-agent runtime.

Per configure-hosted-agent-telemetry: "Hosted agents support the standard environment variables defined in the OpenTelemetry SDK configuration documentation... When OTEL_EXPORTER_OTLP_ENDPOINT is present, the runtime also exports over OTLP to that endpoint." This does not happen for .NET.

What happened:

1. Deployed a hosted agent with these env vars set on the agent version (via azure.yaml's env map, applied through azd deploy):
   • OTEL_EXPORTER_OTLP_ENDPOINT (generic) — tried both with and without this set, across 4 separate deployed agent versions.
   • OTEL_EXPORTER_OTLP_PROTOCOL=http/protobuf
   • OTEL_EXPORTER_OTLP_TRACES_ENDPOINT / OTEL_EXPORTER_OTLP_METRICS_ENDPOINT / OTEL_EXPORTER_OTLP_LOGS_ENDPOINT (per-signal, full paths)
   • OTEL_EXPORTER_OTLP_HEADERS / OTEL_EXPORTER_OTLP_TRACES_HEADERS (auth headers for the destination)
2. A GET on the deployed agent version (.../agents/{name}/versions/{n}?api-version=v1) confirmed every one of these variables was registered on the container exactly as configured, byte-for-byte — ruling out any client-side/deploy-tooling corruption.
3. Invoked the deployed agent. In every case:
   • Application Insights received the trace correctly (confirmed via a Log Analytics query — full span tree, gen_ai.* attributes present).
   • The third-party OTLP destination received nothing — no trace, no log, no metric — confirmed by querying that destination's own API directly for the exact trace ID.
   • The container's own startup log line from Azure.AI.AgentServer.Core.AgentHostBuilder — AgentServer connectivity: ProjectEndpoint=... OtlpEndpoint=(not set) AppInsightsConfigured=True — printed OtlpEndpoint=(not set) in every configuration tried, including the run where OTEL_EXPORTER_OTLP_ENDPOINT alone was set and independently confirmed registered server-side.
4. To rule out the destination itself, ran a minimal standalone .NET console app (Microsoft.Extensions.Hosting + OpenTelemetry.Extensions.Hosting, AddOpenTelemetry().WithTracing(t => t.AddSource(...)).UseOtlpExporter() — the identical env-var-driven mechanism the doc describes), outside the hosted-agent runtime entirely, using the exact same OTEL_EXPORTER_OTLP_PROTOCOL / OTEL_EXPORTER_OTLP_TRACES_ENDPOINT / OTEL_EXPORTER_OTLP_TRACES_HEADERS values already confirmed byte-correct in the deployed agent version. It delivered a real span to the third-party destination on the first try.

That isolation test rules out the destination, the API key, the header format, network reachability, and the .NET OpenTelemetry SDK's own OTLP exporter as causes. The defect is specific to the hosted-agent runtime's OpenTelemetry wiring: it never picks up any client-supplied third-party OTLP destination, while its separate Application Insights export path keeps working correctly throughout.

For context: an equivalent Python hosted agent, using the equivalent per-signal OTEL_EXPORTER_OTLP_* environment variables on the same underlying platform, does successfully deliver telemetry to the same third-party destination — so this appears to be specific to the .NET hosting stack, not a platform-wide limitation.

What I expected: Setting the documented OTEL_EXPORTER_OTLP_* environment variables on the agent version should activate OTLP export to that destination alongside Application Insights, with no code changes — as the platform doc describes.

Steps to reproduce:

1. Create a .NET hosted agent using AgentHost.CreateBuilder(args) + Microsoft.Agents.AI.Foundry.Hosting (see Code Sample — no manual OpenTelemetry instrumentation).
2. Deploy it to a Foundry project (azure.yaml + azd deploy) with these env vars on the agent service: OTEL_EXPORTER_OTLP_ENDPOINT, OTEL_EXPORTER_OTLP_PROTOCOL=http/protobuf, and/or the per-signal OTEL_EXPORTER_OTLP_{TRACES,METRICS,LOGS}_ENDPOINT / _HEADERS variants, pointing at any OTLP-compliant third-party endpoint (reproduced against Datadog's agentless OTLP intake, https://otlp.datadoghq.com).
3. Invoke the deployed agent.
4. Observe: Application Insights receives the trace; the third-party endpoint receives nothing; the container's startup log line reports OtlpEndpoint=(not set) regardless of what was configured.

Code Sample

The entire agent — no manual telemetry code exists to inspect or misconfigure:

using Microsoft.Agents.AI;
using Microsoft.Agents.AI.Foundry.Hosting;

var builder = AgentHost.CreateBuilder(args);

AIAgent agent = /* built via AIProjectClient(...).AsAIAgent(...) */;

builder.Services.AddFoundryResponses(agent);
builder.RegisterProtocol("responses", endpoints => endpoints.MapFoundryResponses());

var app = builder.Build();
await app.RunAsync();

azure.yaml's relevant env block on the azure.ai.agent service (destination/keys genericized):

env:
  OTEL_EXPORTER_OTLP_ENDPOINT: https://otlp.example-provider.com
  OTEL_EXPORTER_OTLP_PROTOCOL: http/protobuf
  OTEL_EXPORTER_OTLP_TRACES_ENDPOINT: https://otlp.example-provider.com/v1/traces
  OTEL_EXPORTER_OTLP_METRICS_ENDPOINT: https://otlp.example-provider.com/v1/metrics
  OTEL_EXPORTER_OTLP_LOGS_ENDPOINT: https://otlp.example-provider.com/v1/logs
  OTEL_EXPORTER_OTLP_HEADERS: api-key=<redacted>

The isolation repro that does work, run locally with the same env vars set in-process (not through the hosted-agent runtime):

using System.Diagnostics;
using Microsoft.Extensions.DependencyInjection;
using Microsoft.Extensions.Hosting;
using OpenTelemetry;
using OpenTelemetry.Trace;

const string sourceName = "OtlpIsolationTest";

var builder = Host.CreateApplicationBuilder(args);
builder.Services.AddOpenTelemetry()
    .WithTracing(tracing => tracing.AddSource(sourceName))
    .UseOtlpExporter();

var host = builder.Build();
await host.StartAsync();

using var activitySource = new ActivitySource(sourceName);
using (activitySource.StartActivity("isolated-test-span")) { }

host.Services.GetRequiredService<TracerProvider>().ForceFlush(10000);
await host.StopAsync();

Error Messages / Stack Traces

info: Azure.AI.AgentServer.Core.AgentHostBuilder[0]
      AgentServer connectivity: ProjectEndpoint=https://<account>.services.ai.azure.com OtlpEndpoint=(not set) AppInsightsConfigured=True

This line printed identically across every deployed agent version tested, regardless of which OTEL_EXPORTER_OTLP_* variables were set on that version (confirmed registered correctly server-side in every case via a GET on the agent version resource).

Package Versions

Microsoft.Agents.AI.Foundry.Hosting: 1.21.0-preview.260911.1 (pulls in Azure.AI.AgentServer.Core transitively)

.NET Version

.NET 10.0.11 (Linux container, dotnet_10 code-deploy runtime)

Additional Context

• Doc this behavior contradicts: Export hosted agent telemetry by using OpenTelemetry.
• This is not an Application Insights problem — that export path worked correctly in every single test in this investigation. It's specific to the third-party/OTLP path.
• This blocks any .NET hosted agent from sending telemetry to a non-Microsoft OpenTelemetry backend (Datadog, Honeycomb, a self-hosted Collector, etc.) via the documented, zero-code mechanism.
• Happy to share the full reproduction repo/branch privately if that's useful for triage.

[he-yufeng]

Boundary evidence from the open-source side, to help route this correctly: the Microsoft.Agents.AI.Foundry.Hosting package in this repo does not read OTEL_EXPORTER_OTLP_* anywhere (repo-wide code search hits only samples, docs and .env.example), and its only telemetry surface is ApplyOpenTelemetry, which just wraps the agent with OpenTelemetryAgent instrumentation (dotnet/src/Microsoft.Agents.AI.Foundry.Hosting/ServiceCollectionExtensions.cs:602-615). The comment on that method is explicit that exporter wiring is Azure.AI.AgentServer.Core's AddAgentHostTelemetry().

That matches the isolation in the report exactly: your standalone UseOtlpExporter() console app proves the SDK-level env-var path works, while the hosted runtime's startup line prints OtlpEndpoint=(not set) in every configuration. The exporter registration that would read those variables lives in Azure.AI.AgentServer.Core, which is not open source in this repository, so the defect is in that package's OTLP wiring rather than anything fixable in the Foundry.Hosting sources here. Flagging it for the team that owns AgentServer.Core seems like the right route; if any part of that package is mirrored elsewhere under the org I am happy to look at the wiring there.

[cristofima]

Turns out Azure.AI.AgentServer.Core is open source after all, in sdk/agentserver/Azure.AI.AgentServer.Core under Azure/azure-sdk-for-net.

Looked at the actual wiring: OpenTelemetryExtensions.AddAgentHostTelemetry just checks FoundryEnvironment.OtlpEndpoint and sets ExportTarget.Otlp accordingly, and FoundryEnvironment.OtlpEndpoint is a plain Environment.GetEnvironmentVariable("OTEL_EXPORTER_OTLP_ENDPOINT") read once at process start. There's no transformation and no validation that could silently swallow a value. The OtlpEndpoint=(not set) startup log line (AgentHostBuilder.LogStartupConfiguration) is just MaskUri() reporting that the same env var was empty or null in-process.

Given that, and given the az rest GET on the deployed agent version showed OTEL_EXPORTER_OTLP_ENDPOINT registered byte-for-byte in the agent version's configuration, the gap isn't in this package's code at all. It sits between that stored configuration and whatever actually injects env vars into the running container process, which is the hosted-agent container provisioning/orchestration layer rather than this repo. Might be worth routing this toward whoever owns that layer instead.