Skip to content

benchmarking: expose raw atelet and ate metrics via Prometheus export… - #1601

Open
Nishanth Kotla (Nishanth29) wants to merge 1 commit into
agent-substrate:mainfrom
Nishanth29:benchmarking/telemetry-meter
Open

benchmarking: expose raw atelet and ate metrics via Prometheus export…#1601
Nishanth Kotla (Nishanth29) wants to merge 1 commit into
agent-substrate:mainfrom
Nishanth29:benchmarking/telemetry-meter

Conversation

@Nishanth29

Copy link
Copy Markdown

Fixes #1589

What this PR does

This PR routes incoming raw OTLP metrics from atelet and ateapi to the Prometheus pull exporter on port 8889 in benchmarking/telemetry/meter.yaml.

Problem

Currently, telemetry-meter only routes aggregated count metrics (substrate_spans and substrate_datapoints) to the Prometheus exporter on port 8889. Raw OTLP metrics (including snapshot sizes, restore durations, and worker pool states) are only forwarded upstream to managed cloud collectors.

In local or standalone benchmarking environments where cloud monitoring backends are not configured, in-cluster Prometheus cannot scrape these raw histograms and gauges.

Proposed Changes

In benchmarking/telemetry/meter.yaml:

  • Add a metrics/raw pipeline under service.pipelines routing incoming otlp metrics through the memory_limiter processor to the prometheus exporter on port 8889.
  • Leaves existing forwarding and counting pipelines intact.

How this was tested

  • Deployed on a benchmark cluster and ran the glutton user workload.

  • Verified that telemetry-meter:8889 exposes atelet_snapshot_size_bytes and ate_workerpool_workers.

  • Verified that in-cluster Prometheus scrapes these metrics via the default pod annotations without error.

  • Tests pass

  • Appropriate changes to documentation are included in the PR

@maxsmythe Max Smythe (maxsmythe) left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[benchmarking] Expose Prometheus pull metrics endpoint on telemetry-meter

2 participants