Skip to content

[feat][fn] Expose the Prometheus metrics registry to Go functions via FunctionContext - #26458

Open
kritharth2005 wants to merge 1 commit into
apache:masterfrom
kritharth2005:feat/fn-expose-prometheus-registry
Open

[feat][fn] Expose the Prometheus metrics registry to Go functions via FunctionContext#26458
kritharth2005 wants to merge 1 commit into
apache:masterfrom
kritharth2005:feat/fn-expose-prometheus-registry

Conversation

@kritharth2005

Copy link
Copy Markdown

Fixes #26403

Motivation

The Go Functions SDK gives user code exactly one way to emit a custom metric:
FunctionContext.RecordMetric(name, value), which funnels every value into a
single fixed-shape SummaryVec (pulsar_function_user_metric, with quantile
objectives 0.5/0.9/0.99/0.999 baked in at pulsar-function-go/pf/stats.go). A
Go function cannot register a plain counter, a gauge, a histogram with its own
buckets, or any other prometheus.Collector.

The SDK already runs a Prometheus registry (reg, an unexported
*prometheus.Registry in pf/stats.go) and serves it over the function's
metrics port via NewMetricsServicer. There is currently no exported accessor
for it, so user code outside package pf has no way to reach it.

Python functions get this for free: the Python SDK relies on
prometheus_client's process-global registry, so a Python function can just
instantiate a prometheus_client.Counter(...) and have it scraped on the same
endpoint. This PR brings the Go SDK to parity by exposing the registry it
already maintains.

Modifications

  • pulsar-function-go/pf/context.go: added
    func (c *FunctionContext) GetMetricsRegistry() prometheus.Registerer,
    which returns the package-level reg. Placed immediately after
    RecordMetric (and next to GetMetricsPort) so the metrics-related
    methods stay grouped. The return type is the prometheus.Registerer
    interface rather than the concrete *prometheus.Registry: callers get
    Register / MustRegister / Unregister but not Gather, keeping the
    scrape/collection path internal to the SDK. The doc comment states that
    pulsar_function_-prefixed names are reserved for SDK metrics and that a
    colliding fully-qualified name fails registration (Register returns an
    error, MustRegister panics).

    RecordMetric and the userMetricSummary definition are left unchanged.

  • pulsar-function-go/pf/stats_test.go: added two tests next to the
    existing TestMetricsServer / TestUserMetrics / TestInstanceControlMetrics:

    • TestGetMetricsRegistry_CustomCollector — starts a goInstance and its
      MetricsServicer, registers a prometheus.Counter (a shape the
      user_metric Summary cannot express) through
      gi.context.GetMetricsRegistry().Register(...), adds 42, scrapes
      /metrics on GetMetricsPort(), and asserts the response body contains
      pulsar_function_go_test_custom_counter_total 42. The collector is
      unregistered on cleanup so the process-global reg is not left mutated.
    • TestGetMetricsRegistry_NameCollision — registers a counter named
      pulsar_function_received_total (colliding with a built-in SDK metric)
      and asserts Register() returns a non-nil error, then asserts
      MustRegister() panics on the same collision.

Design note (open to review) — prefix-collision handling

The issue thread raised whether the SDK should actively reject user
collectors that use the reserved pulsar_function_ prefix (e.g. a wrapper
type around the registry). This PR intentionally does not do that: it
documents the reserved prefix in the method's doc comment and relies on
prometheus.Registerer.Register already returning a non-panicking error
when a fully-qualified name collides with an existing metric — which is the
reason the method returns the Registerer interface. This keeps the
surface minimal and matches how the underlying library behaves. If
reviewers would prefer a stricter guard, that can be added.

Out of scope

Option B from the issue — typed convenience helpers such as NewUserCounter
/ NewUserGauge — is intentionally not included here. Exposing the registry
is the smaller, lower-risk change; the typed helpers can follow in a separate
PR once this lands.

Verifying this change

  • Make sure that the change passes the CI checks.

This change added tests and can be verified as follows:

  • TestGetMetricsRegistry_CustomCollector registers a custom counter and
    confirms it is exposed on the function's /metrics endpoint.
  • TestGetMetricsRegistry_NameCollision confirms a name colliding with a
    built-in SDK metric is rejected by Register() (error, no panic) and
    panics under MustRegister().

Does this pull request potentially affect one of the following parts:

If the box was checked, please highlight the changes

  • Dependencies (add or upgrade a dependency)
  • The public API — adds FunctionContext.GetMetricsRegistry() to the Go
    Functions SDK. Additive only; no existing signatures change.
  • The schema
  • The default values of configurations
  • The threading model
  • The binary protocol
  • The REST endpoints
  • The admin CLI options
  • The metrics — Go functions can now register arbitrary Prometheus
    collectors, which are scraped on the existing function metrics endpoint.
  • Anything that affects deployment

… FunctionContext

Go functions have exactly one way to emit a custom metric:
FunctionContext.RecordMetric(name, value), which funnels every value into a
single fixed-shape SummaryVec (pulsar_function_user_metric). There's no way
to register a Counter, Gauge, Histogram, or any other prometheus.Collector,
and the registry the SDK already runs and serves on the metrics port has no
exported accessor.

This adds FunctionContext.GetMetricsRegistry() prometheus.Registerer,
returning the existing package-level registry typed as the Registerer
interface (Register/MustRegister/Unregister, not Gather), so a function can
register its own collectors alongside the SDK's. RecordMetric and
userMetricSummary are unchanged; typed convenience helpers (Option B from
the issue) are left as a follow-on.

Covered by two new tests in stats_test.go: registering and scraping a
custom Counter, and the collision behavior on a name colliding with a
built-in metric (Register errors, MustRegister panics).

Fixes apache#26403
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Go Functions] No way to register custom metric collectors: every user metric is a summary

1 participant