Skip to content

[FLINK-38961][metrics] Display process metrics (CPU, Memory, I/O) on TaskManager Web UI - #28564

Draft
spuru9 wants to merge 1 commit into
apache:masterfrom
spuru9:FLINK-38961-process-metrics-webui
Draft

spuru9 wants to merge 1 commit into
apache:masterfrom
spuru9:FLINK-38961-process-metrics-webui

Conversation

@spuru9

@spuru9 spuru9 commented Jun 28, 2026

Copy link
Copy Markdown
Contributor

What is the purpose of the change

This pull request surfaces the TaskManager process's own resource usage (CPU, memory and disk I/O) as Flink metrics and displays them on the TaskManager Web UI.

When metrics.system-resource is enabled, SystemResourcesCounter now additionally collects the TaskManager process's CPU usage, resident set size and disk bytes read/written via OSHI, exposing them as the Process.CPU.Usage, Process.Memory.RSS, Process.IO.Read and Process.IO.Write gauges. A new "Process Usage" panel on the TaskManager metrics page renders these values.

Brief change log

  • SystemResourcesCounter collects per-process CPU usage, RSS and disk read/written bytes via OSHI
  • SystemResourcesMetricsInitializer registers the new Process.* gauges
  • Added a "Process Usage" panel to the TaskManager metrics page in the Web UI, querying the new gauges over the REST API
  • Documented the new metrics in docs/content/docs/ops/metrics.md and the zh translation

Verifying this change

This change added unit-test coverage in SystemResourcesCounterTest and can additionally be verified as follows:

  • Start a cluster with metrics.system-resource: true, open the TaskManager metrics page and confirm the new "Process Usage" panel shows CPU, memory (RSS) and combined I/O.

Note: per-process I/O is only populated on platforms where OSHI supports it (e.g. Linux via /proc); on macOS these counters report 0.

Does this pull request potentially affect one of the following parts:

  • Dependencies (does it add or upgrade a dependency): no (OSHI is already a Flink dependency)
  • The public API, i.e., is any changed class annotated with @Public(Evolving): no
  • The serializers: no
  • The runtime per-record code paths (performance sensitive): no
  • Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Kubernetes/Yarn, ZooKeeper: no
  • The S3 file system connector: no

Documentation

  • Does this pull request introduce a new feature? yes
  • If yes, how is the feature documented? docs (metrics.md, en + zh)

Was generative AI tooling used to co-author this PR?
  • Yes (Claude Code)

Generated-by: Claude Code

@flinkbot

flinkbot commented Jun 28, 2026 •

Copy link
Copy Markdown
Collaborator

CI report:

Bot commands The @flinkbot bot supports the following commands:
  • @flinkbot run azure re-run the last Azure build

@github-actions

Copy link
Copy Markdown

This PR is being marked as stale since it has not had any activity in the last 90 days.
If you would like to keep this PR alive, please leave a comment asking for a review.
If the PR has merge conflicts, update it with the latest from the base branch.

If you are having difficulty finding a reviewer, please reach out to the
community, contact details can be found here: https://flink.apache.org/what-is-flink/community/

If this PR is no longer valid or desired, please feel free to close it.
If no activity occurs in the next 30 days, it will be automatically closed.

@github-actions github-actions Bot added the stale label Sep 27, 2026
@spuru9
spuru9 force-pushed the FLINK-38961-process-metrics-webui branch from 9bfc999 to acd71b9 Compare October 1, 2026 20:50
…TaskManager Web UI

When metrics.system-resource is enabled, SystemResourcesCounter now also
collects the TaskManager process's CPU usage, resident set size and disk
bytes read/written via OSHI, exposed as the Process.CPU.Usage,
Process.Memory.RSS and Process.IO.Read / Process.IO.Write gauges.

A new "Process Usage" panel on the TaskManager metrics page displays these
values (CPU, memory RSS and combined I/O), querying them over the REST API.

Per-process I/O is only populated on platforms where OSHI supports it (e.g.
Linux via /proc); on macOS these counters report 0.
@github-actions github-actions Bot removed the stale label Oct 2, 2026
@spuru9
spuru9 force-pushed the FLINK-38961-process-metrics-webui branch from acd71b9 to da80be4 Compare October 2, 2026 07:07
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants