Skip to content

collector: add tainted collector for /proc/sys/kernel/tainted - #3745

Open
arunsrini3082 wants to merge 5 commits into
prometheus:masterfrom
arunsrini3082:kernel-tainted
Open

collector: add tainted collector for /proc/sys/kernel/tainted#3745
arunsrini3082 wants to merge 5 commits into
prometheus:masterfrom
arunsrini3082:kernel-tainted

Conversation

@arunsrini3082

Copy link
Copy Markdown
Contributor

Summary

Add a tainted collector that exposes kernel taint flags from
/proc/sys/kernel/tainted as Prometheus metrics.

This revives the idea originally proposed in #1567. The review feedback
from that PR has been addressed:

Metrics

node_kernel_tainted{bit="14", flag="L"} 0
node_kernel_tainted{bit="12", flag="O"} 1

One series per known taint bit (20 total), value 0 or 1.
Disabled by default — enable with --collector.tainted.

Dependencies

⚠️ This PR depends on prometheus/procfs#844 being merged first.
The go.mod procfs version will be bumped before merging once
that PR is released.

Reference

https://www.kernel.org/doc/html/latest/admin-guide/tainted-kernels.html

@arunsrini3082

Copy link
Copy Markdown
Contributor Author

Both procfs PRs are now merged:

#844 — KernelTainted() implementation
#847 — fix linux build tag (was !windows, caused crossbuild failures on OpenBSD/NetBSD/DragonFly/Solaris)

No new procfs release has been cut yet, so I've temporarily pinned go.mod to the merged commit pseudo-version to unblock CI. Will update to the proper release tag once available.

@arunsrini3082

arunsrini3082 commented Jul 28, 2026

Copy link
Copy Markdown
Contributor Author

Built and tested in our environment. All 20 taint flags are correctly exposed:

vasudeva.sk:~/kernel_taints$ curl -s 0:9100/metrics | grep node_kernel_tainted
# HELP node_kernel_tainted Taint flags set on the running Linux kernel, as reported by /proc/sys/kernel/tainted. Value is 1 if the flag is set, 0 otherwise. See https://www.kernel.org/doc/html/latest/admin-guide/tainted-kernels.html for flag meanings.
# TYPE node_kernel_tainted gauge
node_kernel_tainted{bit="0",flag="P"} 0
node_kernel_tainted{bit="1",flag="F"} 0
node_kernel_tainted{bit="2",flag="S"} 0
node_kernel_tainted{bit="3",flag="R"} 0
node_kernel_tainted{bit="4",flag="M"} 0
node_kernel_tainted{bit="5",flag="B"} 0
node_kernel_tainted{bit="6",flag="U"} 0
node_kernel_tainted{bit="7",flag="D"} 0
node_kernel_tainted{bit="8",flag="A"} 0
node_kernel_tainted{bit="9",flag="W"} 0
node_kernel_tainted{bit="10",flag="C"} 0
node_kernel_tainted{bit="11",flag="I"} 0
node_kernel_tainted{bit="12",flag="O"} 0
node_kernel_tainted{bit="13",flag="E"} 0
node_kernel_tainted{bit="14",flag="L"} 0
node_kernel_tainted{bit="15",flag="K"} 0
node_kernel_tainted{bit="16",flag="X"} 0
node_kernel_tainted{bit="17",flag="T"} 0
node_kernel_tainted{bit="18",flag="N"} 0
node_kernel_tainted{bit="19",flag="J"} 0

This is on a clean kernel (all flags 0). The collector is working as expected.

Comment thread collector/tainted_linux.go Outdated
Comment thread collector/tainted_linux_test.go Outdated
Comment thread go.mod
github.com/prometheus/common v0.70.0
github.com/prometheus/exporter-toolkit v0.17.1
github.com/prometheus/procfs v0.21.1
github.com/prometheus/procfs v0.21.2-0.20260717070424-0cd18237af6e

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

blocking: this pins procfs to an untagged pseudo-version (v0.21.2-0.20260717070424-0cd18237af6e). Rebase onto master and point to a tagged release once one includes #844/#847. Can't merge on an untagged commit.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Waiting on procfs release for #844/#847; will update go.mod once tagged.

@arunsrini3082

Copy link
Copy Markdown
Contributor Author

Blocked on a tagged procfs release for KernelTainted() (#844) and the linux build-tag fix (#847).

Tracking: prometheus/procfs#859

Will update go.mod to the tagged release once published.

@nicolastakashi

Copy link
Copy Markdown
Contributor

@arunsrini3082 can you check failing ci?

@arunsrini3082

Copy link
Copy Markdown
Contributor Author

@arunsrini3082 can you check failing ci?

Only DragonFly fails, and it fails at the dependency installation step (sudo exit code 100) before any tests run. FreeBSD, OpenBSD, NetBSD, Solaris, and Darwin/MacOS all pass. This is a DragonFly VM runner issue on AnyVM.org, not related to this PR. The tainted collector has //go:build linux so it doesn't affect any BSD target anyway.

@ArthurSens ArthurSens left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks like documentation is missing in our README, could you add it to the list?

And looks like we're still waiting for the procfs release 😬

Comment thread collector/tainted_linux.go Outdated
Comment thread collector/tainted_linux_test.go
@arunsrini3082

Copy link
Copy Markdown
Contributor Author

Looks like documentation is missing in our README, could you add it to the list?

And looks like we're still waiting for the procfs release 😬

Review feedback addressed: procfs.FS cached on collector, matching !notainted build tag, README updated for node_kernel_tainted.

Arun S and others added 5 commits August 22, 2026 19:02
Signed-off-by: Arun S <arun.srinivasan@flipkart.com>
Signed-off-by: Arun S <arun.srinivasan@flipkart.com>
Signed-off-by: Arun S <arun.srinivasan@flipkart.com>
Co-authored-by: Cursor <cursoragent@cursor.com>
Signed-off-by: Arun S <arun.srinivasan@flipkart.com>
Store procfs.FS on the collector instead of opening it each scrape.
Add matching !notainted build tag on the implementation file.
Document the tainted collector and node_kernel_tainted metrics in README.

Signed-off-by: Arun S <arun.srinivasan@flipkart.com>

@ArthurSens ArthurSens left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

whoops, I forgot to add the comment in the previous review

Comment thread README.md
node_kernel_tainted{bit="0",flag="P"} 0
...
```

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm not sure about this 🤔; we don't usually specify all the collectors in the node exporter on README. Just adding to the table above is sufficient

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants