[juenglin]

Description

Refs #3195

Note

Temporary incident tracker (not a permanent CI fixture): This workflow and its helper scripts exist solely to track and verify the runner-fleet remediation for the eMudhra/emSign CA compromise. It is report-only (never gates PRs or merges), requires no ongoing maintenance from the library team, and will be deleted once all hosted and self-hosted runner labels pass both checks (or the public test endpoint is retired).

Background

Following the eMudhra/emSign CA key compromise, hash-pinning individual downloads only protects those specific artifacts; general TLS traffic in CI (such as actions/checkout, gh, the Actions cache/artifact services, and unpinned package downloads) still relies on the certificate stores present on the runner. Every runner has two independent trust stores: the operating system store (used by git, curl, wget, gh, and Docker) and the root bundle compiled into the Node.js runtime shipped with the GitHub Actions runner agent (used by JavaScript actions like actions/cache and actions/upload-artifact). Because updating those stores depends on upstream GitHub-hosted image rollouts and self-hosted runner maintenance outside this repository, this probe gives us and the runner operators direct visibility into which runner labels still trust the compromised emSign ECC Root CA - G3 anchor until the rollout completes.

What turns a probe job green (and how to test fixes)

The goal is to get every runner label's probe job to green, at which point this workflow can be removed. A label's job turns green when both stores on that runner are remediated:

1. OS certificate store updated:
   • Linux: emSign ECC Root CA - G3 is absent from /etc/ssl/certs/ca-certificates.crt (on Ubuntu 22.04/24.04/26.04, via ca-certificates >= 20260816ubuntu0.{22,24,26}.04.1; on ubuntu-latest, baked into the base runner image itself rather than only after apt-get upgrade), and curl does not trust https://dv-g3b-emudhra.we-have-a-problem.com/.
   • Windows: emSign ECC Root CA - G3 is absent from Cert:\LocalMachine\Root (or present in Cert:\LocalMachine\Disallowed), and Invoke-WebRequest does not trust the test endpoint.
2. Runner agent's bundled Node.js updated:
   • The runner agent bundles Node 24 >= v24.22.1 (first shipped in actions/runner v2.338.1), and Node's https.get rejects the test endpoint's certificate chain. The probe checks the Node version directly, because a runner image can carry a different Node than its runner version implies (see the takeaways under Verification).

Self-service verification (workflow_dispatch): Beyond the weekly schedule, the workflow supports manual workflow_dispatch triggers with an optional label-filter regex input (for example, ^linux-amd64-cpu8$ or ^windows-amd64-gpu-l4-latest-1$). The runner team can use this at any time to re-probe specific runner labels on demand and verify image or runner-agent updates without waiting for the scheduled run or probing scarce multi-GPU runners.

Changes

• Add .github/workflows/runner-trust-probe.yml (schedule + workflow_dispatch with an optional label-filter regex). GPU runner labels are derived from ci/test-matrix.yml; hosted and self-hosted CPU labels are listed explicitly.
• Probe each runner in the environment used by real CI jobs (ubuntu:24.04 container for Linux GPU labels, host for Linux CPU and Windows labels, and both before and after apt-get install --only-upgrade -y ca-certificates on ubuntu-latest). The post-upgrade probe verifies that this stopgap works; the pre-upgrade probe gates the job as described above.
• Add ci/tools/runner_trust_probe.sh and ci/tools/runner_trust_probe.ps1 to check that emSign ECC Root CA - G3 (86:A1:EC:BA:08:9C:4A:8D:3B:BE:27:34:C6:12:BA:34:1D:81:3E:04:3C:F9:E8:A8:62:CD:5C:57:A3:6B:BE:6B) is absent/untrusted, report the installed ca-certificates package version, verify OS TLS clients (curl / Invoke-WebRequest) reject the test endpoint, and write summary tables to $GITHUB_STEP_SUMMARY.
• Add ci/tools/runner_trust_probe_node.js to check the runner agent's bundled Node runtime (require Node 24 >= v24.22.1 and TLS certificate chain rejection of the test endpoint) plus an informational node20 action runtime check.
• Register the probe files in IGNORED_PATHS in ci/tools/compute_ci_plan.py so probe-only changes do not trigger full wheel builds.

Verification

• The temporary PR trigger exercised the four GitHub-hosted runner labels in run #40577671216 (commit 1070160643). That run predates the final simplification commit, so its step summaries differ slightly from current runs: its ca-certificates rows carry INFO (STALE) / INFO (OK) markers, which current runs replace with the plain installed version, and its Node 20 step prints a single line instead of the full probe table. A full-fleet workflow_dispatch run covering the self-hosted CPU, Linux GPU (container), WSL, and Windows GPU labels is planned right after merge.
• Key takeaways from the hosted-runner run:
  • Ubuntu 26.04 stopgap works: ubuntu-latest now maps to Ubuntu 26.04 with ca-certificates 20260601~26.04.1 pre-installed; upgrading ca-certificates to 20260816ubuntu0.26.04.1 removes emSign ECC Root CA - G3 and causes curl to reject the test endpoint.
  • Runner version != Node version: All four hosted labels report runner version 2.338.0, but ubuntu-latest bundles Node v24.22.1 (fixed) whereas the three hosted Windows images cache Node v24.21.0 (unfixed).
  • node20 actions run on node24: Hosted runners already force node20 actions onto their node24 runtime.

Checklist

• ci/tools/tests/test_compute_ci_plan.py covers the compute_ci_plan.py change; the probe itself is temporary and report-only.
• The documentation is up to date with these changes. (N/A: internal CI workflow only)

[copy-pr-bot]

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

[juenglin]

/ok to test 711671b

[coderabbitai]

📝 Summary

Summary by CodeRabbit

• Chores
  • Added weekly and manually triggered certificate trust checks for Linux and Windows runners, including Linux GPU runners. Manual runs can filter runner labels. Checks report operating system and Node.js trust results; Node 20 results are informational, while required checks can fail when trust requirements are not met. The workflow runs only in repositories owned by NVIDIA.
  • Changes limited to the trust-check workflow and its supporting scripts no longer trigger a full CI run. Other changes to CI workflow paths continue to trigger one.

Adds a scheduled and manually triggered workflow that probes OS and bundled Node trust on selected Linux and Windows runners. It adds platform probe scripts and updates CI planning to exclude probe-related paths from full CI runs.

Changes: Runner trust probing

Layer / File(s)	Summary
Trust probe checks and results ci/tools/runner_trust_probe.sh, ci/tools/runner_trust_probe.ps1, ci/tools/runner_trust_probe_node.js	The scripts check certificate stores and HTTPS endpoints, report results, and fail on their specified gating conditions. The Node probe also checks for Node 24.22.1 or later.
Workflow triggers, runner selection, and probe jobs .github/workflows/runner-trust-probe.yml, ci/tools/compute_ci_plan.py, ci/tools/tests/test_compute_ci_plan.py	The workflow runs weekly or by manual dispatch. It builds filtered runner-label matrices and runs host and Linux GPU probe jobs. CI planning skips probe-related paths, and tests cover those paths.

Possibly related PRs

• NVIDIA/cuda-python#3197: Updates certificate-related packages and pip hardening for the emSign incident.
• NVIDIA/cuda-python#3201: Applies pip hardening introduced for the emSign incident.
• NVIDIA/cuda-python#3248: Replaces sccache binaries because one release embeds the affected emSign root and documents root checks.

Priority: Normal

Change: Other

Merge Risk: Low · up to ca21d

The monitoring workflow can show expected runner versions as failed jobs, and its GPU setup depends on a mutable action reference. These are bounded risks, but the report-only fix should be applied.

- [ ] Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

_{Comment @coderabbitai help to get the list of available commands.}

[coderabbitai]

Actionable comments posted: 2

---

Review info

Run configuration

• Source mode: Auto
• Repository: NVIDIA/cuda-python
• Branch: main
• File: .coderabbit.yaml
• Commit: f9212dd
• Layers: Repository YAML
• Review profile: CHILL
• Plan: Enterprise
• Run ID: f74d6f6c-9bdf-4723-84f2-11497eac233f

Commits

Reviewing files that changed from the base of the PR and between f9212dd and 711671b.

Files selected for processing: 6

• .github/workflows/runner-trust-probe.yml
• ci/tools/compute_ci_plan.py
• ci/tools/runner_trust_probe.ps1
• ci/tools/runner_trust_probe.sh
• ci/tools/runner_trust_probe_node.js
• ci/tools/tests/test_compute_ci_plan.py

Included review availability: This review used your included allowance. Your plan provides up to 12 included reviews per hour; 11 remain after this review.

[jpascucci-nv]

ok

[juenglin]

/ok to test 2d119d7

[leofang]

It's unclear to me how we're supposed to maintain this as a library team... I'll let others chime in

[leofang]

This kinda makes me sad...

1. This can go out of sync easily
2. It seems overly defensive (that we are responsible for testing infra owned and operated by someone else)

Is this check really necessary?

[rwgk]

codex GPT-6.1-Sol ultra found:

• [P2] HTTP errors can hide successful TLS validation — runner_trust_probe.ps1:51. Invoke-WebRequest throws for HTTP 4xx/5xx, which the catch classifies as INCONCLUSIVE. This can leave the job green despite trusting the certificate when the separate LocalMachine check passes, such as with CurrentUser trust. Add -SkipHttpErrorCheck so HTTP error responses also reach the trust-failure branch. A mocked HTTP 503 reproduced INCONCLUSIVE with exit code 0.

[juenglin]

It's unclear to me how we're supposed to maintain this as a library team... I'll let others chime in.

@leofang Thanks for the review!

The new workflow is meant to be temporary. It helps us to track and verify the runner fixes (it goes green when the runner's trust stores are cleaned up). We will want to see it stay green for some time--a month or two, say--and then remove it.

[juenglin]

/ok to test ca21deb

[coderabbitai]

Caution

Some comments are outside the diff and can’t be posted inline due to GitHub limitations.

⚠️ Outside diff range comments (1)

Minor · important: Keep the bundled Node probe report-only. · runner-trust-probe.yml:127-133

.github/workflows/runner-trust-probe.yml:127-133
🩺 Stability & Availability | 🟡 Minor | ⚡ Quick win

important: Keep the bundled Node probe report-only.

The workflow runs on schedule and manual dispatch, but this step enables the helper’s default gating. A runner with Node v24.21.0 therefore calls core.setFailed(...) and fails its job. Set gating: false and continue-on-error: true, as the Node 20 informational probe does.

Suggested fix

       - name: Probe bundled Node runtime and trust store
         if: ${{ always() }}
+        continue-on-error: true
         uses: actions/github-script@3a2844b7e9c422d3c10d287c895573f7108da1b3  # v9.0.0
         with:
           script: |
             const probe = require(`${process.env.GITHUB_WORKSPACE}/ci/tools/runner_trust_probe_node.js`);
-            await probe({ core });
+            await probe({ core, gating: false });

---

Review info

Run configuration

• Source mode: Auto
• Repository: NVIDIA/cuda-python
• Branch: main
• File: .coderabbit.yaml
• Commit: f9212dd
• Layers: Repository YAML
• Review profile: CHILL
• Plan: Enterprise
• Run ID: 49f5dbb6-b04b-418b-b006-a5f3a42ce248

Commits

Reviewing files that changed from the base of the PR and between 2d119d7 and ca21deb.

Files selected for processing: 2

• .github/workflows/runner-trust-probe.yml
• ci/tools/runner_trust_probe.ps1

Included review availability: This review used your included allowance. Your plan provides up to 12 included reviews per hour; 9 remain after this review.

[rwgk]

It's unclear to me how we're supposed to maintain this as a library team... I'll let others chime in

This question made me look at stats, first with cloc, then codex. See below. It turns out, Build, CI, packaging and tooling LOC is nearly ~50% of Library implementation.

---

Yes. Your installed cloc supports excluding files by extension and content directly. Documentation

cloc . --exclude-ext=pyi \
  --exclude-content='CYTHON-BINDINGS-GENERATED-DO-NOT-MODIFY-THIS-FILE'

I reproduced your original totals exactly. That command reduces 355,665 → 135,462 code lines, removing 61.9%. It leaves 1,115 counted files, 38,468 blank lines, and 43,448 comment lines. All 51 tracked .pyi files are generated.

After also excluding documentation prose and vendored code:

Category	Code lines
Library implementation	32,288
Tests and test helpers	52,324
Examples and samples	13,429
Benchmarks	3,376
Build, CI, packaging and tooling	15,704
Documentation build/configuration tooling	1,285
Total	118,406

The library implementation consists of 24,800 lines in core, 2,900 in pathfinder, and 4,588 in the two bindings packages’ authored glue.

So infrastructure narrowly construed is about 17k lines, including documentation tooling. Grouping all tests, examples, benchmarks, and tooling as supporting code gives 86k supporting versus 32k library, about 2.7:1.

These use cloc’s code column and default duplicate suppression; configuration counts as code. Excluding documentation tooling too leaves 117,121 lines.

[leofang]

We will want to see it stay green for some time--a month or two, say--and then remove it.

I created a tracking issue: #3257