Show host GPU driver on fleet instances#4032
Draft
peterschmidt85 wants to merge 1 commit into
Draft
Conversation
Detect the installed accelerator driver (NVIDIA, AMD, Tenstorrent) on the host via dstack-shim, report it through the existing healthcheck response, store it in JobProvisioningData, and surface it on the Instance API model and as a DRIVER column in `dstack fleet -v`. The field stays None when the driver cannot be detected. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
When
dstackprovisions instances, the accelerator driver installed on the host (NVIDIA/AMD/Tenstorrent) is not visible anywhere — anyone listing fleet instances cannot tell which driver a host runs.Solution
Detect the driver on the host and show it on fleet instances. When it cannot be detected, the field stays
None.dstack-shimdetects the driver version once at startup, inside the existing GPU detection: an extradriver_versioncolumn in thenvidia-smiquery,--driveradded to the existingamd-smi staticcall (with a fallback retry for olderamd-smi), and/sys/module/tenstorrent/versionfor Tenstorrent.gpu_vendor/gpu_driver_versionfields. The endpoint, its polling, and the handler cost are unchanged — the handler reads only the GPU list cached at startup.JobProvisioningData.gpu_driver(GpuDriverInfo {vendor, version}, no DB migration): captured by the instance check for VM-based backends (this also backfills fleets provisioned before the upgrade) and at deploy time for SSH fleets.Instance.gpu_driverin the API and aDRIVERcolumn indstack fleet -v.None.Nonefor now;kubernetesandvastaicomputes carry TODOs with implementation notes. Intel is intentionally not detected.🤖 Generated with Claude Code