Implement safe wrapper for nvmlDeviceGetGpuFabricInfoV.

[mattnappo]

This PR provides a safe wrapper over nvmlReturn_t nvmlDeviceGetGpuFabricInfoV ( nvmlDevice_t device, nvmlGpuFabricInfoV_t* gpuFabricInfo ) by invoking the FFI binding provided by nvml-wrapper-sys in a similar way as the rest of this crate's wrappers.

This returns a Rust struct corresponding to nvmlGpuFabricInfo_v3_t:

unsigned int  cliqueId
// ID of the fabric clique to which this GPU belongs.
unsigned char  clusterUuid[NVML_GPU_FABRIC_UUID_LEN]
// Uuid of the cluster to which this GPU belongs.
unsigned int  healthMask
// GPU Fabric health Status Mask. See NVML_GPU_FABRIC_HEALTH_MASK_*.
unsigned char  healthSummary
// GPU Fabric health summary. See NVML_GPU_FABRIC_HEALTH_SUMMARY_*.
nvmlGpuFabricState_t state
// Current Probe State of GPU registration process. See NVML_GPU_FABRIC_STATE_*.
nvmlReturn_t status
// Probe Error status, if any. Must be checked only if Probe state returns "complete".
unsigned int  version
// Structure version identifier (set to nvmlGpuFabricInfo_v2).

One limitation is that the nvml-wrapper-sys crate does not contain bindings for the following health variants:

• Incompatible GPU Firmware
• Invalid Location
• GPU State Invalid

This can be addressed in follow up work. See

nvml-wrapper/nvml-wrapper-sys/src/bindings.rs

Line 468 in eb47417

pub const NVML_GPU_FABRIC_HEALTH_MASK_DEGRADED_BW_NOT_SUPPORTED: u32 = 0;

The older nvmlReturn_t nvmlDeviceGetGpuFabricInfo ( nvmlDevice_t device, nvmlGpuFabricInfo_t* gpuFabricInfo ) does not provide the healthSummary field, so I do not implement it here (see nvmlGpuFabricInfo_t).

Note that the output of GpuFabricInfo::registration_result is only worth reading when GpuFabricState is Completed. Registering a device with the NVLink fabric is async, and status is only reported when registration/initialization completes.

From the Nvidia docs:

GPU Fabric information
State
Indicates the state of the GPU's handshake with the nvidia-fabricmanager (a.k.a. GPU fabric probe)
Possible values: Completed, In Progress, Not Started, Not supported
Status
Status of the GPU fabric probe response from the nvidia-fabricmanager.
Possible values: NVML_SUCCESS or one of the failure codes.
Clique ID
A clique is a set of GPUs that can communicate to each other over NVLink.
The GPUs belonging to the same clique share the same clique ID.
Clique ID will only be valid for NVLink multi-node systems.
Cluster UUID
UUID of an NVLink multi-node cluster to which this GPU belongs.
Cluster UUID will be zero for NVLink single-node systems.
Health
Summary - Summary of Fabric Health <Healthy, Unhealthy, Limited Capacity>
Bandwidth - is the GPU NVLink bandwidth degraded <Degraded/Full>
Route Recovery in progress - is NVLink route recovery in progress <True/False>
Route Unhealthy - is NVLink route recovery failed or aborted <True/False>
Access Timeout Recovery - is NVLink access timeout recovery in progress <True/False>
Incorrect Configuration - Incorrect Configuration status <Incorrect SystemGuid, Incorrect Chassis Serial Number, No Partition, Insufficient Nvlink Resources, Incompatible GPU Firmware, Invalid Location, GPU State Invalid, None>
Partition Assigned - is the GPU NVLink partition correctly assigned <True/False>

On single-node 8xB300 systems, nvidia-smi reports the following when a device's NVLink is healthy:

    Fabric
        State                             : Completed
        Status                            : Success
        CliqueId                          : 0
        ClusterUUID                       : 00000000-0000-0000-0000-000000000000
        Health
            Summary                       : Healthy
            Bandwidth                     : N/A
            Route Recovery in progress    : N/A
            Route Unhealthy               : N/A
            Access Timeout Recovery       : False
            Incorrect Configuration       : N/A

This can now be represented in Rust like so:

        let raw = nvmlGpuFabricInfoV_t {
            version: 0,
            clusterUuid: [0; 16],
            status: nvmlReturn_enum_NVML_SUCCESS,
            cliqueId: 0,
            state: NVML_GPU_FABRIC_STATE_COMPLETED as u8,
            healthMask: health_mask,
            healthSummary: NVML_GPU_FABRIC_HEALTH_SUMMARY_HEALTHY as u8,
        };

        let info = GpuFabricInfo::from(raw);

During fabric-manager initialization, nvidia-smi reports the following:

Fabric
    State                             : In Progress
    Status                            : N/A
    CliqueId                          : N/A
    ClusterUUID                       : N/A
    Health
        Summary                       : N/A

Machines stuck in this in progress state may have underlying NVLink health issues.