[Bug]: dcgm-exporter appears to stall after "Initializing system entities of type 'CPU Core'" on H200 with dense MIG topology

[anoopsinghnegi]

Environment

• GPU Operator version: v26.3.0
• dcgm-exporter image: 4.5.1-4.8.0-distroless
• GPU: 8 x NVIDIA H200 NVL
• MIG enabled: true
• OS Type Ubuntu 24.04.3
• Containerd 2.1.5

MIG configuration

Observed behaviour depends on MIG profile:

• 3g.71gb → exporter starts successfully
• 2g.35gb → exporter startup stalls
• 1g.18gb → exporter startup stalls

Problem

nvidia-dcgm-exporter starts and initializes DCGM successfully, but appears to stall during startup and does not proceed to serve metrics / become ready under dense MIG profiles (1g.18gb, 2g.35gb).

Relevant logs:

time=2026-05-26T12:08:45.313Z level=INFO msg="Starting dcgm-exporter" Version=4.5.1-4.8.0
time=2026-05-26T12:08:45.317Z level=INFO msg="Attempting to initialize DCGM."
time=2026-05-26T12:08:50.342Z level=INFO msg="Initialized DCGM Fields module."
time=2026-05-26T12:08:50.342Z level=INFO msg="Attempting to initialize NVML library."
time=2026-05-26T12:08:50.342Z level=INFO msg="NVML provider successfully initialized for Kubernetes MIG support"
time=2026-05-26T12:08:50.342Z level=INFO msg="DCGM successfully initialized!"
time=2026-05-26T12:08:51.107Z level=INFO msg="Successfully queried DCGM profiling metric groups" reload_id=0 count=2 gpu_model="NVIDIA H200 NVL"
time=2026-05-26T12:08:51.107Z level=INFO msg="Building registry for current GPU topology"
time=2026-05-26T12:08:51.107Z level=INFO msg="Falling back to metric file '/etc/dcgm-exporter/dcgm-metrics.csv'"
time=2026-05-26T12:08:51.108Z level=INFO msg="Initializing system entities of type 'GPU'"
time=2026-05-26T12:08:51.616Z level=INFO msg="Initializing system entities of type 'NvSwitch'"
time=2026-05-26T12:08:51.616Z level=INFO msg="Not collecting NvSwitch metrics; no switches to monitor"
time=2026-05-26T12:08:51.616Z level=INFO msg="Initializing system entities of type 'NvLink'"
time=2026-05-26T12:08:51.616Z level=WARN msg="Failed to initialize NvSwitch/NvLink info" error="no switches to monitor"
time=2026-05-26T12:08:52.017Z level=INFO msg="Initializing system entities of type 'CPU'"
time=2026-05-26T12:08:52.055Z level=INFO msg="Not collecting CPU metrics; error retrieving DCGM CPU hierarchy: This request is serviced by a module of DCGM that is not currently loaded"
time=2026-05-26T12:08:52.055Z level=INFO msg="Initializing system entities of type 'CPU Core'"
time=2026-05-26T12:08:52.056Z level=INFO msg="Not collecting CPU Core metrics; error retrieving DCGM CPU hierarchy: This request is serviced by a module of DCGM that is not currently loaded"

No additional startup logs are emitted after this point (for example, no indication that the metrics server is ready).

Workaround observed

Changing:

DCGM_EXPORTER_INTERVAL=2000

to:

DCGM_EXPORTER_INTERVAL=50000

allows the exporter to start successfully and serve metrics.

This suggests startup/initialization behavior may be affected by the default polling interval when many MIG instances are present.

Expected behavior

dcgm-exporter should complete initialization and begin serving metrics regardless of MIG partition density, without requiring a large DCGM_EXPORTER_INTERVAL.

Observations

• DCGM initialization succeeds.
• NVML initialization succeeds.
• Profiling metric group discovery succeeds.
• Issue appears only when the number of MIG instances is high (1g / 2g profiles).
• This suggests a possible issue during registry/entity initialization for dense MIG topologies.

Request

Please investigate whether dcgm-exporter 4.5.1 has a startup/initialization issue with dense MIG configurations on 8 x H200 NVL, especially with 1g.18gb / 2g.35gb profiles.

Also, please let us know if any additional logs/debug flags should be collected to help further diagnose this issue.

[rahulait]

I tested this on GPU Operator v26.3.0, which matches the version reported by you.

Using an 8x NVIDIA H200 NVL node with MIG strategy mixed, I tested the dense MIG profiles:

• all-1g.18gb
• all-2g.35gb
• all-3g.71gb

I was not able to reproduce the DCGM exporter hang. For each profile, MIG configuration reached success, the DCGM exporter pod became Ready, and both /health and /metrics returned HTTP 200.

Observed DCGM exporter readiness:

• all-1g.18gb: Ready in ~63s
• all-2g.35gb: Ready in ~53s
• all-3g.71gb: Ready in ~52s

The exporter logs reached:

Registry built successfully
HTTP server started - ready to serve metrics

The metrics endpoint also exposed the expected MIG instance counts:

• all-1g.18gb: 56
• all-2g.35gb: 24
• all-3g.71gb: 16

So far, I cannot reproduce the issue on v26.3.0 with H200 NVL.

Can you share more details on how you are installing things and any way to reproduce this more consistently.

[rahulait]

@anoopsinghnegi let me know if you are still hitting this and we can troubleshoot further. I haven't been able to reproduce it with my tests.

[anoopsinghnegi]

@rahulait - I've collected the GPU Operator ClusterPolicy and the DCGM ConfigMap from the affected setup.
Please let me know if any additional logs or diagnostics would help with investigating the issue.

$ kubectl get clusterpolicy.nvidia.com cluster-policy -oyaml
apiVersion: nvidia.com/v1
kind: ClusterPolicy
metadata:
  annotations:
    meta.helm.sh/release-name: gpu
    meta.helm.sh/release-namespace: gpu-operator
  creationTimestamp: "2026-06-11T05:21:21Z"
  generation: 1
  labels:
    app.kubernetes.io/component: gpu-operator
    app.kubernetes.io/instance: gpu
    app.kubernetes.io/managed-by: Helm
    app.kubernetes.io/name: gpu-operator
    app.kubernetes.io/version: v26.3.0
    helm.sh/chart: gpu-operator-v26.3.0
  name: cluster-policy
  resourceVersion: "11472"
  uid: bbb1609a-e3ed-499c-ba81-43bcf341e7ee
spec:
  ccManager:
    defaultMode: "on"
    enabled: true
    env:
    - name: CC_CAPABLE_DEVICE_IDS
      value: 0x2339,0x2331,0x2330,0x2324,0x2322,0x233d
    image: k8s-cc-manager
    imagePullPolicy: IfNotPresent
    repository: 172.20.159.50/nvcr.io/nvidia/cloud-native
    resources:
      requests:
        cpu: 100m
        memory: 128Mi
    version: v0.3.0
  cdi:
    default: false
    enabled: true
    nriPluginEnabled: false
  daemonsets:
    labels:
      app.kubernetes.io/managed-by: gpu-operator
      helm.sh/chart: gpu-operator-v26.3.0
    priorityClassName: system-node-critical
    rollingUpdate:
      maxUnavailable: "1"
    tolerations:
    - effect: NoSchedule
      key: nvidia.com/gpu
      operator: Exists
    updateStrategy: RollingUpdate
  dcgm:
    enabled: false
    image: dcgm
    imagePullPolicy: IfNotPresent
    repository: 172.20.159.50/nvcr.io/nvidia/cloud-native
    resources:
      requests:
        cpu: 100m
        memory: 128Mi
    version: 4.5.2-1-ubuntu22.04
  dcgmExporter:
    config:
      name: custom-dcgm-exporter-metrics
    enabled: true
    env:
    - name: DCGM_EXPORTER_LISTEN
      value: :9400
    - name: DCGM_EXPORTER_KUBERNETES
      value: "true"
    - name: DCGM_EXPORTER_INTERVAL
      value: "2000"
    image: dcgm-exporter
    imagePullPolicy: IfNotPresent
    repository: 172.20.159.50/nvcr.io/nvidia/k8s
    resources:
      requests:
        cpu: 100m
        memory: 256Mi
    service:
      internalTrafficPolicy: Cluster
    serviceMonitor:
      additionalLabels: {}
      enabled: true
      honorLabels: false
      interval: 5s
      relabelings: []
    version: 4.5.1-4.8.0-distroless
  devicePlugin:
    enabled: true
    image: k8s-device-plugin
    imagePullPolicy: IfNotPresent
    repository: 172.20.159.50/nvcr.io/nvidia
    resources:
      requests:
        cpu: 100m
        memory: 128Mi
    version: v0.19.0
  driver:
    certConfig:
      name: ""
    enabled: true
    image: driver
    imagePullPolicy: IfNotPresent
    kernelModuleConfig:
      name: ""
    kernelModuleType: auto
    licensingConfig:
      nlsEnabled: true
      secretName: ""
    manager:
      env:
      - name: ENABLE_GPU_POD_EVICTION
        value: "true"
      - name: ENABLE_AUTO_DRAIN
        value: "false"
      - name: DRAIN_USE_FORCE
        value: "false"
      - name: DRAIN_POD_SELECTOR_LABEL
        value: ""
      - name: DRAIN_TIMEOUT_SECONDS
        value: 0s
      - name: DRAIN_DELETE_EMPTYDIR_DATA
        value: "true"
      image: k8s-driver-manager
      imagePullPolicy: IfNotPresent
      repository: 172.20.159.50/nvcr.io/nvidia/cloud-native
      version: v0.10.0
    rdma:
      enabled: false
      useHostMofed: false
    repoConfig:
      configMapName: custom-repo
    repository: 172.20.159.50/nvcr.io/nvidia
    resources:
      requests:
        cpu: 100m
        memory: 4096M
    startupProbe:
      failureThreshold: 120
      initialDelaySeconds: 60
      periodSeconds: 10
      timeoutSeconds: 60
    upgradePolicy:
      autoUpgrade: true
      drain:
        deleteEmptyDir: false
        enable: false
        force: false
        timeoutSeconds: 300
      maxParallelUpgrades: 1
      maxUnavailable: 25%
      podDeletion:
        deleteEmptyDir: true
        force: true
        timeoutSeconds: 900
      waitForCompletion:
        timeoutSeconds: 0
    useNvidiaDriverCRD: false
    usePrecompiled: false
    version: 580.126.20
    virtualTopology:
      config: ""
  gdrcopy:
    enabled: false
    image: gdrdrv
    imagePullPolicy: IfNotPresent
    repository: 172.20.159.50/nvcr.io/nvidia/cloud-native
    version: v2.5.1
  gds:
    enabled: false
    image: nvidia-fs
    imagePullPolicy: IfNotPresent
    repository: 172.20.159.50/nvcr.io/nvidia/cloud-native
    version: 2.27.3
  gfd:
    enabled: true
    image: k8s-device-plugin
    imagePullPolicy: IfNotPresent
    repository: 172.20.159.50/nvcr.io/nvidia
    resources:
      requests:
        cpu: 100m
        memory: 128Mi
    version: v0.19.0
  hostPaths:
    driverInstallDir: /run/nvidia/driver
    rootFS: /
  kataManager:
    enabled: false
    imagePullPolicy: IfNotPresent
    resources:
      requests:
        cpu: 100m
        memory: 128Mi
  kataSandboxDevicePlugin:
    enabled: true
    image: nvidia-sandbox-device-plugin
    imagePullPolicy: IfNotPresent
    repository: 172.20.159.50/nvcr.io/nvidia/cloud-native
    version: v0.0.2
  mig:
    strategy: single
  migManager:
    config:
      default: all-disabled
      name: custom-mig-config
    enabled: true
    gpuClientsConfig:
      name: ""
    image: k8s-mig-manager
    imagePullPolicy: IfNotPresent
    repository: 172.20.159.50/nvcr.io/nvidia/cloud-native
    resources:
      requests:
        cpu: 100m
        memory: 128Mi
    version: v0.14.0
  nodeStatusExporter:
    enabled: false
    image: gpu-operator
    imagePullPolicy: IfNotPresent
    repository: 172.20.159.50/nvcr.io/nvidia
    resources:
      requests:
        cpu: 100m
        memory: 128Mi
    version: v26.3.0
  operator:
    runtimeClass: nvidia
  psa:
    enabled: false
  sandboxDevicePlugin:
    enabled: true
    image: kubevirt-gpu-device-plugin
    imagePullPolicy: IfNotPresent
    repository: 172.20.159.50/nvcr.io/nvidia
    resources:
      requests:
        cpu: 100m
        memory: 128Mi
    version: v1.5.0
  sandboxWorkloads:
    defaultWorkload: container
    enabled: false
    mode: kubevirt
  toolkit:
    enabled: true
    image: container-toolkit
    imagePullPolicy: IfNotPresent
    installDir: /usr/local/nvidia
    repository: 172.20.159.50/nvcr.io/nvidia/k8s
    resources:
      requests:
        cpu: 100m
        memory: 128Mi
    version: v1.19.0
  validator:
    image: gpu-operator
    imagePullPolicy: IfNotPresent
    plugin:
      env: []
    repository: 172.20.159.50/nvcr.io/nvidia
    resources:
      requests:
        cpu: 10m
        memory: 64Mi
    version: v26.3.0
  vfioManager:
    driverManager:
      image: k8s-driver-manager
      imagePullPolicy: IfNotPresent
      repository: 172.20.159.50/nvcr.io/nvidia/cloud-native
      version: v0.10.0
    enabled: true
    image: k8s-driver-manager
    imagePullPolicy: IfNotPresent
    repository: 172.20.159.50/nvcr.io/nvidia/cloud-native
    resources:
      requests:
        cpu: 100m
        memory: 128Mi
    version: v0.10.0
  vgpuDeviceManager:
    config:
      default: default
      name: ""
    enabled: true
    image: vgpu-device-manager
    imagePullPolicy: IfNotPresent
    repository: 172.20.159.50/nvcr.io/nvidia/cloud-native
    version: v0.4.2
  vgpuManager:
    driverManager:
      image: k8s-driver-manager
      imagePullPolicy: IfNotPresent
      repository: 172.20.159.50/nvcr.io/nvidia/cloud-native
      version: v0.10.0
    enabled: false
    image: vgpu-manager
    imagePullPolicy: IfNotPresent
    kernelModuleConfig:
      name: ""
    repository: 172.20.159.50
    resources:
      requests:
        cpu: 100m
        memory: 128Mi

DCGM ConfigMap

$ kubectl get cm -n gpu-operator custom-dcgm-exporter-metrics
NAME                           DATA   AGE
custom-dcgm-exporter-metrics   1      94m
pcadmin@ez-master03:~$ kubectl get cm -n gpu-operator custom-dcgm-exporter-metrics -oyaml
apiVersion: v1
data:
  dcgm-metrics.csv: |
    # Format
    # If line starts with a '#' it is considered a comment
    # DCGM FIELD, Prometheus metric type, help message

    # Clocks
    DCGM_FI_DEV_SM_CLOCK,  gauge, SM clock frequency (in MHz).
    DCGM_FI_DEV_MEM_CLOCK, gauge, Memory clock frequency (in MHz).

    # Temperature
    DCGM_FI_DEV_MEMORY_TEMP, gauge, Memory temperature (in C).
    DCGM_FI_DEV_GPU_TEMP,    gauge, GPU temperature (in C).

    # Power
    DCGM_FI_DEV_POWER_USAGE,              gauge, Power draw (in W).
    DCGM_FI_DEV_TOTAL_ENERGY_CONSUMPTION, counter, Total energy consumption since boot (in mJ).

    # PCIE
    # DCGM_FI_PROF_PCIE_TX_BYTES,  counter, Total number of bytes transmitted through PCIe TX via NVML.
    # DCGM_FI_PROF_PCIE_RX_BYTES,  counter, Total number of bytes received through PCIe RX via NVML.
    DCGM_FI_DEV_PCIE_REPLAY_COUNTER, counter, Total number of PCIe retries.

    # Utilization (the sample period varies depending on the product)
    DCGM_FI_DEV_GPU_UTIL,      gauge, GPU utilization (in %).
    DCGM_FI_DEV_MEM_COPY_UTIL, gauge, Memory utilization (in %).
    DCGM_FI_DEV_ENC_UTIL,      gauge, Encoder utilization (in %).
    DCGM_FI_DEV_DEC_UTIL ,     gauge, Decoder utilization (in %).

    # Errors and violations
    DCGM_FI_DEV_XID_ERRORS,            gauge,   Value of the last XID error encountered.
    # DCGM_FI_DEV_POWER_VIOLATION,       counter, Throttling duration due to power constraints (in ns).
    # DCGM_FI_DEV_THERMAL_VIOLATION,     counter, Throttling duration due to thermal constraints (in ns).
    # DCGM_FI_DEV_SYNC_BOOST_VIOLATION,  counter, Throttling duration due to sync-boost constraints (in ns).
    # DCGM_FI_DEV_BOARD_LIMIT_VIOLATION, counter, Throttling duration due to board limit constraints (in ns).
    # DCGM_FI_DEV_LOW_UTIL_VIOLATION,    counter, Throttling duration due to low utilization (in ns).
    # DCGM_FI_DEV_RELIABILITY_VIOLATION, counter, Throttling duration due to reliability constraints (in ns).

    # Memory usage
    DCGM_FI_DEV_FB_FREE, gauge, Framebuffer memory free (in MiB).
    DCGM_FI_DEV_FB_USED, gauge, Framebuffer memory used (in MiB).
    DCGM_FI_DEV_FB_RESERVED, gauge, Framebuffer memory reserved (in MiB).

    # ECC
    # DCGM_FI_DEV_ECC_SBE_VOL_TOTAL, counter, Total number of single-bit volatile ECC errors.
    # DCGM_FI_DEV_ECC_DBE_VOL_TOTAL, counter, Total number of double-bit volatile ECC errors.
    # DCGM_FI_DEV_ECC_SBE_AGG_TOTAL, counter, Total number of single-bit persistent ECC errors.
    # DCGM_FI_DEV_ECC_DBE_AGG_TOTAL, counter, Total number of double-bit persistent ECC errors.

    # Retired pages
    # DCGM_FI_DEV_RETIRED_SBE,     counter, Total number of retired pages due to single-bit errors.
    # DCGM_FI_DEV_RETIRED_DBE,     counter, Total number of retired pages due to double-bit errors.
    # DCGM_FI_DEV_RETIRED_PENDING, counter, Total number of pages pending retirement.

    # NVLink
    # DCGM_FI_DEV_NVLINK_CRC_FLIT_ERROR_COUNT_TOTAL, counter, Total number of NVLink flow-control CRC errors.
    # DCGM_FI_DEV_NVLINK_CRC_DATA_ERROR_COUNT_TOTAL, counter, Total number of NVLink data CRC errors.
    # DCGM_FI_DEV_NVLINK_REPLAY_ERROR_COUNT_TOTAL,   counter, Total number of NVLink retries.
    # DCGM_FI_DEV_NVLINK_RECOVERY_ERROR_COUNT_TOTAL, counter, Total number of NVLink recovery errors.
    DCGM_FI_DEV_NVLINK_BANDWIDTH_TOTAL,            counter, Total number of NVLink bandwidth counters for all lanes.
    # DCGM_FI_DEV_NVLINK_BANDWIDTH_L0,               counter, The number of bytes of active NVLink rx or tx data including both header and payload.

    # VGPU License status
    DCGM_FI_DEV_VGPU_LICENSE_STATUS, gauge, vGPU License status

    # Remapped rows
    DCGM_FI_DEV_UNCORRECTABLE_REMAPPED_ROWS, counter, Number of remapped rows for uncorrectable errors
    DCGM_FI_DEV_CORRECTABLE_REMAPPED_ROWS,   counter, Number of remapped rows for correctable errors
    DCGM_FI_DEV_ROW_REMAP_FAILURE,           gauge,   Whether remapping of rows has failed

    # Static configuration information. These appear as labels on the other metrics
    DCGM_FI_DRIVER_VERSION,        label, Driver Version
    # DCGM_FI_NVML_VERSION,          label, NVML Version
    # DCGM_FI_DEV_BRAND,             label, Device Brand
    DCGM_FI_DEV_SERIAL,            label, Device Serial Number
    # DCGM_FI_DEV_OEM_INFOROM_VER,   label, OEM inforom version
    # DCGM_FI_DEV_ECC_INFOROM_VER,   label, ECC inforom version
    # DCGM_FI_DEV_POWER_INFOROM_VER, label, Power management object inforom version
    # DCGM_FI_DEV_INFOROM_IMAGE_VER, label, Inforom image version
    # DCGM_FI_DEV_VBIOS_VERSION,     label, VBIOS version of the device

    # DCP metrics
    DCGM_FI_PROF_GR_ENGINE_ACTIVE,   gauge, Ratio of time the graphics engine is active.
    # DCGM_FI_PROF_SM_ACTIVE,          gauge, The ratio of cycles an SM has at least 1 warp assigned.
    # DCGM_FI_PROF_SM_OCCUPANCY,       gauge, The ratio of number of warps resident on an SM.
    DCGM_FI_PROF_PIPE_TENSOR_ACTIVE, gauge, Ratio of cycles the tensor (HMMA) pipe is active.
    DCGM_FI_PROF_DRAM_ACTIVE,        gauge, Ratio of cycles the device memory interface is active sending or receiving data.
    # DCGM_FI_PROF_PIPE_FP64_ACTIVE,   gauge, Ratio of cycles the fp64 pipes are active.
    # DCGM_FI_PROF_PIPE_FP32_ACTIVE,   gauge, Ratio of cycles the fp32 pipes are active.
    # DCGM_FI_PROF_PIPE_FP16_ACTIVE,   gauge, Ratio of cycles the fp16 pipes are active.
    DCGM_FI_PROF_PCIE_TX_BYTES,      gauge, The rate of data transmitted over the PCIe bus - including both protocol headers and data payloads - in bytes per second.
    DCGM_FI_PROF_PCIE_RX_BYTES,      gauge, The rate of data received over the PCIe bus - including both protocol headers and data payloads - in bytes per second.
kind: ConfigMap
metadata:
  annotations:
    meta.helm.sh/release-name: gpu
    meta.helm.sh/release-namespace: gpu-operator
  creationTimestamp: "2026-06-11T05:21:20Z"
  labels:
    app.kubernetes.io/instance: gpu
    app.kubernetes.io/managed-by: Helm
    app.kubernetes.io/name: gpu-operator
    app.kubernetes.io/version: v26.3.0
    helm.sh/chart: gpu-operator-v26.3.0
  name: custom-dcgm-exporter-metrics
  namespace: gpu-operator
  resourceVersion: "9361"
  uid: f914a8e3-573b-4b0c-b523-2b7a05bbbc70

[anoopsinghnegi]

@rahulait The issue occurs when multiple GPU workloads are active on the node while MIG is disabled, and the node is then reconfigured to enable MIG by changing the partitioning profile from "all-disabled" to "1g.18gb".

[rahulait]

Ah ok, so you don't see it with no workloads but only when there are multiple workloads correct?

[anoopsinghnegi]

No, even with no workloads, the issue was still occurring.