PW/BPCG DFT+U GPU crash at the first SCF iteration for Cr2O3 with CUSOLVER_STATUS_INTERNAL_ERROR

[lanshuyue]

Describe the bug

Description

A plane-wave DFT+U SCF calculation for Cr2O3 crashes reproducibly in the GPU/BPCG path during the first electronic iteration with ABACUS v3.11.0-beta6 (commit 31c899d33).

The same PW physical input completes successfully on the CPU when launched with a correct 16-rank MPI communicator. A corresponding LCAO/GPU calculation for the same Cr2O3 system also converges successfully with cuSolver.

The current evidence therefore points to this PW/BPCG GPU path, but it does not establish a specific source-code root cause or imply that cuSolver or all GPU calculations fail.

Expected behavior

The PW/GPU calculation should proceed through the SCF cycle without a cuSolver internal error or an illegal CUDA memory access, as observed in the PW/CPU control.

Actual behavior

running_scf.log reaches only:

--> #ION MOVE#         1  #ELEC ITER#         1

The process then exits without #SCF IS CONVERGED# or a normal completion marker.

The relevant stderr messages are:

In File <ABACUS_ROOT>/source/source_base/module_container/base/third_party/cusolver.h : CUSOLVER API failed at line 65 with error: CUSOLVER_STATUS_INTERNAL_ERROR (7)
In File <ABACUS_ROOT>/source/source_base/module_device/cuda/memory_op.cu : CUDA API failed at line 230 with error: an illegal memory access was encountered (700)

Steps to reproduce

1. Use ABACUS v3.11.0-beta6 at commit 31c899d33, built for CUDA 11.8.
2. Extract the minimal reproducer and enter pw_gpu_failure.
3. Set ABACUS_ROOT to the ABACUS installation prefix.
4. Add any site-specific Slurm partition/account options.
5. Submit run.slurm.

The configuration is:

basis_type: pw
ks_solver: bpcg
DFT+U: enabled
MPI ranks: 1
OpenMP threads: 1
GPUs: 1
GPU: NVIDIA A800 80 GB PCIe

The script launches ABACUS through MPI even for the single-rank case:

mpirun -np 1 "${ABACUS_ROOT}/bin/abacus"

Successful controls

PW/CPU control

The PW/CPU case uses the same physical input, with only device changed from gpu to cpu, and is launched with:

mpirun -np 16 "${ABACUS_ROOT}/bin/abacus"

The runtime logs confirm:

Local MPI proc number: 16
PROCESSES: 16

It converges successfully after 10 electronic iterations:

!FINAL_ETOT_IS -11727.7703004102786508 eV
Total Time: 598 s
ABACUS job finished successfully.

LCAO/GPU control

The LCAO/GPU control uses the same Cr2O3 structure, pseudopotentials, U values, initial magnetic moments, cutoff, k-point sampling, mixing parameters, and SCF threshold, with:

basis_type: lcao
ks_solver: cusolver
MPI ranks: 1
OpenMP threads: 1
GPUs: 1

It converges successfully after 15 electronic iterations:

!FINAL_ETOT_IS -11725.7764036216649401 eV
Total Time: 284 s
ABACUS job finished successfully.

Reproducibility

The same two fatal signatures occur at the first electronic iteration in all of the following valid GPU tests:

• three identical-configuration repeats;
• GPU runs with 1, 2, and 4 MPI ranks;
• one independent GPU crosscheck.

A compact summary is included in reproducibility_summary.tsv.

Minimal attachment

The attachment contains only:

• inputs and run scripts for the three main cases;
• the required structure, pseudopotential, and orbital files;
• primary logs sanitized for paths and identity information;
• a compact GPU reproducibility summary;
• SHA-256 checksums.

The scientific inputs and error messages were not modified. Wavefunctions, charge densities, restart files, Mulliken/occupation output, and invalid singleton-MPI CPU results are excluded.

abacus_cr2o3_pw_gpu_issue.zip

Expected behavior

No response

To Reproduce

No response

Environment

No response

Additional Context

No response

Task list for Issue attackers (only for developers)

• Verify the issue is not a duplicate.
• Describe the bug.
• Steps to reproduce.
• Expected behavior.
• Error message.
• Environment details.
• Additional context.
• Assign a priority level (low, medium, high, urgent).
• Assign the issue to a team member.
• Label the issue with relevant tags.
• Identify possible related issues.
• Create a unit test or automated test to reproduce the bug (if applicable).
• Fix the bug.
• Test the fix.
• Update documentation (if necessary).
• Close the issue and inform the reporter (if applicable).

[lanshuyue]

I would like to add one scope clarification based on an additional successful
control.

The same ABACUS v3.11.0-beta6 build (commit 31c899d33), CUDA 11.8
environment, and NVIDIA A800 80 GB GPU successfully completed a PW/BPCG DFT+U
SCF calculation for Cu2O. The run used 1 MPI rank, 1 OpenMP thread, and 1 GPU.
It converged in 8 electronic iterations, with a final energy of
-20739.1056799952166330 eV and a total time of 448 s. The logs contain both
the SCF convergence marker and ABACUS job finished successfully.

Neither CUSOLVER_STATUS_INTERNAL_ERROR (7) nor the CUDA illegal-memory-access
error (700) appears in this successful control.

Therefore, the original report should not be interpreted as showing that all
PW/BPCG DFT+U GPU calculations fail. A more precise scope is that the crash
has been observed for the reported Cr2O3 PW/BPCG DFT+U GPU case.

abacus_pw_gpu_success_control.zip