Skip to content

compiler: Write the complex arithmetic header atomically - #3071

Merged
ggorman merged 2 commits into
mainfrom
fix-complex-header-race
Oct 21, 2026
Merged

ggorman merged 2 commits into
mainfrom
fix-complex-header-race

Conversation

@ggorman

@ggorman ggorman commented Oct 17, 2026

Copy link
Copy Markdown
Contributor

_complex_includes writes the extra complex arithmetic definitions a C++ language needs (def-complex) to complex_arith.h in the JIT cache directory, with open(..., 'w'), on every lowering of an Operator that uses a complex dtype. The directory is shared by every process compiling as the same user, and 'w' truncates the file before writing it, so with pytest -n auto one worker can truncate the header while another's compiler is reading it. The compile then fails on the first complex expression, e.g. no match for 'operator*' (operand types are 'int' and 'const std::complex<float>'); in DevitoPRO's CUDA backend the same race shows as user-defined literal operator not found for 1.0_if.

Changes:

  • The header is keyed on its content (complex_arith_<sha1>.h), written to a private temporary file and renamed into place, and skipped when it already exists. A reader therefore sees either no file or a complete one, and two languages with different definitions no longer overwrite each other.
  • test_complex_arith_header checks that two Operators share one header whose content is exactly def-complex.

Verification on main d7f53a3:

  • 32 processes each JIT-compiling 30 distinct complex-valued language='CXX' Operators from one cache directory: main fails 7 of 960 compiles (twice, 7 each) with the error above; this patch 0 of 960, with a single complex_arith_<sha1>.h in the directory and no leftover temporaries.
  • tests/test_dtypes.py: 47 passed, 5 skipped.

🤖 Generated with Claude Code

@ggorman ggorman added bug-py compiler CI continuous integration labels Oct 17, 2026
@ggorman
ggorman requested a review from JDBetteridge October 17, 2026 11:34
@codecov

codecov Bot commented Oct 17, 2026 •

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 95.77465% with 3 lines in your changes missing coverage. Please review.
✅ Project coverage is 84.15%. Comparing base (16e6861) to head (9a332fc).

Files with missing lines Patch % Lines
devito/tools/os_helper.py 85.00% 3 Missing ⚠️
Additional details and impacted files
@@            Coverage Diff             @@
##             main    #3071      +/-   ##
==========================================
+ Coverage   84.13%   84.15%   +0.02%     
==========================================
  Files         258      258              
  Lines       57205    57267      +62     
  Branches     4889     4893       +4     
==========================================
+ Hits        48127    48191      +64     
+ Misses       8250     8249       -1     
+ Partials      828      827       -1     
Flag Coverage Δ
pytest-gpu-aomp-amdgpuX 68.40% <17.39%> (-0.03%) ⬇️
pytest-gpu-gcc- 78.96% <85.91%> (+<0.01%) ⬆️
pytest-gpu-icx- 78.88% <85.91%> (+<0.01%) ⬆️
pytest-gpu-nvc-nvidiaX 68.98% <69.56%> (+<0.01%) ⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

Comment thread devito/passes/iet/dtypes.py Outdated

@FabioLuporini FabioLuporini left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't think this is the right way of fixing it, as we would have to change every possible place in which we construct shared header files accordingly, and there are quite a few across OSS and PRO

instead, I think Compiler's save_header (IIRC its name) should be made race-free

@ggorman
ggorman force-pushed the fix-complex-header-race branch from 0f675e4 to 7d711d0 Compare October 19, 2026 11:22
Comment thread devito/arch/compiler.py Outdated
@ggorman
ggorman force-pushed the fix-complex-header-race branch from 7d711d0 to 8cadc60 Compare October 19, 2026 11:36
@ggorman

ggorman commented Oct 19, 2026

Copy link
Copy Markdown
Contributor Author

Fair point, thanks both. Moved the fix into Compiler.save_header: it now writes to a temp file in the JIT dir and renames it into place, so every header saved through it (OSS and PRO: the f16 arithmetic, TMA cache, CUDA graph and alignment headers) gets the same guarantee. _complex_includes just calls save_header now and the sha1 is gone; since def-complex differs between C++ and CUDA/HIP, the header is keyed on the language name instead, the same way <lang>_f16_arith.h is in PRO (openmp_complex_arith.h, openacc_complex_arith.h, cuda_complex_arith.h, ...). The test is now a concurrency test of save_header itself in test_arch.py (fails on main, passes here), plus a check that the language-specific header is included with the full definitions.

@ggorman
ggorman requested a review from FabioLuporini October 19, 2026 16:49

@mloubout mloubout left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Needs rebase then gtg

The JIT dir is shared by all processes compiling as the same user, e.g.
the workers of a parallel test run, and `save_header` truncated the
header before rewriting it, so a concurrent compiler could read a
truncated `complex_arith.h` and fail. Write the header to a private
temporary file and rename it into place instead, and route the complex
arithmetic header through `save_header` so it benefits too. Its name is
now keyed on the language, like `<lang>_f16_arith.h`, as the definitions
differ across languages.
`Compiler.save_header` now delegates to a generic `devito.tools.atomic_write`,
so other writers to shared files can get the same guarantee. The helper also
removes its temporary file if the write fails.

@FabioLuporini FabioLuporini left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

GTG

@ggorman
ggorman force-pushed the fix-complex-header-race branch from 8b3e853 to 9a332fc Compare October 21, 2026 08:51
@ggorman
ggorman merged commit 078bfe3 into main Oct 21, 2026
42 checks passed
@ggorman
ggorman deleted the fix-complex-header-race branch October 21, 2026 10:15
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

bug-py CI continuous integration compiler

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants