Repository navigation
compiler: Write the complex arithmetic header atomically - #3071
Conversation
Codecov Report❌ Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## main #3071 +/- ##
==========================================
+ Coverage 84.13% 84.15% +0.02%
==========================================
Files 258 258
Lines 57205 57267 +62
Branches 4889 4893 +4
==========================================
+ Hits 48127 48191 +64
+ Misses 8250 8249 -1
+ Partials 828 827 -1
Flags with carried forward coverage won't be shown. Click here to find out more. ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
FabioLuporini
left a comment
There was a problem hiding this comment.
I don't think this is the right way of fixing it, as we would have to change every possible place in which we construct shared header files accordingly, and there are quite a few across OSS and PRO
instead, I think Compiler's save_header (IIRC its name) should be made race-free
0f675e4 to
7d711d0
Compare
7d711d0 to
8cadc60
Compare
|
Fair point, thanks both. Moved the fix into |
The JIT dir is shared by all processes compiling as the same user, e.g. the workers of a parallel test run, and `save_header` truncated the header before rewriting it, so a concurrent compiler could read a truncated `complex_arith.h` and fail. Write the header to a private temporary file and rename it into place instead, and route the complex arithmetic header through `save_header` so it benefits too. Its name is now keyed on the language, like `<lang>_f16_arith.h`, as the definitions differ across languages.
`Compiler.save_header` now delegates to a generic `devito.tools.atomic_write`, so other writers to shared files can get the same guarantee. The helper also removes its temporary file if the write fails.
8b3e853 to
9a332fc
Compare
_complex_includeswrites the extra complex arithmetic definitions a C++ language needs (def-complex) tocomplex_arith.hin the JIT cache directory, withopen(..., 'w'), on every lowering of an Operator that uses a complex dtype. The directory is shared by every process compiling as the same user, and'w'truncates the file before writing it, so withpytest -n autoone worker can truncate the header while another's compiler is reading it. The compile then fails on the first complex expression, e.g.no match for 'operator*' (operand types are 'int' and 'const std::complex<float>'); in DevitoPRO's CUDA backend the same race shows asuser-defined literal operator not foundfor1.0_if.Changes:
complex_arith_<sha1>.h), written to a private temporary file and renamed into place, and skipped when it already exists. A reader therefore sees either no file or a complete one, and two languages with different definitions no longer overwrite each other.test_complex_arith_headerchecks that two Operators share one header whose content is exactlydef-complex.Verification on main d7f53a3:
language='CXX'Operators from one cache directory: main fails 7 of 960 compiles (twice, 7 each) with the error above; this patch 0 of 960, with a singlecomplex_arith_<sha1>.hin the directory and no leftover temporaries.tests/test_dtypes.py: 47 passed, 5 skipped.🤖 Generated with Claude Code