Skip to content

refactor openmp task-based internal use + add policy interface to factorization - #86

Merged
PierreMarchand20 merged 7 commits into
mainfrom
improve_task_based_parallelism
Oct 9, 2026
Merged

PierreMarchand20 merged 7 commits into
mainfrom
improve_task_based_parallelism

Conversation

@PierreMarchand20

@PierreMarchand20 PierreMarchand20 commented Jul 21, 2026 •

Copy link
Copy Markdown
Member

Summary

Improve the OpenMP task-based HMatrix build and factorizations: an execution policy interface for the LU, Cholesky and LDLt factorizations, a task-based LDLt, and several fixes to omp_task_policy, including a use of freed blocks in task-based factorizations with unbalanced cluster trees.

Changes

Execution policies for factorizations

  • lu_factorization, cholesky_factorization and ldlt_factorization accept seq, par and omp_task_policy. The overloads without a policy use seq, so existing callers (including the DDM local solvers) are unchanged.
  • omp_task_policy stores the L0 used as task dependencies in HMatrixTaskDependencies. A factorization keeps it if it is a cut of the block tree of the factorized HMatrix (e.g. the L0 set by its build, or one assigned directly), and recomputes it otherwise.
  • Called in a parallel region, build and factorizations only create their tasks, so that successive calls overlap through their dependencies. The contract is documented in omp_task_policy.
  • When L0 is reduced to the root, they run sequentially.
  • The default L0 size stays at 64 nodes (TODO: adaptive default, see follow-ups).

Task-based LDLt

  • task_based_ldlt_factorization, following sequential_ldlt_factorization. The copies W(q) used by the Schur complement are created with their block tree, filled by tasks, and freed per pivot step.
  • task_based_internal_triangular_ldlt_hmatrix_hmatrix_solve, sharing the recursion of the task-based triangular solve.
  • task_based_internal_apply_ldlt_diagonal.
  • HMatrix::set_low_rank_data, counterpart of set_dense_data.

Python

  • Remove the HTOOL_WITH_PYTHON_INTERFACE macro, so that threading is available through the Python interface.

Fixes

  • Use of freed blocks in task-based factorizations with unbalanced cluster trees (e.g. GeometricSplitting on unevenly distributed points), also affecting task-based Cholesky on main. The triangular HMatrix-HMatrix solves densified a hierarchical right-hand side whose triangular matrix is a dense block, inside a task, while the thread creating the tasks still used its children. They now solve its children with the dense block, which keeps the block structure and the compression. Densification only remains for block trees that are not consistent.
  • omp_task_policy factorizations no longer reuse the L0 of another HMatrix. In a parallel region, pending tasks are completed before L0 is recomputed.
  • Task-based build tasks only use values captured when they are created, so another build with the same builder or policy can start before they complete.
  • Workaround for GCC 12.2 (Debian 12), which destroys a shared_ptr data member of a class template given to firstprivate in a task while tasks still use it. Fixed in GCC 12.3.

API changes

  • Added: policy overload of ldlt_factorization, HMatrix::set_low_rank_data.
  • Removed: HMatrixTreeBuilder::get_false_positive. The number of false positives is now computed from the HMatrix information (get_hmatrix_information, get_distributed_hmatrix_information): number of admissible blocks stored by the build minus number of low-rank blocks. This makes it right for task-based builds too.
  • Removed: HTOOL_WITH_PYTHON_INTERFACE macro.

Tests

  • Task-based factorization tests: LU, Cholesky and LDLt with omp_task_policy (double, complex symmetric and Hermitian, both triangles, full and symmetric storage), including the trivial n1 = 10 case (root made of dense blocks).
  • Full build→factorization tests in a parallel region: policy reused for another HMatrix, two builds in a row, L0 reduced to the root, L0 assigned directly or recomputed after a build, LDLt, false positives.
  • Task-based LDLt triangular solve.
  • Factorizations with an unbalanced cluster tree (double and complex). They catch the use of freed blocks under AddressSanitizer.

CI

  • Leak detection is disabled for the task-based factorization tests (ASAN_OPTIONS=detect_leaks=0), as it already was for the task-based build tests. LeakSanitizer intermittently reports allocations made by libgomp in GOMP_task for tasks with dependencies, with any GCC from 11 to 15 and without htool code involved. Address errors are still checked, and leaks are still checked by the sequential tests.

Follow-ups

  • Python bindings for ldlt_factorization with and without a policy (htool_python).
  • Bump docs/htool in htool_documentation, whose C++ pages already document LDLt and omp_task_policy.
  • omp_task_policy for the HMatrix-vector and HMatrix-matrix products, task-based solves, and a policy interface for the DDM local solvers.
  • Adaptive default for max_number_of_nodes. First measurements give about 256 nodes as the best value for LDLt and Cholesky on 8 threads.

🤖 Generated with Claude Code

@codecov

codecov Bot commented Jul 21, 2026 •

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 97.47368% with 12 lines in your changes missing coverage. Please review.
✅ Project coverage is 98.26%. Comparing base (658d8f0) to head (f56d3f0).
⚠️ Report is 69 commits behind head on main.

Files with missing lines Patch % Lines
...matrix/linalg/triangular_hmatrix_hmatrix_solve.hpp 58.33% 5 Missing ⚠️
...x/linalg/triangular_ldlt_hmatrix_hmatrix_solve.hpp 58.33% 5 Missing ⚠️
...ed/test_task_based_hmatrix_full_complex_double.cpp 93.75% 1 Missing ⚠️
...task_based/test_task_based_hmatrix_full_double.cpp 93.33% 1 Missing ⚠️
Additional details and impacted files
@@            Coverage Diff             @@
##             main      #86      +/-   ##
==========================================
+ Coverage   97.19%   98.26%   +1.06%     
==========================================
  Files          80      191     +111     
  Lines        5132    11457    +6325     
==========================================
+ Hits         4988    11258    +6270     
- Misses        144      199      +55     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@PierreMarchand20
PierreMarchand20 force-pushed the improve_task_based_parallelism branch from c62ada4 to e292614 Compare July 21, 2026 16:22
@PierreMarchand20
PierreMarchand20 force-pushed the improve_task_based_parallelism branch from e292614 to 2469205 Compare October 6, 2026 16:29
@PierreMarchand20
PierreMarchand20 force-pushed the improve_task_based_parallelism branch 5 times, most recently from fd9ae43 to 7fc9a34 Compare October 8, 2026 08:44
PierreMarchand20 and others added 5 commits October 8, 2026 12:30
- Recompute the L0 cached in omp_task_policy when it does not belong to
  the factorized HMatrix (HMatrixTaskDependencies::is_L0_of) or when
  max_number_of_nodes changed.
- Build tasks only use values captured when they are created (L0 node,
  leaves to compute, low-rank generator), so that another build with the
  same builder or policy can start before they complete; atomic update of
  the number of false positives.
- Run the build and factorizations sequentially when L0 is reduced to the
  root.
- Document omp_task_policy inside and outside a parallel region, and
  factor the creation of the parallel region into run_in_task_region.
- Restore the default L0 size to 64 and remove the debug L0 output of the
  build test.
- Tests: policy reused for another HMatrix, two builds in a row, L0
  reduced to the root, Cholesky built with the policy, trivial n1=10 case.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- ldlt_factorization with an execution policy (seq, par,
  omp_task_policy), the existing overload using seq.
- task_based_ldlt_factorization, following sequential_ldlt_factorization:
  the copies W(q) used by the Schur complement are created with their
  block tree and filled by tasks, and freed by a task once the products
  using them complete.
- task_based_internal_triangular_ldlt_hmatrix_hmatrix_solve, sharing the
  recursion of the task-based triangular solve with another leaf solve.
- task_based_internal_apply_ldlt_diagonal, with one task per node of L0.
- HMatrix::set_low_rank_data, counterpart of set_dense_data.
- Tests: task-based LDLt triangular solve, LDLt factorization with seq,
  par and omp_task_policy, build and LDLt in a parallel region.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
As for the task-based build tests, LeakSanitizer reports allocations made
by libgomp when creating tasks with dependencies (136-byte objects, most
likely dependency hash tables) as leaked, intermittently. It happens with
any GCC from 11 to 15, without htool code involved, and in CI it was
triggered by the trivial n1=10 case. Address errors are still checked, and
leaks are still checked by the sequential factorization tests.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
When A is a dense diagonal block and B is hierarchical, the triangular
HMatrix-HMatrix solves densified B. With an unbalanced cluster tree (a leaf
cluster next to a non-leaf one, e.g. GeometricSplitting on unevenly
distributed points), a consistent block tree has such blocks, and in the
task-based factorizations the densification happened in a task while the
thread creating the tasks still used the children of B: heap use after
free in task-based Cholesky and LDLt.

Since the cluster of a dense diagonal block of a consistent block tree is
a leaf, the children of B keep it and are now solved with A, which also
keeps the compression of B. B is only densified for block trees that are
not consistent. Test LU, Cholesky and LDLt with omp_task_policy on an
unbalanced cluster tree, for double and complex double.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The builds counted false positives in the HMatrixTreeBuilder, which the
task-based build updated from its tasks: in a parallel region, they could
still run during a following build with the same builder, which reset the
count.

The builds now store the number of admissible blocks in the HMatrix
information, known before the blocks are computed, and
get_hmatrix_information and get_distributed_hmatrix_information deduce the
number of false positives from it and from the number of low-rank blocks.
HMatrixTreeBuilder::get_false_positive is removed.

Also document that the HMatrixTreeBuilder must stay alive until the tasks
of a build from a VirtualGenerator complete, since it holds the internal
generator wrapping it.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@PierreMarchand20
PierreMarchand20 force-pushed the improve_task_based_parallelism branch from 7fc9a34 to f56d3f0 Compare October 9, 2026 08:33
@PierreMarchand20
PierreMarchand20 merged commit f4ffcf1 into main Oct 9, 2026
11 checks passed
@PierreMarchand20
PierreMarchand20 deleted the improve_task_based_parallelism branch October 9, 2026 10:29
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant