Major performance improvement for re-ordering of large graphs

[garth-wells]

The GPS algorithm is slow for large meshes because it is close to O(n^2). The PR used RCM as default, which is close to O(n). No detectable performance difference for SpMV or assembly.

On high-end GPU systems with one MPI rank per device, meshes can be large and re-ordering was a significant cost.

Fixes #3282.

Claude used in PR.

[chrisrichardson]

Add AI assistance note, as per CONTRIBUTING.md.

[jorgensd]

Could you do a performance check for the case in: #3282
as it would be great to resolve this issue.

[garth-wells]

Could you do a performance check for the case in: #3282 as it would be great to resolve this issue.

This PR gives a major speed up over the reported issue (orders of magnitude). Other changes could be made to address #3282, but I think this is good enough for now.

[jorgensd]

This PR gives a major speed up over the reported issue (orders of magnitude). Other changes could be made to address #3282, but I think this is good enough for now.

I ran the code

from mpi4py import MPI

from dolfinx import mesh, fem, la, log
from basix.ufl import element, mixed_element
import time

# log.set_log_level(log.LogLevel.INFO)
N = 5
P = 4

# Create square quad mesh
start = time.perf_counter()
domain = mesh.create_unit_cube(MPI.COMM_WORLD, N, N, N, mesh.CellType.hexahedron)
Ve = element("Lagrange", domain.basix_cell(), P, shape=(domain.geometry.dim,))
Qe = element("Lagrange", domain.basix_cell(), P)
W_el = mixed_element([Ve, Qe])
W = fem.functionspace(domain, W_el)
print(f"Meshing and creating mixed space: {time.perf_counter()-start}")
domain.comm.Barrier()
start2 = time.perf_counter()
V, WV_map = W.sub(0).collapse()
print(f"Collapsing V: {time.perf_counter()-start2}")
domain.comm.Barrier()
start3 = time.perf_counter()

Q, WQ_map = W.sub(1).collapse()
print(f"Collapsing Q: {time.perf_counter()-start3}")

with main and your branch:

Serial

main

Meshing and creating mixed space: 0.004145166999478533
Collapsing V: 4.375436751999587
Collapsing Q: 0.0006516250005006441

Your branch

Meshing and creating mixed space: 0.007172040999648743
Collapsing V: 0.021545334000620642
Collapsing Q: 0.0010807919998114812

3 procs

main

Meshing and creating mixed space: 0.007273207999787701
Collapsing V: 0.2865862919998108
Collapsing Q: 0.0013121250003678142
Meshing and creating mixed space: 0.008402624999689579
Collapsing V: 0.28659158300069976
Collapsing Q: 0.0013122919999659644
Meshing and creating mixed space: 0.020776042000761663
Collapsing V: 0.28659158400023443
Collapsing Q: 0.0013362499994400423

your branch

Meshing and creating mixed space: 0.014550583000527695
Collapsing V: 0.006663291999757348
Collapsing Q: 0.0013109999999869615
Meshing and creating mixed space: 0.012623084000551898
Collapsing V: 0.006658999999672233
Collapsing Q: 0.0013056249999863212
Meshing and creating mixed space: 0.021382124999945518
Collapsing V: 0.006667582999398292
Collapsing Q: 0.0013128749997122213

which i think is a sufficient improvement, and we can close #3282.
However, I note an increase in mesh creation and mixed space creation.

[garth-wells]

Those measurements for mesh creation are in the noise - the mesh is too small.

The RCM algorithm is roughly O(n) and GPS roughly O(n^2), where n is number of cells.

[jorgensd]

Those measurements for mesh creation are in the noise - the mesh is too small.

The RCM algorithm is roughly O(n) and GPS roughly O(n^2), where n is number of cells.

Running the same code on a 100^3 cube with 3 processes yields:
main

Meshing and creating mixed space: 2.9723226690002775
Meshing and creating mixed space: 2.9585314599989943
Meshing and creating mixed space: 3.022344750999764

this branch

Meshing and creating mixed space: 3.0290222099993116
Meshing and creating mixed space: 3.126498460000221
Meshing and creating mixed space: 3.015944501999911

which I agree are fairly similar timings.

My main goal with the comparison above was to figure out if this resolved #3282 or not.