Fix stochastic contact-ramp livelock: uninitialized contact-point device memory NaNs the angular channel and degenerates CD into an all-pairs sweep (fixes #71)#72
Open
Ward-Vandepitte wants to merge 1 commit into
Conversation
memory poisons angular acceleration (fixes projectchrono#71) The stochastic mid-run stall where kT pegs the GPU inside one contact detection pass (issue projectchrono#71) is not a synchronization bug. The causal chain is: 1. DualArray::resize(n, val) fills host values only; the grown device range keeps whatever the allocator recycled. contactPointGeometryA/B are grown this way in contactEventArraysResize and allocateGPUArrays, are never per-step cleared (prepareForceArrays only clears the two force arrays), and calculateContactForces writes them only for entries whose ContactType is a real contact. Pairs registered by kT's contact margin with no physical overlap -- the majority of the list -- therefore keep uninitialized contact points for their lifetime. 2. forceToAngAcc computes cross(cntPnt, F) / MOI for ALL entries. cross(finite_garbage, 0) is 0, silently fine; cross(NaN/inf, 0) is NaN, so one NaN-holding margin pair injects NaN into its owner's angular acceleration every step. The linear collect never reads cntPnt, so velocities stay finite and the max-velocity error check never fires. 3. NaN angular acceleration propagates to angular velocity and the orientation quaternion within one step, giving NaN sphere positions. 4. The range clamping in getNumberOfBinsEachSphereTouches is written as ternary comparisons; for a NaN position every comparison is false, so the sphere is assigned bins [0, nb-1] in all three dimensions -- the ENTIRE grid. Every affected sphere then meets every other sphere in every bin and the n(n-1)/2 per-bin sweep in getNumberOfSphereContactsEachBin faces an all-pairs workload (tens of billions of pair tests in one launch), which presents as a livelock. Fix, three layers: - contactEventArraysResize and allocateGPUArrays zero the device ranges of contactPointGeometryA/B after resizing (root cause). The two force arrays need no such treatment since prepareForceArrays clears them every step. - forceToAngAcc returns zero early when F + torque_inForceForm is zero, so never-written contact point slots are unconditionally harmless; as these entries are the majority of the list, this also skips their MOI and quaternion loads. - getNumberOfBinsEachSphereTouches errors out on a non-finite position or radius span instead of silently claiming the whole grid, so any future NaN state surfaces as a clean error rather than a hang. This is an intentional new fail-fast: a non-finite sphere position is never a legitimate simulation state. Validated on the issue projectchrono#71 reproduction (600 clumps / 1800 spheres, stock frictional Hertzian): stall rate went from 27%+ per run to 0/21 (17 truncated + 4 full-length runs); five physics regression pins (normal stiffness, Coulomb friction, restitution, wall friction, sphere bridge) unchanged; 24/24 production-shaped settling runs clean under a no-retry watchdog. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Body
Summary
The stochastic "kT wedged at 100% GPU" livelock reported in #71 is not a
synchronization bug. It is a finite work explosion in contact detection, triggered by
NaN owner quaternions, which in turn originate from uninitialized device memory in
the contact recording arrays. This PR fixes the root cause and adds two defensive
layers.
Causal chain (each link verified with env-gated host-side probes on captured
wedges; ~90 instrumented repro runs)
contactEventArraysResizeresizescontactForces/contactTorque_convToForce/contactPointGeometryA/BwithDEME_DUAL_ARRAY_RESIZE(..., make_float3(0)).DualArray::resize(n, val)fillshost values only (its own comment says so);
resizeDeviceraw-allocates andcopies the old prefix — the grown device tail is recycled, uninitialized memory.
prepareForceArraysclears the two force arrays every step, butcontactPointGeometryA/Bare never cleared, andcalculateContactForceswritesthem only for
ContactType != NOT_A_CONTACT. Margin-only pairs (the majority ofthe pair list, by design of the CD margin) therefore keep garbage contact points
for their entire lifetime.
forceToAngAcccomputescross(cntPnt, (F + torque_inForceForm) * mod) / MOIover ALL contact entries. For a margin-only entry the force is zero, and
cross(finite_garbage, 0) = 0— silently fine. Butcross(NaN/inf, 0) = NaN,so one NaN-holding persistent margin pair injects NaN into its owner's angular
acceleration every step. The linear collect (
acc = F/mass) never readscntPnt, which is why only the angular channel dies (observed: velocities stayfinite through the whole event, so
SetErrorOutVelocitynever fires).(0.5*h*omgBar, 1)and normalizes) -> NaN sphere positions. The clamp ingetNumberOfBinsEachSphereTouchesis NaN-blind (every ternary comparison isfalse -> lo=0, hi=nb-1 in all three dims), so each NaN sphere claims the ENTIRE
bin grid. Measured at the wedge: 73,728,000 sphere-bin pairs = 1800 spheres x
40960 bins exactly; maxGeoInBin = all spheres.
getNumberOfSphereContactsEachBin) then faces~6.6e10 pair tests in one launch. One captured instance completed in 56.66 s and
the sibling populate kernel hung next; ordinarily the launch simply never returns
within any reasonable watchdog window. GPU pegged at 100%, kT inside one kernel:
the reported livelock signature.
Notes that may help maintainers reconcile this with prior reports: the failure is
concentrated at the contact ramp because that is when the arrays grow; it is
stochastic because it depends on the recycled memory content (usually finite garbage
= a silent zero-effect; occasionally NaN/inf); and it is config-independent — a
pre-registered campaign falsified tuner on/off, drift clamps, fixed expand factor,
CD frequency (only CDUpdateFreq=1 suppressed the rate, by shrinking the margin pair
population), and a CUDA 12.6/12.8 toolkit swap. When the garbage lands in the force
tail instead, the run dies loudly at SetErrorOutVelocity — likely some of the
"diverged for no reason" reports are the same defect's other face.
Fix (three layers)
contactEventArraysResizeANDallocateGPUArraysnowcudaMemsetthedevice ranges of
contactPointGeometryA/Bafter resizing (the second site isreachable from every
Initialize()and mid-runUpdateClumps(), and its garbageis also user-visible through
GetContactDetailedInfo()default output). The twoforce arrays need no such treatment:
prepareForceArraysclears them every step.The deeper hazard is that
DualArray::resize(n, val)appliesvalto hostmemory only -- happy to follow up with a patch there instead/as well if you
prefer that semantics change.
forceToAngAccreturns zero early whenF + torque_inForceFormisexactly zero, so never-written
cntPntslots are unconditionally harmless. Theseentries are the majority of the list, so this also skips their MOI and quaternion
loads (and removes a silent finite-garbage torque perturbation that nobody would
otherwise notice).
getNumberOfBinsEachSphereTouchesaborts loudly (same style as theerrOutBinSphNumcheck) on a non-finite position or radius span instead ofsilently claiming the whole grid. This is an INTENTIONAL new fail-fast: a
non-finite sphere position is never a legitimate simulation state, so no opt-out
is provided -- any future NaN state becomes a clean error rather than a phantom
hang.
Validation
On the #71 reproduction scene (600 clumps / 1800 spheres, frictional Hertzian,
stock settings; stochastic livelock rate 27%+ per truncated run, 95% CI LB 0.269
over the campaign):
4/4 full-length runs clean.
friction, restitution, wall friction) pass on the fixed build.
17/17 truncated runs clean (arm prshape_final, 2026-07-22).