ILE: make the --internal-use-lnL backends run, and return a lnZ that is right

[oshaughnessy-junior]

What this is

Three defects reported from the ILE-GPU-Paper demo on an A100 (cupy 12.0.0, IGWN CVMFS
python), all reproducing on an unmodified checkout, plus three more found while checking
that the fixes produced numbers rather than just exit code 0. Together they meant a
requested .dgrid marginal-distance-grid came out on AV and GMM and silently did not on
anything else.

The defects

1. mcsamplerGPU.integrate_log: host arrays leaking into device expressions.
identity_convert is cupy.asnumpy and self.xpy.max returns a 0-d device array, so
max(maxval, identity_convert(...)) made maxval a 0-d numpy.ndarray, which the next line
mixes into a device expression — TypeError: Unsupported type <class 'numpy.ndarray'>. This
took down every --internal-use-lnL run on adaptive_cartesian_gpu, the default sampler.
maxlnL has the identical bug one branch up, biting the verbose per-iteration report and the
deltalnL cut, so fixing maxval alone left the demo failing at the same TypeError six lines
later. Both coerced to python floats, which both backends accept. Invisible on CPU, where
numpy.max returns a numpy scalar and cupy accepts those.

2. mcsampler.MCSampler had no identity_convert. Every other sampler defines the
host/device converters in __init__; the driver only assigns them onto the samplers it
recognizes by name, so anything landing on the "original sampler" fallback (e.g.
adaptive_cartesian) reached float(sampler.identity_convert(neff)) after a completed
integration and died with AttributeError, discarding the work.

3. Which exposed a silent wrong answer behind it. The post-processing asked
if not(opts.internal_use_lnL): log_res = numpy.log(res) — reading the option where it had
to read the convention the sampler was actually handed. They differ for exactly one pairing:
adaptive_cartesian is in ok_lnL_methods so the option is accepted, but mcsampler.py has
no use_lnL handling and returns a linear integral. The logarithm was skipped, and the
demo reported lnZ = 1.3e29 (= e^67) with sigma = inf, with the .dgrid inheriting it.
Keyed off return_lnL instead — the predicate the driver already derives, the integrand
already uses, and whose divergence from the option the driver already documents for
_rvs['integrand']. Same substitution in the replica arm and the zero-likelihood stand-in.
The two agree for GMM / adaptive_cartesian_gpu / AV / portfolio, so nothing else moves.

4. mcsamplerPortfolio had no prior_pdf. It delegates the priors to its members and kept
no copy, so --sampler-method portfolio --export-marginal-distance-grid completed the
integration and then died in the exporter on sampler.prior_pdf["distance"]. Mirrored from
member 0, the designated full-support member, alongside llim/rlim.

5. The .dgrid gate also required --internal-use-lnL, a flag that decides nothing this
exporter reads: its weights go through ln_weights_for_posterior with the record's own
convention, and log_res now comes through return_lnL. All it did was make
--export-marginal-distance-grid silently do nothing unless a second, unrelated flag was set
too. Dropped from the gate.

The reported --sampler-method portfolio with no member (TypeError: 'NoneType' object is not iterable) was already fixed in-tree by a134af3 — the reporting worktree predated it. It
is pinned here so it cannot come back.

Deliberately not fixed

maxval in integrate_log is a LOG-scale accumulator initialized at 0, a linear-weight
idiom, so eff_samp = sum(w)/max(w) is floored whenever the largest weight is < 1 (it
under-reports n_eff by 1/max·w and keeps drawing to nmax). Real GW lnL is large and positive so
it does not bite in production, and the identical initializer is copied verbatim into
mcsamplerPortfolio and mcsamplerNFlow — changing one would split n_eff, and hence run
lengths, across backends mid-campaign. Recorded in the code where it is, so the next reader
meets a note rather than a landmine.

Measurements

ldas-pcdev12 (A100, cupy 12.0.0), the demo as reported, same GPU and same seed, this branch
against its own merge-base. Runs are reproducible on this host (3/3 identical repeats), so
the identical columns below are identity, not noise.

config	merge-base	this branch
adaptive_cartesian_gpu	lnZ 58.3091, n_eff 3.4	identical, + .dgrid
adaptive_cartesian_gpu + lnL	TypeError, no output	lnZ 58.4381, .dgrid
GMM	lnZ 57.9598, n_eff 2.1	identical, + .dgrid
GMM + lnL	lnZ 57.9598 + .dgrid	identical incl. .dgrid
adaptive_cartesian + lnL	AttributeError, no output	lnZ 58.5209, .dgrid
portfolio AV,GMM + lnL	AttributeError (prior_pdf)	lnZ 58.1610 identical, + .dgrid
AV + lnL	lnZ 59.0983 + .dgrid	identical incl. .dgrid

Every backend that already produced a number produces the same one. Where a backend can now
export a .dgrid from both record conventions, the two agree — adaptive_cartesian_gpu
peak 63.995 (linear record) vs 63.845 (lnL record); GMM 102.020 vs 102.550 — which is the check
that the exporter's linear branch is right, not merely reachable.

(The demo grid was regenerated mid-session by another branch, so absolute lnZ here is ~58 where
older logs of the same demo show ~67. Both arms of every row above ran on the same grid.)

Tests

test/test_ile_lnL_backend_defects.py, 34 tests, wired into the CI job that already runs the
sampler regression suites. Every fix was verified by reverting it and watching the matching
test fail — the first version of the defect-1 test passed with the bug present and was
replaced. The device-typing tests run twice: against a cupy stand-in that enforces cupy's real
operand rule (CPU lane), and against real cupy where a GPU is present. Where practical the
tests execute the driver's own source — the _neff_val line, the portfolio flattening
comprehension, the return_lnL derivation, the log_res block and the .dgrid gate are
compiled out of the file and run, rather than copied.

Also green on this branch: test_av_empty_live_volume, test_l0_rescue_seed,
test_seq_warmstart_seed, test_fairdraw_double_weighting, test_portfolio_fairdraw_backend,
test_rvs_record, test_limit_cosine_samplers, test_distance_grid, and both
expensive_before_merging audits (audit_rvs_fairdraw, audit_backend_contracts).

Adversarial review

Run before un-drafting. One finding, and it was in the tests, not the fix: _derive_return_lnL
executes the driver's own return_lnL derivation but stops short of a second, unconditional
if opts.sampler_method == "AV": return_lnL=True further down, so an assertion about
(AV, internal_use_lnL=False) was passing against an incomplete model. Unreachable today only
because the AV branch forces the option on — and the test would have gone on passing if that
forcing were removed, which is exactly when AV would join adaptive_cartesian as a place the
two predicates part company. Replaced with a test that pins both halves. Fixed in 61728ce.

Cleared without a finding, each checked rather than assumed:

• self.prior_pdf on the portfolio is written only in __init__ and add_parameter — the
  pin/restore path never touches it, so a pinned pass cannot leave a delta function in the
  prior the .dgrid exporter divides out.
• All six backends populate prior_pdf in add_parameter, so widening the .dgrid gate
  cannot AttributeError on any sampler that now reaches it.
• SamplerOutputMixin (new on the base from Give the retained set and the export resample separate names #94) defines only samples/set_samples — no
  collision with the converters added here.
• return_lnL is never bound inside analyze_event (AST-checked), so the new reads are global
  reads, not UnboundLocalError; and analyze_event has exactly one call site, well after the
  module-level derivation.
• zero_like returns zeros and unit_like returns ones, so the new if return_lnL: mapping is
  the right way round.
• float() on -inf and on nan preserves the previous max()/isinf behaviour exactly.

Each of the six fixes was reverted individually against the final tree; every one makes a
specific test fail, and none of them makes only a source-scan test fail.

🤖 Generated with Claude Code

[interhost-cross-review-canary-v2]

Independent automated review completed at the recorded exact commit. Detailed findings were withheld from public output by the private-context egress policy and require private human declassification.

[interhost-cross-review-canary-v2]

Independent automated review completed at the recorded exact commit. Detailed findings were withheld from public output by the private-context egress policy and require private human declassification.