portfolio: high-SNR L0 auto-rescue (landed 4/9 -> 8/9), save-samples export fix, extrinsic-stability tooling - #34
Merged
Conversation
…1) for condor follow-up The multi-event follow-up runs under condor with the container supplied by MY.SingularityImage, so the harness must build the FULL production args (gwsignal / SEOBNRv5PHM / cosmo prior) while exec'ing python DIRECTLY rather than nesting singularity inside the job. container= now selects the ARGS and wrap= selects whether to nest singularity; NO_SINGULARITY=1 sets wrap=False. Condor wiring lives outside the repo (~/rift_mev_followup/) and deliberately pins ONE image with a require_gpus capability range instead of the container family's $$/ifThenElse per-machine selection: per CITLOCAL_BREADCRUMB_gpus_capability_undefined_holds.md that expression HOLDS on ~45% of CIT GPU slots because it reads machine-level TARGET.GPUs_Capability (frequently undefined), whereas require_gpus matches the per-GPU Capability. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…es NOT generalise 4 typical O4 events, warm, --n-eff 30, real SEOBNRv5PHM (cuda128 container on idle Blackwell nodes): event no-clip lnZ(neff) clip lnZ(neff) dlnZ neff_ratio S231026ab 17.54 (28.9) 17.49 (29.8) -0.05 x1.03 S240426s 29.74 (31.2) 29.56 (30.3) -0.18 x0.97 S240513ei 83.76 (3.1) 83.83 (1.3) +0.07 x0.42 S240703ad 41.89 (3.3) 42.27 (5.0) +0.38 x1.53 Clipping's dramatic S250114ax result (2.3x at the production target) is SPECIFIC to that event's extreme heavy-tailed pathology. On typical events it is a near-noop; on the two under-converged hard events it is a wash within the n_eff~1-5 scatter. ln Z agrees everywhere (|dlnZ| <= 0.38, MC error) -- the portfolio replicates the AV integral with or without clipping. Confirms opt-in/default-off was the right call. Closes the study's last open question. Also confirmed en route: the cuda128/cc90-120 container runs SEOBNRv5PHM+cupy on Blackwell (CC 12.0), matching the A100/cuda118 result -- the path that let this run bypass the saturated A100 condor pool. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
… floor on GPU Resolves a cross-box reproduction gap reported from cardassia on PR #26 and answers its ask. ROOT CAUSE of the gap: TARGET-POINT MISMATCH, not CPU-vs-GPU and not the warm start. cardassia integrated the loud on-source coinc point (lnLmax~3020); the reference matrix and all quoted numbers use the iteration-0 worker point (overlap-grid-0.xml.gz --event 0, lnLmax~1212). Their own COLD run (n_eff=1.00 vs reference ~6 / CIT av_cold 3.7) proves it: cold uses no seed, so a cold discrepancy can only come from the likelihood. Seed volumes agree to 3 digits across boxes (0.056/0.649 vs 0.0567/0.647), so there is no backend divergence in the warm start. MEASURED (reference point, GPU, rescue OFF, --n-chunk 10000, nmax 4M), varying only the coverage floor: run cover rescue Neff>=5 >=10 >=20 >=50 >=100 final lnZ av_warm(ref) 0.05 ON 0.695M 1.374M 1.691M 2.268M 3.638M 100.2 1191.79 cf_cover005 0.05 OFF 0.837M 1.146M 1.546M 2.234M -- 64.2 1192.16 cf_cover050 0.5 OFF 1.350M -- -- -- -- 8.1 1193.83 => The warm-start demo does NOT survive the bias-safe floor (8.1 @4m vs the 100 @3.35M headline). Shape-gate verdict: n_eff 8.1 << 100 => STARVED, shape untestable at bias-safe coverage even on GPU. Part of the headline was the RESCUE, not the seed (cover 0.05 rescue-OFF gives 64.2, not 100). lnZ 1191.8/1192.2 at cover 0.05 vs 1193.8 at 0.5 -- under-covered runs sit ~2 nats LOW, the direction under-coverage predicts (caveat: 0.5 run n_eff=8.1, ~0.35 nat MC error; repeat before quoting as a bias measurement). Note --sampler-warmstart-cover-frac ALREADY defaults to 0.5; the demo ini's 0.05 was an override into under-covered territory. The harness pins the reference point explicitly and documents that it is deliberately NOT the on-source coinc point, so this mismatch cannot recur silently. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…GMM both fail alone Distinguishes the two problems that were conflated across this study and PR #26: * TRIAL grid point (overlap-grid-0.xml.gz --event 0, m1/m2 28.29/26.69, lnLmax~1212): integrator policy A/B; not the science target. * BEST-FIT on-source point (target_params.xml.gz, m1/m2 37.71/34.03, lnLmax~3040, rho~78): the evaluation that must work for a loud event, and where AV stalls. Measured (GPU, warm, bias-safe cover 0.5): AV alone : n_eff 1.0 @4m (stalls; confirms cardassia CPU result on GPU) GMM alone (adaptive) : NaN chunk-1 (no coverage floor -> weights blow up) AV+GMM portfolio adaptive: n_eff 14.7 @4m lnZ 3016 (~15x AV; works where NEITHER works alone) Mechanism: a GMM-peak + AV-coverage event. GMM wraps the correlated peak AV cannot; the "dead" AV member (nan per-chunk n_ess in 381/400 chunks, parked at the 1% floor) is NOT wasted -- its broad warm density enters q_mix = frac_AV q_AV + frac_GMM q_GMM and provides the coverage floor that keeps GMM's weights bounded. Drop AV -> GMM NaNs; drop GMM -> AV stalls. This is the clearest demonstration of why the portfolio exists, and why the never-freeze/allocation machinery matters (it hands the budget to whichever member is working -- here GMM). Harness bench_onsource.sh pins the best-fit point and documents it is deliberately NOT the trial point, so the mismatch that broke the PR #26 reproduction cannot recur. 14.7 is a rescue, not yet target; whether more adaptive GMM coverage pushes higher is under test. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
… (14.7 -> 56.1), unbiased Best-fit point (target_params.xml.gz, lnLmax~3040), AV+GMM portfolio, warm cover 0.5, GPU: GMM BIC cap 8, inflate 1.0 : n_eff 14.7 lnZ 3016.13 GMM BIC cap 16, inflate 1.3 : n_eff 56.1 lnZ 3016.08 <-- 56x standalone AV (which stalls at 1.0) lnZ identical -> real efficiency gain, not a coverage-shortcut bias. Confirms the "GMM event with ADAPTIVE coverage" framing quantitatively: the rescue lever is GMM adaptive coverage (BIC component count + inflation). Remaining gap to production n_eff looks like coverage/budget tuning, not a fundamental barrier. Pushing further (cap 24 / inflate 1.5) under test. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…1.3, over-cranking biases Extending the coverage ladder on the best-fit high-SNR point: cap 8 / infl 1.0 : n_eff 14.7 lnZ 3016.13 cap 16/ infl 1.3 : n_eff 56.1 lnZ 3016.08 <-- sweet spot, 56x AV, unbiased cap 24/ infl 1.5 : n_eff 2.3 lnZ 3009.5 <-- collapse; lnZ -6.6 nats => biasing, not just noise More coverage is NOT better: over-inflation produces an over-wide GMM proposal whose rare enormous weights both crush n_eff and bias lnZ low (the same heavy-tail mode weight clipping targets). So the "GMM event with adaptive coverage" rescue is real but the coverage knobs need tuning to a sweet spot, not maximizing. Isolating whether BIC cap or inflation drives the collapse. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…ples exports n_eff alone hides a wrong/absent posterior SHAPE. This reads the extrinsic cloud saved by --save-samples (weights in the alpha1/loglikelihood column of the sim_inspiral table) and reports, per extrinsic parameter, the WEIGHTED marginal (mean/std) plus global failure signatures: Kish n_eff, max single-sample weight FRACTION (outlier dominance -- the over-broad-GMM failure), and raw count. First use already paid off: a flaky-collapsed high-SNR run (cap16/inflate1.3, seed 1, n_eff 1.3) saved ZERO rows -- after fairdraw pruning nothing survives. So a collapse is not a wrong shape, it is a DEGENERATE/empty posterior, which the shape check makes unmistakable. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…le in this regime
cap16/inflate1.3 n_eff across 5 seeds: {1.2,1.3,1.5,9.2,56.1}, median ~1.5 -- the
56.1 sweet-spot was a lucky draw, not typical. One low-n_eff run on a high-SNR event
is not a posterior; pool many copies (weighted by reliability, not naively) or find a
proposal that reliably lands high n_eff. cap-too-high confirmed as the collapse mode.
Documents that --save-samples exports 0 rows for peaked portfolio runs at 4 layers
(fairdraw / save-P=0.1 / log-weight cumsum bug / XML carries lnL-not-IS-weight);
correct weight-aware shape check is --extrinsic-proposal-output. Bug flagged separately.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…inf guard) Step 2 of the _rvs cleanup in mcsamplerPortfolio.py paired the sorted index with the LOG-weight ln_wt[k] and cumsummed that, so the "cumulative weight" was never a cumulative probability. Any -inf ln_wt entry (rejected/underflowed sample) poisoned the cumsum (cum_sum[-1] = -inf -> nan on normalize), and the cum_sum > deltaP mask kept 0 rows -- 0 samples exported even at n_eff~9. Fix: pair/cumsum the LINEAR weight wt[k] (already computed one line earlier), matching the canonical mcsampler/mcsamplerEnsemble truncation, and map any non-finite log-weight to -inf (-> zero linear weight) with a degenerate fallback that keeps all rows if no finite-weight sample survives. Verified to reproduce canonical linear-weight index selection exactly on finite peaked input and to keep all real samples (dropping only zero-weight rejects) when -inf entries are present. Change is confined to the portfolio Step 2 block; AV/GMM standalone samplers are untouched. Also document (integrate_likelihood_extrinsic_batchmode) that the sparse sim_inspiral XML from --save-samples carries lnL only and must not be reweighted by likelihood for a weighted-posterior/shape check; point users to the ASCII --extrinsic-proposal-output (full log-weight) / --calibration-export-posterior. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> (cherry picked from commit ab28fec)
…tory; add ensemble+pooling harness Correcting my own 3-seed 'cap8 is reliably modest' claim (survivorship bias): 10 cap8 draws span 1->70 with ~40% collapsed to ~1 -- statistically the same bimodal lottery as cap16. correlate-all is strictly worse (collapses 2/3, lnZ 11 nats low). So no single-run config is a posterior on this high-SNR best-fit point; the robust recipe is MANY copies pooled by reliability (the reviewer's original point, now on data). Harness: bench_onsource_ensemble.sh (portfolio must be selected explicitly -- the driver default adaptive_cartesian_gpu is NOT the portfolio/AV) + compare_extrinsic_breadcrumbs.py (weight-correct per-group posterior via --extrinsic-proposal-output; reliability-weighted vs naive pooling; collapse detector = out-of-bounds degenerate GMM). Weight-correct export unblocked by PR #35 (cherry-picked). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…pool by reliability 9-copy cap8 pool (4 landed n_eff 15-41, 5 collapsed 1-3): weight-correct per-group GMM fits show collapsed runs fit 1 mode/group (lost sky ring / dL-i arc / phase-pol) while landed runs fit 3-4 modes -- so low n_eff IS an extrinsic-degeneracy collapse, detectable by n_eff or mode count. Landed copies agree (sky mean ~0.01, incl ~0.02); phase-pol is the one genuinely-broad direction. Reliability-weighted pooling tracks good-only; naive pooling is biased by collapsed copies. Kish-over-n_eff effective #copies = 4.1 of 9. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…k lottery; enable L0 auto-rescue for portfolio Frame-matched rotate-phase + force-adapt-all: landed fraction unchanged (4/9) but landed n_eff up (41-52 vs 15-41). Lottery is robust across all configs -> root cause is AV peak-lock, not proposal/coordinates. Relax --sampler-warmstart-retry-neff (L0 auto-rescue) gate from AV-only to AV|portfolio so a collapsed portfolio run re-seeds from its own peak (frame-safe, cannot bias). Documents the coordinate-flag / warm-start-seed frame-mismatch confound. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…lio (sampler_method is clobbered to GMM) The prior gate (sampler_method in AV|portfolio) never matched for an AV+GMM portfolio: line 1231 clobbers opts.sampler_method to 'GMM' during member setup (to force GMM arg-parsing). Detect the portfolio via opts.sampler_portfolio instead. Verified: rescue now fires for the portfolio (--sampler-warmstart-retry-neff 100 forced a 2nd pass; n_eff 7.2 -> 43.4 after peak re-seed). Same clobber makes the portfolio block at ~1641 dead code (harmless: the GMM branch picks up the gmm_adaptive forwarding) -- flagged for separate cleanup. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…> 8/9 --sampler-warmstart-retry-neff 5 (portfolio-enabled) re-seeds each collapsed run from its own peak: 6/9 rescues fired, 5 -> clean lands (chronic n_eff~1 collapsers s11/s14 now 25-33). The rescue attacks the COLLAPSE RATE (coordinates/adaptation only helped the landers). Validated high-SNR recipe recorded. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
oshaughnessy-junior
changed the title
The portfolio member loop set opts.sampler_method='GMM' to force the GMM-specific argument blocks to run for a GMM member. That silently broke EVERY downstream 'portfolio' test: - the portfolio-only setup block became dead code, - the L0 auto-rescue gate never fired for a portfolio (fixed by workaround in 76e1ea4), - a portfolio with --internal-use-lnL took GMM's branch and got return_lnI, which the portfolio does not consume. Replace with a non-destructive use_gmm_member flag and use_gmm_args = (method=='GMM') or use_gmm_member. Re-key the three blocks that genuinely need GMM handling for a portfolio (GMM arg-parsing, per-iteration distance/inclination reset, --force-reset-all) -- all touch gmm_dict, so they must still fire. Guard the now-live portfolio block against double-specifying gmm_adaptive: the GMM block's per-group DICT is the path every portfolio benchmark exercised, so the portfolio block's scalar cap now only applies when there is no GMM member. L0 gate simplified back to sampler_method in ('AV','portfolio'). Regression (seed 13, cap8+force-adapt-all+rotate-phase+retry-neff 5): identical setup forwarding ('gmm_adaptive': {(4,5):8,(3,2):8,(0,1):8}), rescue fired (cold n_eff 2.9<5, 2 passes), landed n_eff 29.3 lnZ 3017.30 (pre-cleanup 47.0/GPU-nondeterministic, same regime). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
… lnZ-bias claim retracted) The n_eff lottery was found late, so several earlier claims rest on n=1. Explicitly downgrade the cap-ladder numbers (incl. the cap24 'over-cranking biases lnZ' claim) and correlate-all (n=3) to suggestive; keep never-freeze, the multi-event lnZ replication (unbiasedness, structural), the lottery, mode-collapse and the L0 rescue as robust. Record that the sampler_method clobber does NOT invalidate anything (return_lnI is never read by the portfolio; use_lnL was set either way) and flag the open question of whether the lottery is high-SNR-only. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…ime True) Nearest-sample-bin time evaluation injects a superfluous non-smoothness into the extrinsic likelihood surface, so the samplers chase a discretization artifact rather than physics. Cubic interpolation removes it and makes convergence more robust. Requires the maintained NoLoop likelihood (the --vectorized --gpu --force-xpy combo the bench already sets). Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
… fires on degenerate termination
(1) mcsamplerEnsemble.update_sampling_prior filtered NaN-weight samples by REASSIGNING
ln_weights, which is loop-invariant (built once, before the loop over dim_groups). The first
group containing any NaN shrank it (10000 -> 8686); every later group rebuilt temp_samples at
full length but reused the stale shorter weights -> IndexError 'boolean index did not match
indexed array' inside GMM.update/fit. Only reachable when weights contain NaN, i.e. a
degenerate/cold pass -- which is why warm runs never hit it and every COLD AV+GMM portfolio
start on the high-SNR point died at chunk ~8 with no output. Filter into a loop-LOCAL
ln_weights_group instead. Verified PRE-EXISTING: base rift_O4d crashes identically.
(2) L0 auto-rescue treated neff=None as 'skip'. But mcsamplerPortfolio/AV return
(None,None,None,None) from their early-termination branch precisely when the live volume never
finds finite in-volume samples -- the cold, very sharp peak the rescue exists for. Such a pass
still populates _rvs, so the peak-seed is available. Treat neff=None as below-threshold.
(3) The FAILED ANALYSIS handler now prints the traceback; the bare message ('boolean index did
not match...') was not enough to locate a fault inside the sampler stack, and this handler is
often the only record a batch job leaves.
(4) bench_onsource.sh: WT is env-overridable so the same command line can be run against a base
worktree for A/B.
Verified: cold portfolio start now runs to completion (n_eff 2.5 @300K) and the rescue fires.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…py was 11 nats wrong Post-fix cold: 9/9 produce output (was 0/9, the pre-existing crash), 5/9 land. But among landers lnZ spans 12 nats and the HIGHEST-n_eff copy (58) is the outlier, 11 nats below a 4-copy consensus that agrees to 1.4 nats. n_eff measures weight concentration, not coverage: a run that locks onto one narrow region looks confident and is wrong. Recommend consensus/median across copies over n_eff-argmax or n_eff-weighted pooling. Copies are needed to DETECT a bad run, not just find a good one. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Collected asynchronously. Raw measured numbers only (no interpretation): per-config n_eff, lnZ and final AV draw fraction for the cold high-SNR ensemble, plus the lnZ spread per config against the 12.2-nat no-floor baseline. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…not just a floor Measured on the loud-event best-fit point (cold, 4 seeds, judged by lnZ consensus not n_eff): no floor, cap8 : AV share -> 0.0099 (GMM-alone), lnZ spread 12.2 nats, highest-n_eff copy 11 low cap3, no floor : AV share -> 0.0099, lnZ spread 11.9 nats, n_eff 123.6 with lnZ 10 low floor 0.25, cap8 : 3 seeds at floor + 1 RUNAWAY to AV=0.99, lnZ spread 5.1 nats floor 0.25, cap3 : AV pinned at floor on all 4, lnZ spread 2.27 nats Both degeneracies (peaked-member-only and VARAHA-only) strip q_mix of a component and let a missed mode go uncovered -> lnZ silently low while n_eff looks good. A one-sided floor only blocks one of them. --portfolio-varaha-max-frac adds the cap so the share is constrained to a BAND and the mixture stays genuinely mixed by construction. Default 0 = no cap (fully back-compatible). The band applies to the COMBINED VARAHA share, so it composes with MULTIPLE INSTANCES of the same integrator: --sampler-portfolio AV --sampler-portfolio AV --sampler-portfolio GMM constructs two independent AV members (sampler_types is not deduped; --sampler-portfolio-args aligns positionally), and the band held their combined share at 0.34 in [0.25,0.75] in a smoke test. Kept as an available option for hard cases rather than a default. Also generalizes the enforcement to handle an all-zero group on either side. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
… component cap Per TESTING.md, opt-in code must be probed with the flag ON -- the default-path merge gate is bitwise-identical for opt-in paths and proves nothing about them. The proposed high-SNR settings (--portfolio-varaha-min-frac / --portfolio-varaha-max-frac, and a reduced GMM BIC cap) were measured only on a real event, never through the gate. Add three configurations: varaha floor .25 / varaha band .25-.75 / band + gmm cap3 scored by the gate's own evaluate(), so a PASS here passes by exactly the gate's criteria. The GMM cap needed a small extension: it is not a portfolio attribute but lives on the GMM MEMBER's integrator (gmm_adaptive), which the portfolio forwards via setup(); the probe patches AFTER build, so it now reaches into portfolio_realizations for the reserved key '_gmm_adaptive_cap'. The gate targets are NOT pathological, so these constraints should be near no-ops there; the point is to prove they do not REGRESS shape recovery before proposing them anywhere. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
… confirmation running) band3 (floor .25 + cap .75 + GMM cap3) sd 0.85 / spread 1.79 nats; bkmd (floor + cap3) sd 1.02. Neither lever alone helps: cap3 with unconstrained share is no better than baseline (5.43 vs 5.96) and produced the study's worst confidently-wrong case (n_eff 123.6, lnZ 10 low). Records the n=4 fragility explicitly and that these settings have NOT yet cleared the gate probe. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…; gate PASSED Extends baseline/bkmd/band3 to the same nine seeds. sd 5.04 -> 3.01/3.08, worst deviation from median 10.4 -> 6.1/7.2 nats, but F-test p=0.083/0.092: NOT significant at n=9, and the cap adds nothing measurable over the floor (p=0.48). Records that no config makes a single run trustworthy (6+ nat worst deviation) so consensus pooling stays mandatory. Also records the gate PASS (0 blocking regressions over 96 rows) and that the gate needs --jobs 1 to avoid its pool deadlock. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…l three bars varaha floor / varaha band / band+gmm cap3 all match flags-OFF verdicts on every probe target, scored by the gate's own evaluate(). Settings are SAFE (merge gate + flag-ON probe both clear) but the real-event benefit remains unproven (p~0.08 at n=9). Posture: keep opt-in, change no defaults. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Owner
Author
|
Auto-review update: the required 96-row CPU base-vs-candidate shape-recovery comparison is PASS: 0 blocking regressions with This PR’s portfolio changes are opt-in, so default-path equivalence alone is insufficient. The required flag-on probe is currently running; I will record that final result before clearing the integrator gate. Non-blocking cleanup: |
Owner
Author
|
Final integrator-gate update: the required flag-on portfolio probe also passed with 0 opt-in regressions across adaptive allocation, weight clipping, VARAHA floor/band, and band+GMM-cap configurations. Together with the base/candidate gate, the mandatory integrator validation is clear for this head. The only finding remains the non-blocking trailing whitespace in the design note. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Follow-on to #28 (merged). Started as multi-event harness tooling; the branch now also carries the
high-SNR extrinsic study and the fix that came out of it. No default behaviour changes for
typical events — the one behaviour change is an opt-in flag gate (below).
1. L0 auto-rescue now works for the portfolio (the high-SNR fix)
On a very sharp high-SNR peak (SNR~80 best-fit point), AV contracts onto the peak or the wrong spot
~50/50, so a large fraction of independent runs collapse to
n_eff~1. This is a bimodal lottery,reproduced across every GMM cap,
correlate-all, and coordinate configuration tried — one run is nota posterior at any budget.
--sampler-warmstart-retry-neff(L0 auto-rescue) re-seeds a collapsed run from the peak it didfind and re-runs — same-problem reuse, cannot bias, frame-safe. It was gated to standalone AV;
now enabled for a portfolio carrying an AV member.
Result on a 9-copy pool: landed fraction (n_eff>=5) 4/9 -> 8/9. 6 rescues fired, 5 converted to
clean lands, chronic
n_eff~1collapsers landed at 25-33. Cost: a rescued run does 2 passes.Gate bug found doing this:
opts.sampler_methodis clobbered to'GMM'during portfoliomember setup (the GMM branch sets it to force GMM arg-parsing), so for an AV+GMM portfolio every
downstream
sampler_method == 'portfolio'test is False. The gate now detects a portfolio viaopts.sampler_portfolio(the member list, which is not clobbered). A separate cleanup of the clobberitself is in progress — it also makes the portfolio-only setup block dead code (currently harmless,
since the GMM branch covers the
gmm_adaptiveforwarding).2.
--save-samples0-row export fix (PR #35, cherry-picked)The
_rvscleanup cumsummed log-weights and any-infpoisoned it, so peaked portfolio runsexported 0 rows (even at n_eff~9). Fixed to cumsum linear weights with an
-infguard. This alsounblocked
--extrinsic-proposal-output, which is the weight-correct export (the--save-samplesXMLcarries lnL only and must not be reweighted by likelihood for a shape check).
3. Study results (DESIGN_portfolio_freeze_policy.md)
even at fixed
--seed. Take >=8 draws.dL-inclination arc, phase-pol); landed copies fit 3-4 and agree copy-to-copy. Mode count detects it.
--internal-gmm-correlate-allis strictly worse here (needs too many eff-samples/component).--force-adapt-all --internal-rotate-phaseimproves the landers (n_eff ~50 vs ~30) but does notchange the collapse rate. Coordinate-transform flags need a frame-matched warm-start seed —
--sampler-warmstart-samplesmaps by column NAME without transforming values, so a physical seedpoisons a rotated proposal (0/9 landed until the seed was phase-rotated).
4. Tooling
bench_onsource.sh/bench_onsource_ensemble.sh(seed ensembles on the pinned best-fit point),compare_extrinsic_breadcrumbs.py(cross-copy extrinsic-posterior stability + reliability-weightedpooling),
bench_coverfrac.sh,bench_multi_event.pycontainer mode.Transferable integrator lore extracted to a public repo:
https://github.com/oshaughnessy-junior/rift-integrator-lore
Validation note
The shape-recovery merge gate has not been re-run since the export fix + rescue gate change;
worth running before merge. The rescue gate change only activates with the opt-in
--sampler-warmstart-retry-neffflag (default None = off), so default behaviour is unchanged.🤖 Generated with Claude Code