integrators: stabilize the MC error estimate (tail diagnostic, replicas, disclosed budgets) - #38
Conversation
…as, disclosed budgets) The reported sigma_lnL is the pooled weight sample-variance -- algebraically 1/ESS_hat - 1/n restated, computed from the SAME weights as the integral. It is tail-blind: a run that misses the likelihood peak reports BOTH a low lnL AND a small sigma, so the error bar fails conditionally on the run being wrong (toy model: median underestimate 1.0-2.4x for tau=1-5, 90% coverage collapses to 0.24, reported sigma ANTI-correlated with actual error; corroborated by dslice exports where sigmaL understates redraw scatter 3-18x at high SNR). Defenses, all disclosed in the job log: * statutils: new helpers pareto_khat_from_log (Zhang-Stephens GPD tail fit, PSIS xi convention), ess_from_log_weights, block_scatter_sigma (delete-one jackknife over per-chunk lnZ), bootstrap_lnZ_quantiles. * mcsampler + mcsamplerGPU: record per-chunk lnZ during the run; at finalize emit pareto_khat, n_ESS, sigma_lnZ_block, and (when relative error > 0.3) bootstrap lnZ quantiles via dict_return. Also fix a bug where the _rvs['weights'] record appended new weights onto joint_s_prior. * mcsamplerAdaptiveVolume: the naive term is blind to the stochastic live volume (Z ~ V*mean(w)) and to the truncated probability; accumulate Var(lnV) per cycle and add var_lnV + trunc_p^2 to the returned relative variance, with components in dict_return. * ILE (integrate_likelihood_extrinsic_batchmode): floor the reported sigma at the between-chunk scatter; print k-hat (k>0.7: sigma is a lower bound) and bootstrap CI; new opt-in --mc-error-replicas N (+ --mc-error-sigma-trigger 0.4, --mc-error-khat-trigger 0.7, --mc-error-ess-trigger 30): when triggered, rerun cold replicas in-process and report the LINEAR mean of the replica integrals with sigma = max(propagated, replica scatter), neff = sum; posterior export keeps the best-converged replica. Default off. * util_CleanILE: combine repeated evaluations by the ntot-weighted LINEAR mean with sigma = max(propagated, between-replica scatter). The previous 1/sigma^2 weighting overweighted exactly the replica that missed the peak (low lnL AND low sigma): toy model shows -0.86 nat combine bias vs -0.26 for the linear mean. Validation: 19/19 in the roboto-paper demos/error_estimate/run_validation.py (k-hat recovers GPD k=0.3/0.7 to +-0.04; all samplers emit the diagnostics; CleanILE matches the new formula) and .travis test_mcsamplerEnsemble_extended --as-test passes both variants unchanged. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
|
2026-08-16 00:42 UTC read-only interhost monitoring: PR #94 ( |
|
2026-08-18 00:42 UTC read-only Stage-8 monitoring verified completed cycle PR #114: reviewed replacement head f85d07c over rift_O4d_junior@0784e20e had successful replacement dispatch and feature-relevant CI; its only private CHANGES_REQUESTED review was stale on b9bb892 and was not accessed. The owner manually merged it as 5ebfa4a at 22:53 UTC; subsequent junior merge CI 32078013079 succeeded. Stacked PR #118 also merged before its parent. No route, queue, worker, source, checkout, review, or merge action was taken. Evidence: activities/2026-08-06-PRReviewSessionRouter/runs/2026-08-18-pr114-completed-cycle-0042.md. Next: observe only later exact-head cycles; retain observer/developer and checkout-ownership interlocks. |
|
2026-08-18 03:45 UTC read-only review-flow monitoring: new PR #130 is OPEN/CLEAN at exact head |
|
2026-08-18 09:41 UTC read-only Stage-8 monitoring: PR #130 ( |
|
2026-08-19 09:41 UTC read-only Stage-8 monitoring: PR #162 is exact |
1 participant
integrators: stabilize the MC error estimate (tail diagnostic, replicas, disclosed budgets)
Supersedes #37, rebased onto
rift_O4dper review (base now includes the hard shape-gate suite and optional-plugin isolation). Same single commit, cherry-picked; one trivial conflict vs the AV warm-start block, resolved keeping both.What this changes
The reported
sigma/Lis the pooled weight sample-variance -- algebraically1/ESS_hat - 1/nrestated, computed from the SAME weights as the integral. It is tail-blind and fails conditionally on the run being wrong (a run that misses the peak reports BOTH low lnL AND small sigma). Changes:pareto_khat_from_log(Zhang-Stephens GPD tail fit, PSIS xi convention),ess_from_log_weights,block_scatter_sigma(delete-one jackknife over per-chunk lnZ),bootstrap_lnZ_quantiles.pareto_khat,n_ESS,sigma_lnZ_block,lnZ_ci90viadict_return; fixes_rvs['weights']appending ontojoint_s_prior.Var(lnV)(live volume is a stochastic product of per-cycle binomial survival fractions) andtrunc_p^2terms to the returned relative variance, disclosed indict_return.--mc-error-replicas(+ sigma/khat/ESS triggers) reruns cold replicas in-process, combined by the LINEAR mean with scatter-based error. Default off.max(propagated, scatter)error.Merge-gate evidence (this head =
junior/rift_O4d@ cbb67cc + 1 commit)test_mcsamplerEnsemble_extended.py --as-test --n-max 100000exit 0, and--use-lnLvariant exit 0 (CPU,CUDA_VISIBLE_DEVICES="").rift_O4d: 8 strict failures, 5 warn-only, 45 starvedcompare_shape_results.py: 0 blocking regressions (strict = AV, GMM); 5 IMPROVED(fail->pass); 4 PREEXISTING-FAIL (fail on both).mix_d8_n1_s101bias +0.642 on both sides. What changed is the tolerance, which the gate scales by the reported sigma: with the previously-missingVar(lnV) + trunc_p^2terms disclosed, several borderline targets now pass because the error bar is honest, not because the bias shrank. The residual AV lnZ biases (d4_n1_s202-0.289,d6_n1_s101+0.528,d8_n1_s101+0.642) are pre-existing and are exactly the class of error this PR's k-hat flag + replicate-when-loud machinery is designed to expose downstream.mix_d2_n1_s101shape failure; AV lnZ biases above.~/RIFT_develUWM/src/research-projects-RIT/.claude/worktrees/{base_shape.json,pr_shape.json,base_shape.log,pr_shape.log}(run viarun_shape_recovery.sh, SHAPE_JOBS=8).Validation beyond the gate (paper repo
demos/error_estimate/): 19/19 unit/e2e checks incl. GPD k-hat recovery on known tails; log-normal toy quantifying the conditional underestimate (coverage 0.90->0.24 over tau=1..5; inverse-variance combine bias -0.86 vs -0.26 nats for linear mean).🤖 Generated with Claude Code