Repository navigation
MadNIS: replay buffer unused at default settings, ignored loss option, dead torch path (findings to document) #246
Description
These findings came up while checking MadNIS on signed integrands (interference processes; see #244). Line numbers refer to main at 44b8e19. This issue collects them so they can be documented, and so a few behaviours can be decided on.
1. The replay buffer is never used with the default run card
What happens. The defaults are:
train_batches = 1000(madgraph/various/banner.py:6876);buffer_skip_batches = 1000(banner.py:6896).
Storing into the buffer starts only once _batch_index >= buffer_skip_batches (madspace/src/driver/madnis_training.cpp:507,551). Training ends at batch 1000, so the buffer never receives a sample. The other buffer defaults have no effect either: buffer_capacity = 60000, minimum_buffer_size = 10000, buffered_steps_fraction = 0.8, buffer_unweighting_quantile = 0.95.
Evidence. p p > u u~ QCD^2==2 with [madnis] enable = true and everything else default. In info.json, madnis_trainings[0] has buffer_sizes = [0, 0, …, 0] and buffered_fractions = [0.0, …] for all 10 log points.
To decide and document. Is this intended, i.e. is the buffer meant only for long trainings? If so, the card should say so. If not, buffer_skip_batches should default below train_batches (or scale with it).
The auto-settings already anticipate long trainings:
lr = (10000 - train_batches) / 8000 * 7e-4 + 3e-4;fixed_cwnet_fraction = 1 - 10000 / train_batches(madgraph/iolibs/template_files/mg7/launch.py:2769-2790).
2. Buffer unweighting with signed weights (fixed in #244)
BufferUnweighter (madspace/src/phasespace/unweighter.cpp) accepts an event with probability min(1, |w|/w_max), but it took w_max from the quantile of the signed weights. Nothing changes for a positive integrand. For an interference:
| synthetic batch (200k events, quantile 0.95) | kept, signed quantile | kept, |w| quantile |
|---|---|---|
| positive | 28.5% | 28.5% |
| all negative | 100% (no unweighting at all) | 28.5% |
| 85% negative, short positive tail | 94% | < 50% |
The buffer stayed unbiased throughout, because the adaptive_prob rescaling by acceptance × N/N_kept holds for any w_max.
Effect on training. It is not measurable on our samples. MadNIS was forced on with train_batches = 4000, so the buffer is active from batch 1000; same seed for both:
| process | quantile | RSD | unw. eff. | final loss |
|---|---|---|---|---|
p p > u u~ QCD^2==2 |
signed | 0.1081 | 0.658 | 0.502 |
p p > u u~ QCD^2==2 |
|w| | 0.1085 | 0.649 | 0.501 |
heft g g > t t~ HIG^2==1 (m_H = 500, Γ_H = 30) |
signed | 0.0886 | 0.694 | 0.993 |
heft g g > t t~ HIG^2==1 (m_H = 500, Γ_H = 30) |
|w| | 0.0880 | 0.714 | 0.990 |
Both buffers reach capacity (120k) within about 1200 batches either way.
What #244 adds:
ms.BufferUnweighteris bound to Python, withmadspace/tests/test_buffer_unweighter.py. The signed-quantile code fails the negative and mixed cases.- The class comment is corrected. It described a channel event-file unweighter with a running maximum, and its
quantilewas documented as "allowed fraction of over-weight events" while it is the quantile taken as w_max.
3. The madnis.loss card option is ignored
What happens. The run card offers loss = stratified_variance | kl_divergence | rkl_divergence (banner.py:6874, run_card.toml). MadgraphProcess.train_madnis (launch.py:1317-1370) never forwards it: there is no loss field in the C++ config. MadnisLoss (madspace/src/phasespace/madnis.cpp:118-155) always builds the (stratified) variance loss.
The class comment of MadnisTraining (madspace/include/madspace/driver/madnis_training.hpp:17-24) says the steps are "on the KL-divergence loss".
To decide. Either implement the choice, or drop the option and fix the comment.
4. The Python/torch training path is dead code
MadgraphSubprocess.train_madnis (launch.py:3073) and madgraph/iolibs/template_files/mg7/train_madnis.py are never called. The live path is process.train_madnis() → C++ MultiMadnisTraining, at launch.py:4131.
train_madnis.py imports madspace.madnis, which does not exist in the repository; madspace/madspace/ only has __init__.py, _madspace_py_loader.py and torch.py.
To decide. Remove it, or document it as an external or optional path.
5. For the record: MadNIS on signed integrands works
The loss uses |f| (abs_mean = mean(|f|/q), variance optimum g ∝ |f|), there is no log of f, and the channel weights are positive.
On the heft sample above, MadNIS gives −0.363(21) pb, against VEGAS −0.405(16) pb and madevent −0.403(17) pb (1.6σ). σ_abs agrees: 5.56 vs 5.57 pb.
Auto mode never enables MadNIS for 2 → 2 (n_out <= 2), so these runs forced it on.
Checklist
- Decide and document whether the buffer should be active at default settings (§1).
- Implement or remove
madnis.loss, and fix theMadnisTrainingcomment (§3). - Remove or document the dead torch path (§4).
- Document that MadNIS supports signed integrands (interference), with the buffer behaviour of §2.
Activity
theoheimel commentedon Oct 9, 2026
ramonpeter commentedon Oct 9, 2026
On 5: I'm not convinced this is established yet, and I'd rather not close it with "good".
VEGAS and madevent agree to 0.002 pb, while MadNIS is ~10% low (~1.7σ against the combined reference) from a single run, with the largest error of the three.
The σ_abs agreement (5.56 vs 5.57 pb) doesn't settle it.
Here σ is only a ~7% residual of σ_abs, so the 0.042 pb shift corresponds to a ~0.75% mismatch in how |f| is split between positive and negative regions, which σ_abs does not constrain.
The final loss (~0.99, item-2 table) also sits at its floor: with g ∝ |f| the remaining variance is entirely due to the cancellation, so training can't improve it further.
So "the flow learns |f| and the estimator is unbiased by construction" is fine, but the signed result is the actual test, and at this precision it is inconclusive.
One possible source of a real effect: channel dropping is on by default (
channel_dropping_threshold = 0.01of Σ⟨|f|⟩,madnis_training.cpp:859ff).For this process, 1% of σ_abs ≈ 0.056 pb, the same size as the shift.
A masked channel stays covered by the other flows, so there's no bias in expectation, but if it carries a net signed piece it becomes undersampled and the weights heavy-tailed, so the quoted error would undercover.
Proposal: rerun with ~10× statistics and a few seeds, with and without channel dropping:
At this statistics the per-method error should be ≲ 0.007 pb, enough to see a 0.04 pb shift at ≳4σ.
If MadNIS moves to −0.404, fine; if it stays near −0.36, or its error shrinks slower than 1/√N, we have a real issue.
It would also be good to check whether the quoted σ is computed before or after
max_overweight_truncation, since truncation affects MadNIS and VEGAS differently on a cancelling integrand.theoheimel commentedon Oct 10, 2026
Hm, if there is indeed an issue with channel dropping, we might need to keep track of the positive and negative parts separately when deciding which channels to drop instead of using the absolute value. But first we should confirm that this is really the reason. I have never actually checked the signed integrands because until quite recently the signed matrix elements were broken.