fix: restore full precision to erf's interval constants#412
Open
agene0001 wants to merge 1 commit into
Open
Conversation
`erf_impl` reconstructs `erfc(z)` for `z >= 0.5` as
`exp(-z^2)/z * (b + r(z))`, where `r` is a minimax rational fit and `b` is a
per-interval constant. All 13 of those constants carried only 10 significant
decimal digits:
0.3440242112 // interval [0.5, 0.75)
0.419990927 // interval [0.75, 1.25)
Solving for the `b` each interval's fit implies, at 60 digits, gives a value
that is constant to ~19 digits across the whole interval - so the rational fits
are good to ~1e-19 and the truncated constants were the only error source. The
recovered values turn out to be exactly `f32`-representable, i.e. Boost declared
them as `float` literals and the port that statrs inherits transcribed the
printed decimal:
0.3440242112 -> 0.3440242111682891845703125
0.419990927 -> 0.4199909269809722900390625
Also compensates the squaring: `z * z` is rounded before `exp` sees it, which
costs roughly `z^2` ulps (~460 by z = 27). A Dekker product recovers the
rounding error of the square exactly and folds it back in as
`exp(-z^2) = exp(-sq) * (1 - err)`. (Dekker rather than `f64::mul_add`, which
falls back to a slow software FMA on targets without the instruction.)
Measured against mpmath at 45 dps over a 700-point sweep plus every interval
boundary:
before after
erf ~1e-10 rel 0.24 median / 1.1 max ulp
erfc ~1e-10 rel 0.52 median / 2.8 max ulp
Normal::cdf ~5e-11 rel 0.32 median
This propagated to everything built on `erfc` - `Normal` cdf/sf, `LogNormal`,
`Levy`. The test suite was masking it with `epsilon = 1e-9`/`1e-11`
tolerances, now tightened to 4 ulp.
Six reference literals in the tests were themselves wrong, having been fitted to
the old output; they are replaced with mpmath values. Three are `LogNormal::sf`
expectations (off by up to 5.1e-12) and three are `erfc_inv` expectations - one
of which, `erfc_inv(1e-10)`, was off by 8.8e-9 and had needed
`epsilon = 1e-7` to pass. `erf_inv`/`erfc_inv` themselves were already good to
<2 ulp; only the expectations were wrong.
This was referenced Jul 26, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
erf_implreconstructserfc(z)forz >= 0.5asexp(-z²)/z · (b + r(z)), whereris a minimax rational fit andba per-interval constant. All 13bconstants carry only 10 significant decimal digits:Solving for the
beach interval's fit implies (at 60-digit precision) gives values constant to ~19 digits across each interval — so the rational fits are good to ~1e-19 and the truncated constants were the only error source. The recovered values are exactlyf32-representable: Boost declared them asfloatliterals and the port that statrs inherits transcribed the printed decimal. This PR restores the full values, e.g.0.3440242112→0.3440242111682891845703125.It also compensates the squaring:
z * zis rounded beforeexpsees it, costing ~z² ulps (~460 by z = 27). A Dekker product recovers the rounding error of the square exactly (Dekker rather thanf64::mul_add, which is a slow software FMA on targets without the instruction, e.g. baseline x86-64).Measured against mpmath at 45 dps over a 700-point sweep plus every interval boundary:
erferfcNormal::cdfThis propagates to everything built on
erfc—Normalcdf/sf,LogNormal,Levy. The suite masked it withepsilon = 1e-9/1e-11tolerances, now tightened to 4 ulp of each expected value.Six reference literals in the tests were themselves fitted to the old output and are replaced with mpmath values: three
LogNormal::sfexpectations (off by up to 5.1e-12) and threeerfc_invexpectations — one of which,erfc_inv(1e-10), was off by 8.8e-9 and had neededepsilon = 1e-7to pass.erf_inv/erfc_invthemselves were already ≤2 ulp; only the expectations were wrong.🤖 Generated with Claude Code