Skip to content

test: add multi-source accuracy fixtures for log1p - #16627

Draft
batpigandme wants to merge 5 commits into
stdlib-js:developfrom
batpigandme:test/accuracy-log1p
Draft

batpigandme wants to merge 5 commits into
stdlib-js:developfrom
batpigandme:test/accuracy-log1p

Conversation

@batpigandme

Copy link
Copy Markdown
Member

type: pre_commit_static_analysis_report
description: Results of running static analysis checks when committing changes.
report:

  • task: lint_filenames
    status: passed
  • task: lint_editorconfig
    status: passed
  • task: lint_markdown_pkg_readmes
    status: na
  • task: lint_markdown_docs
    status: na
  • task: lint_markdown
    status: na
  • task: lint_package_json
    status: na
  • task: lint_repl_help
    status: na
  • task: lint_javascript_src
    status: passed
  • task: lint_javascript_cli
    status: na
  • task: lint_javascript_examples
    status: na
  • task: lint_javascript_tests
    status: na
  • task: lint_javascript_benchmarks
    status: na
  • task: lint_python
    status: na
  • task: lint_r
    status: na
  • task: lint_c_src
    status: na
  • task: lint_c_examples
    status: na
  • task: lint_c_benchmarks
    status: na
  • task: lint_c_tests_fixtures
    status: na
  • task: lint_shell
    status: na
  • task: lint_typescript_declarations
    status: passed
  • task: lint_typescript_tests
    status: na
  • task: lint_license_headers
    status: passed

Description

What is the purpose of this pull request?

This pull request:

  • Adds shared inputs for math/base/special/log1p (10 regimes of 500 points each, plus 19 edge cases), drawn by scripts/inputs.js from a seeded PRNG so they reproduce byte for byte.
  • Adds fixture runners that evaluate those inputs with MPFR (the reference), Julia, Python's math and SciPy, R, Boost, Cephes, and stdlib's own C. Each set's meta records the library, its version, what it calls underneath, and the platform.
  • Adds the 88 sets they generate, all from one x86_64 Linux machine with glibc 2.39.
  • Adds scripts/accuracy.js, which checks that every set echoes the shared inputs bit for bit, then scores each source against MPFR by regime.
  • Tests the implementation against MPFR in test.js, moves the existing Julia tests unchanged to test.julia.js, and adds test.scipy.js.

Scores against MPFR on the 5,000 regime points:

Source Exact 1 ULP 2 ULP
stdlib (JavaScript and C, identical) 4,800 200 0
Python math, R, Boost (glibc) 4,800 200 0
Julia 4,995 5 0
SciPy 4,697 212 91
Cephes 4,590 319 91

Related Issues

Does this pull request have any related issues?

This pull request has the following related issues:

Reference Packages

Did the content of this pull request follow one or more existing packages? Answering is optional.

This pull request followed these packages:

  • math/base/special/fresnel (Cephes runner), math/base/special/digamma (non-finite values stored as strings)

What carried over from the reference(s)?

  • Runner layout under test/fixtures/<source>/, and calling Cephes directly.

What differed, or what did you have to work out yourself?

  • Shared inputs read by every runner, and a scorer that checks the echo before scoring.
  • c/stdlib/ builds the package's C from its manifest.json, as compile_c_benchmark does for benchmarks; no fixture runner did this before.
  • The two C runners read JSON numbers with strtod, checked exact on 205,019 values on macOS and x86_64 Linux.

Questions

Any questions for reviewers of this pull request?

  • Should every transcendental package get this test layout, or only those whose current reference is inaccurate?
  • Non-finite values are stored as "Infinity" and "NaN"; digamma uses "Inf". Which spelling should be standard?
  • Should scripts/accuracy.js stay per package until a few packages use it, or move to a shared tool now?

Other

Any other information relevant to this pull request? This may include screenshots, references, and/or implementation notes.

  • Each bound is the tightest that passes on the committed fixtures, per regime: 1 in the regimes holding stdlib's 200 points at 1 ULP, and 0 elsewhere in test.js. test.scipy.js compares against SciPy, so it reaches 2 where SciPy is 2 ULP off. The bounds describe these fixtures, not the function: sampled at 10^6 points per regime, stdlib also reaches 1 ULP in small_negative and small_positive (about 0.02% of points), where all 500 fixture points are exact.
  • Separately, stdlib reaches 2 ULP in a sliver of about 3.4 × 10^9 doubles just below √2/2 − 1, where its threshold test differs from FDLIBM's, so the comment "the error is always less than 1 ulp" (lib/main.js, src/main.c) doesn't hold. That's left for its own fix.
  • test.native.js is unchanged.

Checklist

Please ensure the following tasks are completed before submitting this pull request.

AI Assistance

When authoring the changes proposed in this PR, did you use any kind of AI assistance?

  • Yes
  • No

If you answered "yes" above, how did you use AI assistance?

  • Code generation (e.g., when writing an implementation or fixing a bug)
  • Test/benchmark generation
  • Documentation (including examples)
  • Research and understanding

Disclosure

If you answered "yes" to using AI assistance, please provide a short disclosure indicating how you used AI assistance. This helps reviewers determine how much scrutiny to apply when reviewing your contribution. Example disclosures: "This PR was written primarily by Claude Code." or "I consulted ChatGPT to understand the codebase, but the proposed changes were fully authored manually by myself.".

The runners, scorer, and tests were written by Claude Code (Opus 5.5), and the fixtures generated in a Claude Code session on x86_64 Linux. I set the design, reviewed every file, and checked the scores against independent reruns. This follows (to the best of my ability) the layout for "accuracy-comparison" tables across languages (AKA the accuracy-visibility project) as discussed with @kgryte.

@stdlib-js/reviewers

Every fixture runner reads the same inputs, so the sets from different
sources can be compared value for value against MPFR. The inputs are
drawn per regime, with the bounds of the package's existing Julia
fixtures, from a seeded PRNG, so the script reproduces them byte for
byte. Regions whose magnitudes span more than a factor of ten are drawn
log-uniformly, for negative ranges too, and the files are JSON only.

Assisted-by: Claude Opus 5.5

---
type: pre_commit_static_analysis_report
description: Results of running static analysis checks when committing changes.
report:
  - task: lint_filenames
    status: passed
  - task: lint_editorconfig
    status: passed
  - task: lint_markdown_pkg_readmes
    status: na
  - task: lint_markdown_docs
    status: na
  - task: lint_markdown
    status: na
  - task: lint_package_json
    status: na
  - task: lint_repl_help
    status: na
  - task: lint_javascript_src
    status: passed
  - task: lint_javascript_cli
    status: na
  - task: lint_javascript_examples
    status: na
  - task: lint_javascript_tests
    status: na
  - task: lint_javascript_benchmarks
    status: na
  - task: lint_python
    status: na
  - task: lint_r
    status: na
  - task: lint_c_src
    status: na
  - task: lint_c_examples
    status: na
  - task: lint_c_benchmarks
    status: na
  - task: lint_c_tests_fixtures
    status: na
  - task: lint_shell
    status: na
  - task: lint_typescript_declarations
    status: passed
  - task: lint_typescript_tests
    status: na
  - task: lint_license_headers
    status: passed
---
Add runners which evaluate `log1p` on the shared inputs with MPFR (the
reference), Julia, Python's `math` and SciPy, R, Boost, Cephes, and
stdlib's own C implementation. Each runner reads the shared JSON inputs,
echoes them, and records in `meta` the library, its version, what it
calls underneath, and the platform, so a scorer can check every set and
say what each column measures.

The Boost runner reads its inputs with Boost.JSON's precise number
parsing. The two C runners read them with `strtod`, as C has no JSON
parser, and stdlib's C runner builds the package's C sources from its
`manifest.json`, so no add-on is needed.

Assisted-by: Claude Opus 5.5

---
type: pre_commit_static_analysis_report
description: Results of running static analysis checks when committing changes.
report:
  - task: lint_filenames
    status: passed
  - task: lint_editorconfig
    status: passed
  - task: lint_markdown_pkg_readmes
    status: na
  - task: lint_markdown_docs
    status: na
  - task: lint_markdown
    status: na
  - task: lint_package_json
    status: na
  - task: lint_repl_help
    status: na
  - task: lint_javascript_src
    status: na
  - task: lint_javascript_cli
    status: na
  - task: lint_javascript_examples
    status: na
  - task: lint_javascript_tests
    status: na
  - task: lint_javascript_benchmarks
    status: na
  - task: lint_python
    status: passed
  - task: lint_r
    status: passed
  - task: lint_c_src
    status: na
  - task: lint_c_examples
    status: na
  - task: lint_c_benchmarks
    status: na
  - task: lint_c_tests_fixtures
    status: passed
  - task: lint_shell
    status: na
  - task: lint_typescript_declarations
    status: passed
  - task: lint_typescript_tests
    status: na
  - task: lint_license_headers
    status: passed
---
Score every fixture set on the shared inputs against MPFR, by regime, and
write a summary which records each source's version and what it calls
underneath. The scorer first checks that every set echoes the shared
inputs bit for bit and names the input file it read, so a misread input
fails loudly instead of skewing a score.

Assisted-by: Claude Opus 5.5

---
type: pre_commit_static_analysis_report
description: Results of running static analysis checks when committing changes.
report:
  - task: lint_filenames
    status: passed
  - task: lint_editorconfig
    status: passed
  - task: lint_markdown_pkg_readmes
    status: na
  - task: lint_markdown_docs
    status: na
  - task: lint_markdown
    status: na
  - task: lint_package_json
    status: na
  - task: lint_repl_help
    status: na
  - task: lint_javascript_src
    status: passed
  - task: lint_javascript_cli
    status: na
  - task: lint_javascript_examples
    status: na
  - task: lint_javascript_tests
    status: na
  - task: lint_javascript_benchmarks
    status: na
  - task: lint_python
    status: na
  - task: lint_r
    status: na
  - task: lint_c_src
    status: na
  - task: lint_c_examples
    status: na
  - task: lint_c_benchmarks
    status: na
  - task: lint_c_tests_fixtures
    status: na
  - task: lint_shell
    status: na
  - task: lint_typescript_declarations
    status: passed
  - task: lint_typescript_tests
    status: na
  - task: lint_license_headers
    status: passed
---
Add the fixture sets each runner generates on the shared inputs, from
MPFR, Julia, Python's `math` and SciPy, R, Boost, Cephes, and stdlib's
own C. All were generated in one session on x86_64 Linux with glibc
2.39, so every value in a package's tables comes from one platform, and
each set's `meta` records the library, its version, what it calls
underneath, and the platform. Every set echoes the shared inputs bit for
bit.

Assisted-by: Claude Opus 5.5

---
type: pre_commit_static_analysis_report
description: Results of running static analysis checks when committing changes.
report:
  - task: lint_filenames
    status: passed
  - task: lint_editorconfig
    status: passed
  - task: lint_markdown_pkg_readmes
    status: na
  - task: lint_markdown_docs
    status: na
  - task: lint_markdown
    status: na
  - task: lint_package_json
    status: na
  - task: lint_repl_help
    status: na
  - task: lint_javascript_src
    status: na
  - task: lint_javascript_cli
    status: na
  - task: lint_javascript_examples
    status: na
  - task: lint_javascript_tests
    status: na
  - task: lint_javascript_benchmarks
    status: na
  - task: lint_python
    status: na
  - task: lint_r
    status: na
  - task: lint_c_src
    status: na
  - task: lint_c_examples
    status: na
  - task: lint_c_benchmarks
    status: na
  - task: lint_c_tests_fixtures
    status: na
  - task: lint_shell
    status: na
  - task: lint_typescript_declarations
    status: passed
  - task: lint_typescript_tests
    status: na
  - task: lint_license_headers
    status: passed
---
Test the implementation against MPFR on the shared inputs, so its bounds
reflect its own error rather than a reference's. The existing tests
against Julia's fixtures move unchanged to `test.julia.js`, and
`test.scipy.js` tests against SciPy's set. Each regime's bound is the
tightest which passes: exact where every value matches, otherwise 1 ULP,
or 2 against SciPy where SciPy itself is off by 2.

Assisted-by: Claude Opus 5.5

---
type: pre_commit_static_analysis_report
description: Results of running static analysis checks when committing changes.
report:
  - task: lint_filenames
    status: passed
  - task: lint_editorconfig
    status: passed
  - task: lint_markdown_pkg_readmes
    status: na
  - task: lint_markdown_docs
    status: na
  - task: lint_markdown
    status: na
  - task: lint_package_json
    status: na
  - task: lint_repl_help
    status: na
  - task: lint_javascript_src
    status: na
  - task: lint_javascript_cli
    status: na
  - task: lint_javascript_examples
    status: na
  - task: lint_javascript_tests
    status: passed
  - task: lint_javascript_benchmarks
    status: na
  - task: lint_python
    status: na
  - task: lint_r
    status: na
  - task: lint_c_src
    status: na
  - task: lint_c_examples
    status: na
  - task: lint_c_benchmarks
    status: na
  - task: lint_c_tests_fixtures
    status: na
  - task: lint_shell
    status: na
  - task: lint_typescript_declarations
    status: passed
  - task: lint_typescript_tests
    status: na
  - task: lint_license_headers
    status: passed
---
@stdlib-bot stdlib-bot added Math Issue or pull request specific to math functionality. Good First PR A pull request resolving a Good First Issue. labels Nov 14, 2026
@batpigandme batpigandme added the Numerical Accuracy Issue or pull request concerns numerical accuracy. label Nov 14, 2026
@stdlib-bot

Copy link
Copy Markdown
Contributor

Coverage Report

Package Statements Branches Functions Lines
math/base/special/log1p $\\color{red}523/525$
$\\color{green}+99.62\\%$
$\\color{red}41/42$
$\\color{green}+97.62\\%$
$\\color{green}3/3$
$\\color{green}+100.00\\%$
$\\color{red}523/525$
$\\color{green}+99.62\\%$

The above coverage report was generated for the changes in this PR.

@batpigandme batpigandme changed the title test: add shared inputs for math/base/special/log1p test: add multi-source accuracy fixtures for log1p Nov 14, 2026
@batpigandme

Copy link
Copy Markdown
Member Author

"Lint Copyright Years" failure is expected. It flags test/test.julia.js ("Expected 2026, found 2018"), which holds existing Julia-fixture tests from test/test.js, moved over unchanged. Since it's moved code, it keeps the original 2018 year.

Followed precedence from other moved files (e.g., #13218 and #12964), which failed this check the same way when code moved into stats/strided.

The warnings for the fixture JSON files are only "no copyright notice found," since JSON fixtures don't carry one.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Good First PR A pull request resolving a Good First Issue. Math Issue or pull request specific to math functionality. Numerical Accuracy Issue or pull request concerns numerical accuracy.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants