[avehtari]

Summary

Some test files remove objects that testthat removes anyway. Other code in the tests changes the R session and does not undo the change: setup.R and several test files set options and the random seed, and setup.R copies objects into the global environment. On master, a devtools::test() run of test_misc.R alone leaves 18 changed options in the session; on this branch it leaves none. The new restores also run when a test fails.

This PR does not change the tested code. In two places it changes how a test fails or skips (items 4 and 6).

There is a plan for further testthat cleanup, but I'm making separate PRs to keep these easier to review

This PR was assisted by Claude.

Changes

One commit per item:

1. Remove the ls_bu <- ls() ... rm(list = setdiff(ls(), ls_bu)) blocks (7 files). testthat runs each test file in its own environment, so these blocks never removed anything that reached another file.

2. Restore options and the random seed after an error too:

   • withr::local_seed() and withr::local_preserve_seed() replace the rng_old save and restore blocks, and withr::with_seed() replaces the short ones. The old blocks did not run when a test failed, and exists("rng_old") also found an rng_old of an enclosing environment. testthat itself does not restore the random seed.
   • withr::with_options() and withr::local_options() replace the x_orig <- options(...) ... options(x_orig) pairs.
   • This PR adds withr (>= 3.0.3) to Suggests in DESCRIPTION. withr 3.0.2 has a bug: local_seed() does not restore the seed (fixed in 3.0.3, Problem with local_seed? r-lib/withr#286).

3. setup.R cleans up when the tests end (with withr::defer(envir = testthat::teardown_env())):

   • it restores the options and removes the options that the tests add (testthat restores its own options);
   • it restores the random seed;
   • it removes the f_<family> objects that it copies into the global environment.

   test_parallel.R resets its options and the parallel backend at the end of the file, also after an error.

4. Keep test state out of the global environment and report empty tests:

   • test_div_minimizer.R passes the results between its tests in a file-level environment instead of the global environment. Before this change, the comparison test could use stale objects from an earlier run in the same session.
   • test_varsel.R undoes its <<- change of trms_universe_split with withr::defer().
   • test_misc.R checks a fixture with stopifnot() instead of a skip.
   • The new helper skip_if_no_setups() (tests/testthat/helpers/skips.R) skips the 11 tests that only loop over refmods or datafits, with a reason. On CRAN, these lists are empty, and testthat reported the tests as empty. test_helpers_tst.R checks that skip_if_no_setups() skips only for an empty list.

5. Remove the 30 context() calls (deprecated in testthat 3e).

6. test_augdat.R: for K-fold CV only, if the comparison on the logit scale (log scale) fails, the test compares on the probability scale (exp scale). Before this change, tryCatch(expect_equal(...), error = ...) hid the failure of the first comparison, and for the other CV methods a failure ended as an error in stopifnot(is_kfold). Now the first comparison is an all.equal() check, and the other CV methods get a normal expect_equal() failure.

7. test_varsel.R: the rstanarm cvfits test reports failed kfold() refits as a skip with the setup names and the error messages, instead of a message().

8. NEWS.md: one "Tests:" entry.

Not changed

• setup.R drops test setups whose varsel(), cv_varsel() or project() call fails with a known error message (for example the gamm4 getVb() problem). A new bug that gives the same message reduces the coverage with no failure. An explicit list of the setups that are allowed to fail would make this visible. I propose this as a separate change.
• Tests in test_proj_predfun.R, test_project.R and test_varsel.R compare two code paths with tolerances of 1e-14 or 1e-15. They pass on Roihu. The result can depend on the BLAS library. I did not change them, because no case fails.
• Snapshot tests are local regression checks by design. _snaps is in .gitignore and .Rbuildignore. run_snaps is off on CI and in R CMD check.

Tests

With NOT_CRAN=true, the full suite gives the same result on master and on this branch, except for the new test of skip_if_no_setups() (1 test, 2 expectations): 253 and 254 tests, 55,803 and 55,805 expectations, 0 failures. In CRAN mode (installed package, NOT_CRAN=false), master gives 253 tests and 927 expectations, and this branch 254 tests and 929 expectations; both have 180 skipped tests and 0 failures. The skip count is the same, because testthat already counted the empty tests as skipped. Now each of these skips states a reason. Tested with R 4.6.1, testthat 3.3.2, withr 3.0.3 and the development version of lme4 (2.1-1) on CSC Roihu (Linux, AMD EPYC).

Overlap with other PRs

• Faster GLM submodel fits for non-Gaussian families #581 and Cache the submodel formulas and model matrices per predictor-term set #582 change the first and the last lines of tests/testthat/test_glm_ridge.R, where this PR removes the cleanup and the seed blocks. The conflict is textual; keep both changes when you resolve it.
• Keep the data-dependent bases #573, Faster submodel projections: one model matrix and one Gaussian solve for all draws #574 and Leave the left-out observation out of the fold projections in validated LOO CV #580-Add CRAN tests for the submodel fitters from other packages #585 add tests with context() calls. Faster augmented-data projection for ordinal and categorical submodels #583 and Faster projections onto multilevel and additive submodels #584 also add rng_old blocks, and Faster augmented-data projection for ordinal and categorical submodels #583 adds ls_bu blocks. A follow-up PR can convert them after they are merged. (Vectorize the draw-wise predictive variance of the Gaussian and Student-t #575, Cluster the draws in the coordinates of their low-dimensional span #576, Avoid repeated formula updates, term extraction and training-data model matrices #578 and Gaussian forward search and validated LOO evaluation from cross-products of the model matrix #579 are not checked.)