Use testthat 3e and other tests cleanup #565

Activity

1. VisruthSK

   self-assigned this

   on Jul 24, 2026
2. VisruthSK

   removed their assignment

   on Aug 28, 2026
3. avehtari

   changed the title [-]Use testthat 3e[/-] [+]Use testthat 3e and other tests cleanup[/+] on Oct 28, 2026
4. avehtari commented on Oct 28, 2026

   

   avehtari

   on Oct 28, 2026

   Member

   More actions

   While working on fixing and updating tests, I saw many things that should be cleaned. I already made one tests code cleaning PR #586

   I also asked Claude to suggest what other testthat best practices we could use from CmdStanR

   Items

   id	practice in cmdstanr	change in projpred	effort
   A	Mocking with testthat::local_mocked_bindings() (82 uses)	Tests of the automatic retries in fit_glmer_callback() and fit_gamm_callback() (optimx after "not positive definite", higher nAGQ after a PIRLS error, larger tolPwrss after pwrssUpdate) and of the MASS::glmmPQL() error branches: a mock of lme4::glmer() (.package = "lme4") throws each error, and the test checks the retry arguments and the final error message. These paths have no test now. The tests run on CRAN (no reference models).	small
   B	Fits built on demand (testing_fit(name, method))	Build the reference models, varsel() and cv_varsel() objects of setup.R only when a test needs them, with a cache per session. One test file can then run alone in minutes instead of after the full 20-minute setup. Part of the setup.R split (findings item C2/C4).	large
   C	Snapshots of messages and errors in git (expect_snapshot(error = TRUE), transform = for paths and times; _snaps tracked, 24 files)	Snapshot the projpred warnings and errors instead of long regexes in expect_error() / expect_warning(). A changed message then shows as a diff in the PR and CI checks it. Needs testthat edition 3, so do it with the edition-3 PR.	medium
   D	helper-*.R files, sourced automatically by testthat and by devtools::load_all()	Rename tests/testthat/helpers/*.R to tests/testthat/helper-*.R (or source them from one helper-projpred.R) and remove the source() calls from setup.R. The test helpers are then available in an interactive session after load_all().	small
   E	Custom expectations with fail() and a clear message	Replace expect_true(all(...)) in helpers/testers.R (29 sites) by expectations that show the values, for example expect_equal() on the two objects or a small custom expectation.	medium
   F	withr::local_tempdir() / local_tempfile() for files that tests write (127 uses)	Put the brms fit cache (tests/testthat/bfits) outside the package source tree, for example under tools::R_user_dir("projpred", "cache"), with an option or environment variable for the location.	small

   Order (proposal)

   1. A (small, adds coverage, runs on CRAN).
   2. D and F (small, no change of test results).
   3. Edition 3 together with C.
   4. E.
   5. B

   I plan to make PRs for these, but as there are many PRs waiting for reviews I'm waiting that most of them have been merged
5. VisruthSK commented on Oct 28, 2026

   

   VisruthSK

   on Oct 28, 2026

   Last edited by VisruthSK

   MemberAuthor

   More actions

   I haven't looked too closely at the test suite (to see how hard this would be), but 3e also supports running tests in parallel, which should be tested to see if there's any worthwhile speedup. Might not be that useful after B, but worth trying. Also see stan-dev/cmdstanr#1325