Skip to content

Document how the cache behaves under parallel test runners - #82

Open
navidemad wants to merge 1 commit into
Gusto:mainfrom
navidemad:document-parallel-cache-behavior
Open

Document how the cache behaves under parallel test runners#82
navidemad wants to merge 1 commit into
Gusto:mainfrom
navidemad:document-parallel-cache-behavior

Conversation

@navidemad

Copy link
Copy Markdown

Closes the question in #80 by writing down what the cache actually does under each kind of parallel runner. The answer is different for each one, and none of it was documented.

Rails parallelize

ActiveSupport::Testing::Parallelization forks its workers in Minitest.run, before any suite runs, and each worker runs individual test methods (Minitest.run_one_method), never run_suite. Since FixtureKit generates from FixtureKit::Minitest::ClassMethods#run_suite, Runner#start and every generate happen in the parent, and the workers only mount what the parent wrote.

So one shared cache_path is correct here, and it is what the default already gives you. There is also nothing to key the path on: Rails names the per-worker databases itself and never sets TEST_ENV_NUMBER (no reference to it anywhere in activerecord, activesupport or railties). Keying on a worker number would point the workers at directories the parent never writes to.

parallel_tests

The opposite case, and the one you suggested in #80. Each worker is a full process that boots the app and calls Runner#start, so each one clears the directory the others are generating into or mounting from. config.cache_path = "tmp/cache/fixture_kit/#{ENV["TEST_ENV_NUMBER"]}" is the fix; the cost is that every worker generates its own copy of the fixtures its share of the suite needs.

One directory shared between processes

The case with a trap in it, and where our own suite got burned. Reusing a directory across processes (a warm-up run, a CI cache restored between jobs) requires FIXTURE_KIT_PRESERVE_CACHE, or the next process to start deletes it. And preserving the cache hands invalidation to the caller, because Cache#exists? is File.exist?: nothing compares that file against the definitions, the factories they call, or the schema. A cache written before a guard existed mounts without complaint, and the failure surfaces far from the fixture that produced it — for us, a PG::UniqueViolation on versions_pkey from a cache written before a PaperTrail guard landed, with nothing pointing at the cache.

The section documents the pattern that bounds it without asking the gem to do cache busting: a digest of the definitions, factories and schema, folded into cache_path, so a cache written under any other state of the code is ignored and regenerated instead of read. It also names the two costs, obsolete digest directories that are not reclaimed and one changed file regenerating everything, so the tradeoff is visible before someone adopts it. This is the shape we run in CI on a large suite.

That last part is the practical answer to #63, which you closed on the grounds that cache busting is too hard to get right in the gem. Agreed, and this documents how to do it in the application, where the list of inputs is knowable.

Notes

Docs only, no behavior change. docs/reference.md gains a ## Parallel Test Suites section after "Cache Identifiers and Paths", and the FIXTURE_KIT_PRESERVE_CACHE entry links to it.

Issue Gusto#80 asks whether cache files generated by one test process can be reused
by another, and the answer depends entirely on which runner is in use. Nothing
in the reference said so.

Rails `parallelize` forks its workers before Minitest runs any suite, and those
workers run test methods rather than suites, so generation happens in the parent
and workers only mount. One shared `cache_path` is right there, and there is no
worker number to key it on: Rails names the per-worker databases itself and
never sets `TEST_ENV_NUMBER`.

`parallel_tests` is the opposite case. Every worker is a full process that calls
`Runner#start`, so every worker clears the directory the others are using, and
each one needs a `cache_path` of its own.

Sharing a directory across processes is the third case, and the one with a
trap in it: it requires `FIXTURE_KIT_PRESERVE_CACHE`, and preserving the cache
makes invalidation the caller's problem, because `Cache#exists?` is
`File.exist?`. A cache written under older definitions, factories or schema
mounts without complaint and fails somewhere else entirely. The section shows
the digest-in-`cache_path` pattern that bounds it, and what it costs.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant