Skip to content

chore: tests never read documentation - #530

Merged
markshust merged 3 commits into
developfrom
feature/remove-roadrunner-audit-page
Oct 9, 2026
Merged

markshust merged 3 commits into
developfrom
feature/remove-roadrunner-audit-page

Conversation

@markshust

@markshust markshust commented Oct 9, 2026 •

Copy link
Copy Markdown
Collaborator

Closes #529

Docs explain how things work for humans and AI; they are not a spec for the implementation. This removes every test that depends on documentation and makes that a hard rule.

Changes

RoadRunner docs

  • Delete docs/packages/roadrunner-state-leaks.md. It was an internal discovery-spike record ("Task 005"), was listed under Packages because the sidebar builds that section from every file in docs/packages/, and still marked the Inertia and ReadWriteConnection leaks as open after both were fixed.
  • Add its user-relevant points to the Stateful Singleton section of roadrunner.md: admin sections live for the whole worker, custom casts must be stateless, login throttle counts persist on purpose, and workers need a restart after switching between MySQL and MariaDB. Remove the link to the deleted page.

Tests that read docs

  • Delete whole doc-only test files: roadrunner DocsTest, the authentication, testing and claude-plugins ReadmeTests, devai DocsPageTest and ReadmeStandardsTest, framework ArchitectureDocTest, validation DocsTest, docs-markdown AiAssistedDocsTest and AstroPipelineTest, and root ReleaseSkillTest and ReleaseDocumentationTest.
  • Remove individual README, docs-page and .claude/ assertions from about 20 more files (package scaffolding and structure tests, rename-reference tests, the claude-plugins plugin tests, root CiWorkflowTest, RemoveDemoDirectoryTest and the integration HarnessTest).
  • Remove the state-leak spike's three findings-document tests and the moduleSingletonIdentifiers() helper only they used. The behavioral harness tests are kept.
  • MarkdownRepositoryTest now runs against tests/Fixtures/docs/ instead of the real docs tree.

Hard rule

  • Added to the CLAUDE.md Key Conventions, a new section in .claude/testing.md, and the code-quality list and merge checklist in .claude/pr-review-process.md.
  • Files the code ships or emits (templates, generated config, installed skill files, exception messages containing docs URLs) are product, not docs, and are still tested.

Docs lint (composer docs:lint)

  • The check that every Marko\... class in a README or docs PHP example exists moved out of the test suite into bin/docs-lint.php. It now also scans the repository README.md.
  • Runs in the required CI Lint job on every PR (no paths filter), even when an earlier lint step failed, and in composer ci. A wrong docs update fails a required check and can't be merged.
  • Predictable: classes resolve by PSR-4 file existence from packages/*/composer.json, nothing is autoloaded. Takes about 0.5s. Exits 1 with file:line for every missing class, and refuses to pass if it finds no Markdown to scan.
  • The scanner's own tests (and new tests for the script's exit codes) run against fixtures. CiWorkflowTest checks that the Lint job and composer ci both run it.

CI service images

  • ci.yml and nightly.yml now pull Redis, Postgres, MySQL and MariaDB from public.ecr.aws/docker/library/ (AWS's mirror of the official Docker images). Anonymous Docker Hub pulls on shared runners were hitting toomanyrequests and failing Tests and Integration before any test ran. CiWorkflowTest now fails if a workflow image points back at Docker Hub. Local tests/Integration/compose.yml is unchanged.

Verification

  • composer ci: 11,831 tests passed (333 skipped), phpcs, php-cs-fixer, PHPStan and docs:lint clean
  • CI: Tests, Lint, Static analysis and Integration all green
  • Planting a fake Marko\Roadrunner\DoesNotExist in roadrunner.md makes docs:lint fail with its file and line
  • phpcs and php-cs-fixer --dry-run: clean
  • composer phpstan: no errors

🤖 Generated with Claude Code

https://claude.ai/code/session_01HHxAV6dBdcdgA6Fm8j34Zn

Docs explain how things work; they are not a spec for the implementation.
Remove every test that reads a docs page, README, CLAUDE.md or .claude/
file, test MarkdownRepository against fixtures, and make it a hard rule
in CLAUDE.md, .claude/testing.md and the PR review checklist.

Delete the public RoadRunner state-leak audit page (an internal spike
record listed under Packages, with stale verdicts) and fold its
user-relevant points into the Stateful Singleton section of roadrunner.md.

Closes #529

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HHxAV6dBdcdgA6Fm8j34Zn
@markshust markshust added the testing Test additions or improvements label Oct 9, 2026
@github-actions github-actions Bot added maintenance Dependency updates and housekeeping and removed testing Test additions or improvements labels Oct 9, 2026
The check read the real docs from inside the test suite. It now runs as
bin/docs-lint.php (composer docs:lint) in the required CI Lint job and in
composer ci, and also scans the repository README. It fails on an empty
scan, resolves classes by PSR-4 file existence only, and takes ~0.5s. The
scanner's own tests run against fixtures.

Refs #529

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HHxAV6dBdcdgA6Fm8j34Zn
@github-actions github-actions Bot added maintenance Dependency updates and housekeeping and removed maintenance Dependency updates and housekeeping labels Oct 9, 2026
Anonymous Docker Hub pulls on shared runners hit `toomanyrequests` and
failed the Tests and Integration jobs before a single test ran.
public.ecr.aws/docker/library mirrors the same official images.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HHxAV6dBdcdgA6Fm8j34Zn
@github-actions github-actions Bot added maintenance Dependency updates and housekeeping and removed maintenance Dependency updates and housekeeping labels Oct 9, 2026
@markshust
markshust merged commit 78c4579 into develop Oct 9, 2026
5 checks passed
@markshust
markshust deleted the feature/remove-roadrunner-audit-page branch October 9, 2026 21:30
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

maintenance Dependency updates and housekeeping

Projects

None yet

Development

Successfully merging this pull request may close these issues.

chore: tests must never read documentation

1 participant