Skip to content

Run an app's specs before its first preview - #80

Merged
raghubetina merged 1 commit into
mainfrom
spec-run-before-preview
Oct 11, 2026
Merged

raghubetina merged 1 commit into
mainfrom
spec-run-before-preview

Conversation

@raghubetina

@raghubetina raghubetina commented Oct 11, 2026 •

Copy link
Copy Markdown
Contributor

From the deploy of firstdraft/firstdraft#1103, a generated app's AGENTS.md says to run its specs before other work, and its README says to build the assets and run the specs before the first preview. Two things in this Codespace would have broken that:

  • The workshop notes that setup copies into ~/.claude/CLAUDE.md and ~/.codex/AGENTS.md sent the agent from Compile straight to bin/dev, and they took precedence over the app's own guides. After root Compile they are the only Drawing Board guidance a session sees.
  • System specs drive a remote browser at selenium:4444 (SELENIUM_HOST, CAPYBARA_SERVER_HOST), but runServices leaves Selenium stopped, and only the bin/ci recipe in AGENTS.md started it. That file moves under .firstdraft/design/ at root Compile, so a plain bundle exec rspec failed every system spec.

This changes only .devcontainer/workshop/app-instructions.md:

  • The precedence sentence now leaves testing to the app's AGENTS.md and README.
  • A new Setup and specs section: after Compile, set up the app, build its assets and run its specs as its README says, before the first preview. In that first run, a NotImplementedError that names behavior listed under the README's Behavior to write is work to do; any other failure is a First Draft defect to tell the user about. Never edit or delete a generated spec to make it pass. Each sentence also holds before the deploy, when apps have no stubs and their README has no Behavior to write.
  • A start, use and stop recipe for Selenium, for every spec run. It starts the app's own Compose selenium service, whose health check makes --wait mean ready, in the running container's Compose project, so Rails and the browser share a network. It runs the specs only once the browser is ready, always stops it, and exits nonzero if any step failed, under bash, sh and zsh (where status is read-only, hence spec_run_status).
  • The web app starts once the specs have run after Compile.

Pins in agent-versions.env are unchanged.

Verified on local Docker Desktop (arm64), not in a Codespace. The Dev Containers CLI 0.89.0 started this branch with lifecycle commands skipped, from the pinned workspace image, with only rails-app and postgres running and the Codespace's CAPYBARA_SERVER_HOST, SELENIUM_HOST and DB_HOST. Then, without restarting the container, the Drawing Board moved under .firstdraft/design/ and the Service's pinned Core tree (6eaeef54, from the archive whose SHA-256 the Compiler pins) went to the root, as root Compile does.

  • bin/setup --skip-server and both asset builds passed.
  • Without Selenium, a system spec failed with Socket::ResolutionError, the failure this fixes.
  • The recipe, read from the notes text, ran bundle exec rspec: 162 examples, 67 of them system examples, 0 failures, exit 0. Selenium waited for its health check, then stopped; rails-app and postgres kept running.
  • A warm start with one system spec file took about 13 seconds in all.
  • A failing spec command exited 1 under bash, bash with set -e, sh and zsh, and a failed project lookup or Compose start exited 1 without running specs; Selenium ended stopped every time.
  • The firstdraft/dockerfiles contract check (drawing-board/script/check at 70e6d6d) passed inside the workspace container.

The simulation used a cached selenium/standalone-chromium:latest from June, so it did not download the image; a Codespace pulls the current one on first start.

Not run: a GitHub Codespace, a real Compile (so no Domain specs and no NotImplementedError stubs), and an agent reading the notes.

CI: contract fails before it reaches this change. Runner image ubuntu24/20261004.327 ships Buildx 0.37.2, which rejects the Dev Containers CLI 0.89.0 Compose build with additional privileges requested: pass "--allow=fs.read=...Dockerfile-with-features" (devcontainers/cli#1320). #79 fails the same way; the last green run used image 20260927.320.

Not for merging until the owner approves: merging changes what new workshop Codespaces get.

From the next service deploy, a generated app's AGENTS.md says to run
its specs before other work, and its README says to build the assets
and run the specs before the first preview. The workshop notes sent
the agent from Compile straight to bin/dev, and they took precedence
over the app's own guides, so they would have skipped that step.

The notes now set up the app and run its specs first, say what a
failure in that first run means, and let the app's AGENTS.md and
README govern testing. A NotImplementedError that names behavior
listed under Behavior to write is work to do; any other failure is a
First Draft defect. Both hold before the deploy, when apps have no
stubs and every failure is a defect.

System specs need a Selenium service that this Codespace leaves
stopped, and only the bin/ci recipe in AGENTS.md started it. That file
moves away at root Compile, so a plain bundle exec rspec failed every
system spec. The notes now carry a start, use and stop recipe for
every spec run. It starts the app's own Compose service, whose health
check makes the wait mean ready, in the running container's Compose
project, where Rails and the browser share a network. It runs the
specs only once the browser is ready, always stops it, and keeps a
failure's exit status. The status variable avoids zsh's read-only
status.
@raghubetina
raghubetina force-pushed the spec-run-before-preview branch from 3bb971b to 589c884 Compare October 11, 2026 23:56
@raghubetina
raghubetina merged commit fe6f9c9 into main Oct 11, 2026
1 check passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant