Repository navigation
test: a session eval for the scripted upgrade - #100
Merged
Merged
Conversation
Adds the scripted-upgrade case to the upgrade suite: a packaged project at v2.3.0 with the docker module, the team's customizations, and covered allow rules, upgraded by /adf:upgrade. inspect.sh checks the order of the commands, the end state, and that nothing was pushed. Until a release carries scripts/upgrade.py, a PATH shim runs the checkout's script as an unreleased commit. Not run in a session: the CLI here isn't signed in. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
…-case # Conflicts: # CHANGELOG.md
…-case # Conflicts: # CHANGELOG.md
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Adds a
scripted-upgradecase to the upgrade session suite, following #98 (decision 0046). The existing case starts at v1.0.0, below the script's v2.0.0 floor, so no session had runadf-upgrade.What's here
project-v2.sh: a packaged project at v2.3.0 with thedockermodule. It has the team's lines inAGENTS.mdanddeployment.md, a Makefile whosewipe-dbtask deletes data, andBash(make test)andBash(git status)beside the rules that cover them.scripted-upgrade.setup.sh: swaps the suite's project for that one, setsADF_FRAMEWORKto the session's copy of the checkout, and, until the newest release tag carriesscripts/upgrade.py, puts anadf-upgradeshim first onPATHthat runs the checkout's script as an unreleased commit.scripted-upgrade.input.mdand.expected.md: the prompt (/adf:upgrade), a follow-up that approves the permission changes and asks for the PR body without pushing, and the invariants.inspect.sh: 14 checks:--dry-runran before the run;mainis untouched, everything is committed, nothing was pushed;autoUpdateis on the pin;askis intact;wipe-dbtask has an ask rule;evals/dynamic/README.md, the runner's header, and the CHANGELOG (no upgrade impact).Not run in a session
The Claude Code CLI here isn't signed in, so the session itself hasn't run. Validated offline: the project builds, the setup works, the shim runs the real upgrade, and
inspect.shpasses all 14 checks on a simulated end state../evals/run-evals.shpasses.Deviation from the brief
No
parallel-agentsin the project. It makes the project a hub, and the upgrade would first dispatch itself into a worktree, which a headless run can't follow. The README says so.To run it:
cd evals/dynamic && ./run-session-evals.sh --suite upgrade --cases scripted-upgrade --models sonnet --source "$PWD/../.."(afterclaude auth login).🤖 Generated with Claude Code