Skip to content

Reject CLI tests that assert only an exit code - #814

Merged
jviotti merged 3 commits into
mainfrom
vacous-dsl
Aug 12, 2026
Merged

Reject CLI tests that assert only an exit code#814
jviotti merged 3 commits into
mainfrom
vacous-dsl

Conversation

@jviotti

@jviotti jviotti commented Aug 11, 2026

Copy link
Copy Markdown
Member

Signed-off-by: Juan Cruz Viotti jv@jviotti.com

Review in cubic

Signed-off-by: Juan Cruz Viotti <jv@jviotti.com>
@augmentcode

augmentcode Bot commented Aug 11, 2026

Copy link
Copy Markdown
🤖 Augment PR Summary

Summary: This PR adds static validation to the CLI-test DSL so tests must observe produced results.

  • Splits script parsing from execution through a reusable Statement representation.
  • Parses WRITE ... UNTIL bodies before creating the sandbox or invoking the CLI.
  • Adds a preflight data-flow check for RUN observations, generated artifacts, and checksum variables.
  • Tracks comparisons, copies, and downstream input consumption to permit asserted output flows.
  • Reports preflight diagnostics using the relevant DSL source line and stops before test execution.
  • Retains the existing interpreter command dispatch, now operating on parsed statements.
  • Extends decode and encode tests with assertions for previously unchecked captured output.
  • Extends lint, install, test, and validate fixtures with explicit expected-output comparisons.
  • Normalizes platform-dependent paths and benchmark timing values before newly added comparisons.
  • Adds __pycache__/ to the repository ignore list.

🤖 Was this summary useful? React with 👍 or 👎

@augmentcode augmentcode Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Review completed. 2 suggestions posted.

Fix All in Augment

Comment augment review to trigger a new review at any time.

Comment thread scripts/clitest.py Outdated
Comment thread scripts/clitest.py

@cubic-dev-ai cubic-dev-ai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

All reported issues were addressed across 56 files

Reply with feedback, questions, or to request a fix.

Re-trigger cubic

Comment thread scripts/clitest.py Outdated
Comment thread scripts/clitest.py Outdated
Comment thread scripts/clitest.py Outdated
Comment thread scripts/clitest.py Outdated
Comment thread scripts/clitest.py Outdated
Comment thread scripts/clitest.py Outdated
Signed-off-by: Juan Cruz Viotti <jv@jviotti.com>

@cubic-dev-ai cubic-dev-ai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

All reported issues were addressed across 1 file (changes from recent commits).

Reply with feedback, questions, or to request a fix.

Re-trigger cubic

Comment thread scripts/clitest.py Outdated
Comment thread scripts/clitest.py Outdated
Signed-off-by: Juan Cruz Viotti <jv@jviotti.com>
@jviotti
jviotti merged commit cee67e6 into main Aug 12, 2026
15 of 16 checks passed
@jviotti
jviotti deleted the vacous-dsl branch August 12, 2026 00:43
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant