Skip to content

Preserve Stirrup per-scenario code workspaces for benchmark verification #453

Description

@ChathurangiShyalika

Benchmark Stirrup runs currently create code-execution workspaces during each scenario, but those files are cleaned up when the Stirrup session exits. This makes it difficult to inspect scripts, intermediate outputs, and other artifacts produced by code_exec after a scenario finishes.

Expected behavior:

  • Scenario-suite Stirrup runs can write each scenario’s code workspace to a stable host directory.
  • Workspaces are nested by agent/model/run id.
  • Users can opt into preserving workspaces for later verification.
  • Existing benchmark options such as scenario ids, trajectory root, reports root, model id, and continue-on-error continue to work.

Metadata

Metadata

Labels

No labels
No labels

Type

No type

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions