Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
137 changes: 104 additions & 33 deletions README.md
Original file line number Diff line number Diff line change
@@ -1,69 +1,140 @@
<div align="center">

# grudges

Save the tasks your coding agent failed. Replay them on another model—or check whether your project rules help.
### Your agent forgot. Keep receipts.

Save coding-agent failures. Replay them across models. Find out whether your rules actually help.

[![CI](https://github.com/alexdevmotion/grudges/actions/workflows/ci.yml/badge.svg)](https://github.com/alexdevmotion/grudges/actions/workflows/ci.yml)
[![Release](https://img.shields.io/github/v/release/alexdevmotion/grudges?include_prereleases&color=e9a745)](https://github.com/alexdevmotion/grudges/releases)
[![License: Apache-2.0](https://img.shields.io/badge/license-Apache--2.0-blue.svg)](LICENSE)

[Get started](#get-started) · [Capture a failure](#capture-the-next-failure) · [Compare models & rules](#replay-the-same-task-change-one-thing) · [Docs](docs/usage.md)

<picture>
<source media="(prefers-reduced-motion: reduce)" srcset="docs/assets/demo.png">
<img src="docs/assets/demo.gif" alt="Grudges demo: capture an exact starting state, replay bundled tasks with nop and oracle controls, and inspect solved-attempt counts." width="900">
</picture>

<sub>Edited walkthrough: real local capture and verified starter-pack controls. No model API calls. [Static version](docs/assets/demo.png) · [Demo source & transcript](docs/demo.md)</sub>

</div>

## Turn “not this again” into a test

Your agent fixed the wrong thing. A new model repeats an old mistake. You added another paragraph to `AGENTS.md`, but you don't know whether it helped.

**Grudges gives those moments a memory.** Capture the starting code and failed attempt, build a task with a verifier you review, then replay it under another model or another set of rules.

**Local alpha, 0.1.0a1.** Not yet published to PyPI. Linux is the tested platform. The example packs demonstrate the workflow; they are not a representative model leaderboard.
- **Save the evidence.** A prompt hook captures the starting state; `/grudge` or `$grudge` pins it alongside the failed attempt and session.
- **Test your own work.** Turn a captured failure into a Harbor task, with human review of the reconstruction and verifier.
- **Compare repeat attempts.** Run the same frozen task across models or compare it as captured against a copy with your rules.
- **Read the whole result.** HTML cards show solved/scored counts, missing attempts, errors and partial costs.

## Try a pack
Local capture. No telemetry. Open source. Task authoring and model replay use your configured providers.

From this checkout, with Python 3.11+ and Git:
## Get started

**Python 3.11+ and Git.** Install the published alpha with [uv](https://docs.astral.sh/uv/):

```bash
uv tool install https://github.com/alexdevmotion/grudges/releases/download/v0.1.0a1/grudges-0.1.0a1-py3-none-any.whl
grudges --help
```

<details>
<summary>Install from source instead</summary>

```bash
git clone https://github.com/alexdevmotion/grudges.git
cd grudges
uv tool install .
grudges --version
```

</details>

**Try a replay without an API key.** You'll also need a running Docker daemon with Compose and Buildx:

```bash
uv tool install 'harbor==0.24.0'
# Requires a running Docker daemon with Compose and Buildx.
grudges run --pack starter -a nop -k 1
grudges run --pack starter -a oracle -k 1
```

These controls make no model API calls: nop checks the starting implementation; oracle runs the reference solution. Docker may download images. The wheel includes eight explicit starter tasks and six intent tasks (`--pack starter-intent`). Read each pack's README and FAIRNESS notes before interpreting results.
`nop` leaves the broken code alone; `oracle` runs the reference fix. Expect **0/8** and **8/8** solved. Open the `card.html` path printed after each run. These controls make no model calls; Docker may download images.

For a model run, set `OPENAI_API_KEY` in your environment and choose a model available to your account:
> **Alpha:** Linux tested; macOS and Windows unvalidated. Available on GitHub, not PyPI. The 14 bundled tasks demonstrate the workflow, not a representative model leaderboard.

## Capture the next failure

Install the hook and skill once:

```bash
grudges run --pack starter -a codex -m YOUR_MODEL_ID -k 3
grudges run path/to/task -a codex -m YOUR_MODEL_ID --rules ./AGENTS.md
grudges install
```

Live runs cost money. Codex and Claude Code replay require provider API keys; subscription credentials are not forwarded. Each run freezes task copies and records its configuration before execution. Rules comparisons support the `environment/repo` plus `COPY repo/` Dockerfile layout and the Codex/Claude Code harnesses. Prebuilt `docker_image` tasks are rejected for rules comparisons because they would bypass the modified files. The comparison is **as captured** versus **with rules**: the baseline can already contain rules. The supplied text replaces the root harness rules file in the second copy; it does not remove other instruction files.
Then, when your agent gets something wrong:

```text
Claude Code → /grudge The amount should stay editable until blur.
Codex → $grudge The amount should stay editable until blur.
```

## Capture your own failure
Find the saved ID and build a replayable task:

```bash
grudges install
# In Claude Code: /grudge what the correct outcome should have been
# In Codex: $grudge what the correct outcome should have been
grudges requests
grudges list
grudges build GRUDGE_ID --shareable
```

Installation adds a prompt hook and skill to your local agent configuration. Existing config is backed up; unrelated hooks and edited skills are preserved. Codex may ask you to approve the hook. `grudges install --uninstall` removes managed hooks and unchanged managed skills.
Building opens an interactive authoring session. **Review the reconstructed task and verifier before using the score.** The `--shareable` option requests fresh code and names; you still review the result before publishing it.

Capture preserves your working tree, normal Git index and branches. Missing checkpoints are labeled approximate. To remove the managed hooks and unchanged managed skills: `grudges install --uninstall`.

## Replay the same task. Change one thing.

Set `OPENAI_API_KEY` in your environment for Codex replay. Use model IDs available to your account; live runs incur provider charges.

**Compare models** on the same task, three attempts each:

```bash
grudges run path/to/task -a codex \
-m MODEL_A -m MODEL_B -k 3
```

**Test your project rules** against the captured setup:

```bash
grudges run path/to/task -a codex \
-m YOUR_MODEL_ID --rules ./AGENTS.md -k 3
```

The two arms are **as captured** and **with rules**. Existing baseline rules are retained; the second copy replaces the root harness rules file. Inputs are frozen before execution. Native Codex and Claude Code replay use API keys, not subscription logins. See [supported rules layouts and behavior](docs/usage.md#try-a-pack).

Before each request, capture snapshots tracked and non-ignored untracked files using a temporary Git index. Saving pins the starting and failed states under `refs/grudges/` and copies the session transcript into `~/.grudges`. Your working tree, normal index and branches are preserved. Missing or failed checkpoints are labeled approximate. Avoid concurrent writers while capturing; filesystem snapshots are not atomic across files. Large snapshots are skipped.
## Scores you can inspect

Task building fetches a pinned [evals-from-sessions](https://github.com/ibragim-bad/evals-from-sessions) authoring skill and launches your chosen local agent interactively. Review the reconstruction and verifier before accepting the task. `--shareable` requests fresh code and names; it is not an automatic privacy guarantee. `--print` prints the authoring brief without launching the agent, but still prepares files and fetches the skill.
| What you see | What it means |
| --- | --- |
| Solved / scored | Successful attempts out of valid scored attempts; timeouts count as failures. |
| Errors & missing | Provider, grader and infrastructure errors, plus planned attempts with no receipt. |
| Majority-fixed | Every planned attempt was scored and strictly more than half solved the task. |
| Cost | Reported usage, with unavailable or partial totals labeled. |

## Read results
Each run keeps `results.json`, the HTML card and Harbor receipts locally. A whole-task score doesn't tell you which requirement failed; inspect the verifier output. You can [set public titles and regenerate a card](docs/usage.md#read-results) before sharing it.

Each run writes `results.json` and `card.html` under `~/.grudges/runs/`. `grudges card` renders the latest run, or pass a run directory. `--titles titles.json` supplies public task titles.
## Built on tools that already work

Cards show solved/scored attempts, errors and missing attempts. A task is majority-fixed only when all planned attempts are scored and strictly more than half solve it. Agent timeouts count as scored failures; provider, infrastructure and grader errors do not. Partial cost totals are labeled. A whole-task reward cannot identify which individual requirement failed. Inspect verifier receipts for that detail. CLI exit 2 signals setup/execution errors; a valid scored task failure returns 0.
Grudges handles capture, replay comparisons and reporting. [evals-from-sessions](https://github.com/ibragim-bad/evals-from-sessions) supplies the task-authoring workflow; [Harbor](https://github.com/harbor-framework/harbor) runs the tasks.

## Privacy and limits
The example packs include [eight explicit tasks](packs/starter/README.md) and [six intent tasks](packs/starter-intent/README.md). Their fairness notes distinguish stated requirements from inferred behavior. [Release checks](RELEASE-CHECKS.md) document what has actually been validated.

Capture is local, but snapshots and transcripts can contain secrets or private code—even when the current files are later deleted. Git ignore rules do not exclude already tracked secrets. Keep `~/.grudges`, Git refs, build workspaces and raw Harbor results private. Building and replaying tasks can send code to the configured provider. Docker tasks execute code; review unfamiliar tasks before running them.
## Keep your captures private

Cards omit raw paths, prompts and run metadata, and use neutral task names when public titles are unavailable. Still review titles, model names and all visible content before sharing. No telemetry or hosted service is included. macOS and Windows have not been validated.
Snapshots and transcripts can contain secrets and private code. Capture stays local; authoring and model replay can send code to your provider. Docker tasks execute code. Review tasks, titles and report contents before running or sharing them. [Privacy details](docs/usage.md#privacy-and-limits) · [Report a vulnerability privately](SECURITY.md)

## Development and license
## Contribute a fix—or a good failure

```bash
python3 -m pip install -e . build
python3 -m unittest discover -s tests -v
python3 -m build
python3 scripts/check_dist.py dist
```
Useful contributions include reproducible bugs, clearer verifiers, and re-authored tasks with explicit contracts. Don't upload raw sessions or private code. Start with [CONTRIBUTING.md](CONTRIBUTING.md); development commands are in the [usage guide](docs/usage.md#development-and-license).

See [CONTRIBUTING.md](CONTRIBUTING.md), [SECURITY.md](SECURITY.md) and [RELEASING.md](RELEASING.md). Apache-2.0; adapted portions retain their MIT notice in [NOTICE](NOTICE). Execution uses [Harbor](https://github.com/harbor-framework/harbor).
Apache-2.0. Adapted portions retain their MIT notice in [NOTICE](NOTICE).
15 changes: 15 additions & 0 deletions docs/assets/demo-data.json
Original file line number Diff line number Diff line change
@@ -0,0 +1,15 @@
{
"capture": "saved grudge DEMO_ID \u00b7 exact starting state \u00b7 demo-repo\nepisode starts at: 'Format the amount on blur.'",
"controls": [
{
"agent": "oracle",
"solved": 8,
"scored": 8
},
{
"agent": "nop",
"solved": 0,
"scored": 8
}
]
}
Binary file added docs/assets/demo.gif
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file added docs/assets/demo.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
36 changes: 36 additions & 0 deletions docs/demo.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,36 @@
# README demo

The animation is an edited walkthrough, not a real-time recording. It loops for 18 seconds and has a [static alternative](assets/demo.png). No paid model calls were made to produce it.

## What is real

1. **Capture:** a temporary Git repository contained a tiny amount-formatting fixture. The prompt hook ran before a deliberately incorrect edit. `grudges save --session demo --note "Keep the amount editable until blur."` saved an exact starting state. The displayed output is real, with the generated ID replaced by `DEMO_ID`. No user project or agent session is included.
2. **Replay:** the starter-pack nop and oracle counts come from the release validation's individual Harbor receipts: 0/8 and 8/8 solved, no errors. These are controls on the bundled pack, not a replay of the capture fixture, a model comparison or a speed claim. Progress logs and waiting time are omitted.
3. **Compare:** the final scene shows next-step command examples. `MODEL_A`, `MODEL_B`, `YOUR_MODEL_ID` and `path/to/task` are placeholders. This scene reports no measured model results. Author and review a task with `grudges build` before replaying your own capture.

## Text transcript

```console
$ grudges save --session demo --note "Keep the amount editable until blur."
saved grudge DEMO_ID · exact starting state · demo-repo
episode starts at: 'Format the amount on blur.'

$ grudges run --pack starter -a nop -k 1
nop: fixed 0/8 · attempts 0/8 · no errors

$ grudges run --pack starter -a oracle -k 1
oracle: fixed 8/8 · attempts 8/8 · no errors

$ grudges run path/to/task -a codex -m MODEL_A -m MODEL_B -k 3
$ grudges run path/to/task -a codex -m YOUR_MODEL_ID --rules ./AGENTS.md -k 3
```

## Render

The sanitized inputs are in [demo-data.json](assets/demo-data.json); the renderer is [scripts/render_demo.py](../scripts/render_demo.py). Use Python with Pillow and the DejaVu Sans font family installed:

```bash
python scripts/render_demo.py
```

This regenerates `docs/assets/demo.gif` and `demo.png`. Rendering does not run Grudges, Docker or any model. To validate the underlying control results yourself, run the nop/oracle commands above and inspect the receipts.
65 changes: 65 additions & 0 deletions docs/usage.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,65 @@
# Usage and behavior

## Try a pack

From this checkout, with Python 3.11+ and Git:

```bash
uv tool install .
grudges --version
uv tool install 'harbor==0.24.0'
# Requires a running Docker daemon with Compose and Buildx.
grudges run --pack starter -a nop -k 1
grudges run --pack starter -a oracle -k 1
```

These controls make no model API calls: nop checks the starting implementation; oracle runs the reference solution. Docker may download images. The wheel includes eight explicit starter tasks and six intent tasks (`--pack starter-intent`). Read each pack's README and FAIRNESS notes before interpreting results.

For a model run, set `OPENAI_API_KEY` in your environment and choose a model available to your account:

```bash
grudges run --pack starter -a codex -m YOUR_MODEL_ID -k 3
grudges run path/to/task -a codex -m YOUR_MODEL_ID --rules ./AGENTS.md
```

Live runs cost money. Codex and Claude Code replay require provider API keys; subscription credentials are not forwarded. Each run freezes task copies and records its configuration before execution. Rules comparisons support the `environment/repo` plus `COPY repo/` Dockerfile layout and the Codex/Claude Code harnesses. Prebuilt `docker_image` tasks are rejected for rules comparisons because they would bypass the modified files. The comparison is **as captured** versus **with rules**: the baseline can already contain rules. The supplied text replaces the root harness rules file in the second copy; it does not remove other instruction files.

## Capture your own failure

```bash
grudges install
# In Claude Code: /grudge what the correct outcome should have been
# In Codex: $grudge what the correct outcome should have been
grudges requests
grudges list
grudges build GRUDGE_ID --shareable
```

Installation adds a prompt hook and skill to your local agent configuration. Existing config is backed up; unrelated hooks and edited skills are preserved. Codex may ask you to approve the hook. `grudges install --uninstall` removes managed hooks and unchanged managed skills.

Before each request, capture snapshots tracked and non-ignored untracked files using a temporary Git index. Saving pins the starting and failed states under `refs/grudges/` and copies the session transcript into `~/.grudges`. Your working tree, normal index and branches are preserved. Missing or failed checkpoints are labeled approximate. Avoid concurrent writers while capturing; filesystem snapshots are not atomic across files. Large snapshots are skipped.

Task building fetches a pinned [evals-from-sessions](https://github.com/ibragim-bad/evals-from-sessions) authoring skill and launches your chosen local agent interactively. Review the reconstruction and verifier before accepting the task. `--shareable` requests fresh code and names; it is not an automatic privacy guarantee. `--print` prints the authoring brief without launching the agent, but still prepares files and fetches the skill.

## Read results

Each run writes `results.json` and `card.html` under `~/.grudges/runs/`. `grudges card` renders the latest run, or pass a run directory. `--titles titles.json` supplies public task titles.

Cards show solved/scored attempts, errors and missing attempts. A task is majority-fixed only when all planned attempts are scored and strictly more than half solve it. Agent timeouts count as scored failures; provider, infrastructure and grader errors do not. Partial cost totals are labeled. A whole-task reward cannot identify which individual requirement failed. Inspect verifier receipts for that detail. CLI exit 2 signals setup/execution errors; a valid scored task failure returns 0.

## Privacy and limits

Capture is local, but snapshots and transcripts can contain secrets or private code—even when the current files are later deleted. Git ignore rules do not exclude already tracked secrets. Keep `~/.grudges`, Git refs, build workspaces and raw Harbor results private. Building and replaying tasks can send code to the configured provider. Docker tasks execute code; review unfamiliar tasks before running them.

Cards omit raw paths, prompts and run metadata, and use neutral task names when public titles are unavailable. Still review titles, model names and all visible content before sharing. No telemetry or hosted service is included. macOS and Windows have not been validated.

## Development and license

```bash
python3 -m pip install -e . build
python3 -m unittest discover -s tests -v
python3 -m build
python3 scripts/check_dist.py dist
```

See [CONTRIBUTING.md](../CONTRIBUTING.md), [SECURITY.md](../SECURITY.md) and [RELEASING.md](../RELEASING.md). Apache-2.0; adapted portions retain their MIT notice in [NOTICE](../NOTICE). Execution uses [Harbor](https://github.com/harbor-framework/harbor).
4 changes: 2 additions & 2 deletions pyproject.toml
Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@
[project]
name = "grudges"
version = "0.1.0a1"
description = "Save the tasks your coding agent failed. Re-run them on every new model."
description = "Turn coding-agent failures into replayable tests. Compare models and project rules."
readme = "README.md"
license = "Apache-2.0"
license-files = ["LICENSE", "NOTICE"]
Expand All @@ -27,4 +27,4 @@ packages = ["src/grudges"]
"packs" = "grudges/packs"

[tool.hatch.build.targets.sdist]
include = ["src", "packs", "tests", "scripts", "README.md", "CHANGELOG.md", "CONTRIBUTING.md", "SECURITY.md", "RELEASING.md", "RELEASE-CHECKS.md", "LICENSE", "NOTICE", "pyproject.toml"]
include = ["src", "packs", "tests", "scripts", "docs", "README.md", "CHANGELOG.md", "CONTRIBUTING.md", "SECURITY.md", "RELEASING.md", "RELEASE-CHECKS.md", "LICENSE", "NOTICE", "pyproject.toml"]
Loading
Loading