From 0b7bc4d700c4f975a124417dc3d78684fa9c7ddd Mon Sep 17 00:00:00 2001 From: Adam Henson Date: Mon, 20 Apr 2026 11:04:17 -0400 Subject: [PATCH 1/3] Add cost-tracker workflow for agent spend observability --- README.md | 1 + docs/cost-tracker.md | 77 +++++++++++++++++ workflows/cost-tracker.md | 173 ++++++++++++++++++++++++++++++++++++++ 3 files changed, 251 insertions(+) create mode 100644 docs/cost-tracker.md create mode 100644 workflows/cost-tracker.md diff --git a/README.md b/README.md index 69acc1c..c334ff9 100644 --- a/README.md +++ b/README.md @@ -18,6 +18,7 @@ Investigate faults proactively and improve CI. - [🏥 CI Doctor](docs/ci-doctor.md) - Monitor CI workflows and investigate failures automatically - [🚀 CI Coach](docs/ci-coach.md) - Optimize CI workflows for speed and cost efficiency +- [💰 Cost Tracker](docs/cost-tracker.md) - Post per-run agent spend summaries on pull requests using token-usage.jsonl from gh-aw's firewall ### Code Review Workflows diff --git a/docs/cost-tracker.md b/docs/cost-tracker.md new file mode 100644 index 0000000..af0607b --- /dev/null +++ b/docs/cost-tracker.md @@ -0,0 +1,77 @@ +# 💰 Cost Tracker + +> For an overview of all available workflows, see the [main README](../README.md). + +**Automated agent cost reporter that posts a spend summary after every agent workflow run** + +The [Cost Tracker workflow](../workflows/cost-tracker.md?plain=1) fires after your configured agent workflows complete, downloads the `token-usage.jsonl` data written by gh-aw's firewall, calculates per-model spend, and posts a cost breakdown on the associated pull request or creates a cost report issue. + +## Installation + +```bash +# Install the 'gh aw' extension +gh extension install github/gh-aw + +# Add the workflow to your repository +gh aw add-wizard githubnext/agentics/cost-tracker +``` + +This walks you through adding the workflow to your repository. + +## How It Works + +```mermaid +graph LR + A[Agent Workflow Completes] --> B[Download agent-artifacts] + B --> C{artifact found?} + C -->|No| D[Exit silently] + C -->|Yes| E[Parse token-usage.jsonl] + E --> F[Calculate per-model cost] + F --> G{PR associated?} + G -->|Yes| H[Post comment on PR] + G -->|No| I[Create cost report issue] + F --> J{Cost over threshold?} + J -->|Yes| K[Create high-spend alert issue] +``` + +The workflow reads `token-usage.jsonl` from the `agent-artifacts` artifact written by +gh-aw's firewall on every agent run. It calculates cost using a built-in per-model +pricing table and posts the result where it is most useful — the PR that triggered the +agent run, or a new issue when there is no PR. + +Runs that do not produce an `agent-artifacts` artifact (non-agent CI workflows) are +skipped silently. + +## Usage + +### Configuration + +After installing, open the workflow file and update the `workflows` list under `on.workflow_run` +to match the names of your agent workflows: + +```yaml +on: + workflow_run: + workflows: ["agent-implement", "agent-pr-fix"] # your workflow names here + types: + - completed +``` + +To adjust the high-spend alert threshold, find the `$1.00` value in the workflow body +and change it to your preferred limit. + +After editing run `gh aw compile` to update the workflow and commit all changes to the +default branch. + +### Token data source + +Cost Tracker reads `sandbox/firewall/logs/api-proxy-logs/token-usage.jsonl` from the +`agent-artifacts` artifact. This file is written automatically by gh-aw's firewall on +every agent run. No additional configuration is needed to produce it — the data is +already there if you are running gh-aw with firewall enabled (the default). + +## Learn More + +- [token-usage.jsonl reference](https://github.github.com/gh-aw/reference/token-usage/) +- [gh-aw firewall documentation](https://github.github.com/gh-aw/reference/firewall/) +- [CI Doctor workflow](ci-doctor.md) — investigate CI failures automatically diff --git a/workflows/cost-tracker.md b/workflows/cost-tracker.md new file mode 100644 index 0000000..533ea75 --- /dev/null +++ b/workflows/cost-tracker.md @@ -0,0 +1,173 @@ +--- +description: | + Automated agent cost tracker that fires after monitored workflows complete, downloads + the agent-artifacts artifact written by gh-aw's firewall, parses token-usage.jsonl, + calculates per-model spend, and posts a cost summary on the associated pull request. + Creates a cost report issue when no pull request is found. Optionally creates a + high-spend alert issue when a single run exceeds a configurable threshold. + +on: + workflow_run: + workflows: ["agent-implement", "agent-pr-fix"] # Edit to match your agent workflow names + types: + - completed + +permissions: read-all + +network: defaults + +safe-outputs: + add-comment: + target: "*" + create-issue: + title-prefix: "[cost-tracker] " + labels: [automation, cost] + max: 2 + +tools: + github: + toolsets: [default] + bash: true + +timeout-minutes: 10 + +--- + +# Agent Cost Tracker + +You are the Agent Cost Tracker. Your job is to read the token usage data written by +gh-aw's firewall after an agent workflow completes and report back what that run cost. + +## Current Context + +- **Repository**: ${{ github.repository }} +- **Triggered by**: ${{ github.event.workflow_run.name }} +- **Run**: [#${{ github.event.workflow_run.run_number }}](${{ github.event.workflow_run.html_url }}) +- **Run ID**: ${{ github.event.workflow_run.id }} +- **Conclusion**: ${{ github.event.workflow_run.conclusion }} +- **Head SHA**: ${{ github.event.workflow_run.head_sha }} + +## Instructions + +### Step 1: Download the agent-artifacts artifact + +Use bash to download the `agent-artifacts` artifact from the triggering run: + +```bash +gh run download ${{ github.event.workflow_run.id }} \ + --name agent-artifacts \ + --dir /tmp/agent-artifacts \ + --repo ${{ github.repository }} 2>&1 +echo "exit: $?" +``` + +**If this command fails** (the artifact does not exist), the triggering workflow did not +produce cost data — it was not an agent run or the firewall was not enabled. Exit without +creating any issue or comment. Do not report an error. + +### Step 2: Parse token-usage.jsonl + +Read the token usage file: + +```bash +cat /tmp/agent-artifacts/sandbox/firewall/logs/api-proxy-logs/token-usage.jsonl 2>/dev/null +``` + +Each line is a JSON object. Example: + +```json +{"model":"claude-sonnet-4-5","input_tokens":1200,"output_tokens":340,"cache_read_input_tokens":500,"cache_creation_input_tokens":100} +``` + +If the file is missing or empty, exit without creating any output. + +### Step 3: Calculate cost + +Aggregate token counts by model across all lines. Use this pricing table (USD per 1M tokens): + +| Model | Input | Output | Cache write | Cache read | +|-------|-------|--------|-------------|------------| +| claude-opus-4-5 | $15.00 | $75.00 | $18.75 | $1.50 | +| claude-sonnet-4-5 | $3.00 | $15.00 | $3.75 | $0.30 | +| claude-haiku-4-5 | $0.80 | $4.00 | $1.00 | $0.08 | +| claude-3-opus | $15.00 | $75.00 | $18.75 | $1.50 | +| claude-3-5-sonnet | $3.00 | $15.00 | $3.75 | $0.30 | +| claude-3-5-haiku | $0.80 | $4.00 | $1.00 | $0.08 | +| gpt-4o | $2.50 | $10.00 | — | $1.25 | +| gpt-4o-mini | $0.15 | $0.60 | — | $0.075 | +| gpt-4.1 | $2.00 | $8.00 | — | $0.50 | +| o3 | $10.00 | $40.00 | — | $2.50 | +| o4-mini | $1.10 | $4.40 | — | $0.275 | +| gemini-1.5-pro | $1.25 | $5.00 | — | — | +| gemini-2.0-flash | $0.10 | $0.40 | — | — | + +For any model not in this table, use $3.00 input / $15.00 output as a conservative fallback. + +Cost per model = (input_tokens * input_rate + output_tokens * output_rate + + cache_creation_input_tokens * cache_write_rate + + cache_read_input_tokens * cache_read_rate) / 1_000_000 + +Total cost = sum across all models. + +Format costs with 4 decimal places: `$0.0123`. Use `< $0.0001` if below that threshold. + +### Step 4: Find the associated pull request + +Check whether the triggering run is linked to a pull request: + +```bash +gh api "repos/${{ github.repository }}/actions/runs/${{ github.event.workflow_run.id }}" \ + --jq '.pull_requests[0].number // empty' +``` + +Store the PR number if one is returned. + +### Step 5: Post the cost report + +Build a report using this template. Fill in `$TOTAL_COST` and the breakdown table: + +```markdown +## Agent run cost + +| | | +|---|---| +| **Workflow** | ${{ github.event.workflow_run.name }} | +| **Run** | [#${{ github.event.workflow_run.run_number }}](${{ github.event.workflow_run.html_url }}) | +| **Conclusion** | ${{ github.event.workflow_run.conclusion }} | +| **Total cost** | $TOTAL_COST | + +
+Token breakdown by model + +| Model | Input | Output | Cache write | Cache read | Cost | +|-------|------:|------:|------------:|-----------:|-----:| +[one row per model with actual token counts and per-model cost] + +
+ +*Data from [token-usage.jsonl](https://github.github.com/gh-aw/reference/token-usage/) written by gh-aw's firewall.* +``` + +**If a PR number was found**: post this as a comment on that PR using the `add_comment` +GitHub tool. + +**If no PR was found**: create an issue using the `create_issue` GitHub tool. Use this +title format: +`[cost-tracker] ${{ github.event.workflow_run.name }} #${{ github.event.workflow_run.run_number }}: $TOTAL_COST` + +### Step 6: High-spend alert (optional) + +If the total cost exceeds **$1.00**, create a second issue using the `create_issue` +GitHub tool with title: +`[cost-tracker] High spend alert: $TOTAL_COST for ${{ github.event.workflow_run.name }}` + +Include the full breakdown and a link to the run. The $1.00 threshold is a conservative +starting point. Edit this workflow to raise or lower it to match your budget. + +## Guidelines + +- **Silent on non-agent runs**: If the artifact does not exist, produce no output at all. +- **One report per run**: Do not create more than one comment or issue per triggering run. +- **Accurate math**: Double-check token counts and cost calculations before posting. +- **No retries**: If the artifact download fails with a transient error, exit silently + rather than retrying — the next run will generate its own report. From 87877ec75de6b5c54316652102be596004f176a0 Mon Sep 17 00:00:00 2001 From: Adam Henson Date: Mon, 20 Apr 2026 11:17:35 -0400 Subject: [PATCH 2/3] docs: add AgentMeter as optional persistent history upgrade path --- docs/cost-tracker.md | 4 ++++ 1 file changed, 4 insertions(+) diff --git a/docs/cost-tracker.md b/docs/cost-tracker.md index af0607b..3ccbe06 100644 --- a/docs/cost-tracker.md +++ b/docs/cost-tracker.md @@ -75,3 +75,7 @@ already there if you are running gh-aw with firewall enabled (the default). - [token-usage.jsonl reference](https://github.github.com/gh-aw/reference/token-usage/) - [gh-aw firewall documentation](https://github.github.com/gh-aw/reference/firewall/) - [CI Doctor workflow](ci-doctor.md) — investigate CI failures automatically + +## Going Further + +Cost Tracker works standalone — no external services required. For teams that want persistent run history, per-repo spend trends, and budget alerts across multiple repos, [AgentMeter](https://agentmeter.app) ingests the same token data via a single Action step and surfaces it in a shared dashboard at agentmeter.app. From 47c4e7f899e4455dd01d56c40a2155d247f2477c Mon Sep 17 00:00:00 2001 From: Adam Henson Date: Mon, 20 Apr 2026 11:19:05 -0400 Subject: [PATCH 3/3] docs: add agentmeter-action YAML snippet to Going Further section --- docs/cost-tracker.md | 13 ++++++++++++- 1 file changed, 12 insertions(+), 1 deletion(-) diff --git a/docs/cost-tracker.md b/docs/cost-tracker.md index 3ccbe06..28d58ef 100644 --- a/docs/cost-tracker.md +++ b/docs/cost-tracker.md @@ -78,4 +78,15 @@ already there if you are running gh-aw with firewall enabled (the default). ## Going Further -Cost Tracker works standalone — no external services required. For teams that want persistent run history, per-repo spend trends, and budget alerts across multiple repos, [AgentMeter](https://agentmeter.app) ingests the same token data via a single Action step and surfaces it in a shared dashboard at agentmeter.app. +Cost Tracker works standalone — no external services required. For teams that want +persistent run history, per-repo spend trends, and budget alerts across multiple repos, +add [AgentMeter](https://agentmeter.app) to your agent workflow: + +```yaml +- uses: agentmeter/agentmeter-action@v1 + with: + api-key: ${{ secrets.AGENTMETER_API_KEY }} +``` + +AgentMeter ingests the same token data and surfaces it in a shared dashboard with +per-repo trend charts and budget alerts.