feat(harbor): the agent's first baseline eval is budget-free

[shehabyasser-scale]

Stacked on #20 (top of the sidecar chain: #11 -> #19 -> #20 -> this).

The problem, observed twice in live trials

The seeded baseline is the reference every candidate is implicitly compared to, yet it can never win selection (auto_best excludes base_commit, correctly, so "do nothing" cannot win). Metering its evaluation therefore forces the optimizer into a bad choice:

1. Pay for it: the incident behind fix(harbor): floor rewards on no-candidate outcome instead of erroring #11/docs(harbor): warn in the task instruction that baseline evals create no candidate #12, an optimizer spent its whole budget measuring the baseline, left the candidate pool empty, and the trial died (now floored by fix(harbor): floor rewards on no-candidate outcome instead of erroring #11, warned against by docs(harbor): warn in the task instruction that baseline evals create no candidate #12).
2. Skip it and fly blind: today's TB2 trial (exp5 v3). The optimizer, warned off baseline evals by docs(harbor): warn in the task instruction that baseline evals create no candidate #12's instruction, never learned the baseline scored 0.375. Its first edit scored 0.375, and with no reference it could not tell a no-op from an improvement; its second edit scored 0.125; it rationally gave up with 3 of its 5 budgeted evals unspent. Final reward: 0.375, exactly baseline.

#12 treated the symptom (don't waste budget); this treats the cause (the reference measurement should not cost budget).

The fix

EvaluationSidecar.evaluate: when the transferred commit resolves to base_commit and the free eval hasn't been used, run it through the engine's unmetered (admin) path while still routing results through the agent's real split tier. Capped at exactly one, so free compute is bounded; subsequent baseline evals debit normally. /status now reports base_commit and free_baseline_available, so agents can discover the mechanism.

Selection integrity is untouched: the baseline experiment is recorded as before and remains excluded from auto_best by the existing base_commit filter.

Tests

Seven new tests: first baseline eval unmetered but tier-routed; second one metered; non-baseline commits always metered; no-base_commit tasks never free; /status surfaces and flips the flag.

Follow-up (separate, compiler layer): update the #12 instruction paragraph to say the first baseline eval is free once this lands.

🤖 Generated with Claude Code

Greptile Summary

This PR makes the agent's first evaluation of the seeded baseline commit budget-free by routing it through the engine's admin (unmetered) path while still writing results to the agent volume via the normal tier. The flag is consumed only after a successful engine response so transient failures don't permanently burn the free slot, and /status now exposes base_commit and free_baseline_available so agents can discover the mechanism.

• server.py: adds _free_baseline_used instance flag; evaluate detects baseline-sha match and passes admin=True to the engine while keeping admin=False for result routing; status() forwards the new fields.
• protocol.py / serve.py: additive plumbing — StatusSummary gains base_commit/free_baseline_available; ServeConfig.base_commit is threaded into the sidecar constructor.
• test_harbor_server.py: seven new tests cover the happy path, second-call metering, non-baseline commits, no-base_commit tasks, status surfacing, admin isolation, and engine-failure resilience.

Confidence Score: 4/5

Safe to merge for sequential agents; a concurrency edge case in server.py should be addressed before the sidecar is exposed to parallel request patterns.

The core logic is well-reasoned and the test suite is thorough, but the "capped at one" free-baseline guarantee in server.py can be broken when two evaluate coroutines for the same baseline are in-flight simultaneously — both read _free_baseline_used = False before either writes True, so both are served as unmetered admin calls. The safety comment in the code focuses on the assignment site and is correct there, but misses the window opened by the earlier read being separated from the write by an await. In a single-agent, sequential-request context the practical risk is low, but the invariant is explicitly stated as a design constraint and the reasoning justifying its safety is incorrect.

vero/src/vero/harbor/server.py — specifically the _free_baseline_used read/write pattern around the await self.engine.evaluate() call.

Important Files Changed

Filename	Overview
vero/src/vero/harbor/server.py	Adds free-baseline logic and surfaces base_commit in status; contains a concurrency race where two concurrent baseline evals can both be served as unmetered admin calls, violating the stated "capped at one" guarantee.
vero/src/vero/harbor/protocol.py	Adds base_commit and free_baseline_available fields to StatusSummary and build_status; clean additive change.
vero/src/vero/harbor/serve.py	Threads base_commit from ServeConfig into EvaluationSidecar; single-line plumbing change, no issues.
vero/tests/test_harbor_server.py	Adds 7 well-structured tests covering first-use unmetered, second-use metered, non-baseline commit, no base_commit, status surfacing, admin-call isolation, and transient failure resilience; addresses both issues flagged in the previous review round.

Sequence Diagram

%%{init: {'theme': 'neutral'}}%%
sequenceDiagram
    participant Agent
    participant Sidecar as EvaluationSidecar
    participant Engine as EvaluationEngine

    Note over Sidecar: _free_baseline_used = False

    Agent->>Sidecar: "evaluate(commit=base_commit) [1st call]"
    Sidecar->>Sidecar: "_transfer_commit → sha == base_commit"
    Sidecar->>Sidecar: "free_baseline = True"
    Sidecar->>Engine: "evaluate(admin=True) [unmetered]"
    Engine-->>Sidecar: Experiment
    Sidecar->>Sidecar: "_free_baseline_used = True"
    Sidecar->>Sidecar: "_route_results(admin=False) → write to agent volume"
    Sidecar-->>Agent: EvalSummary (budget_remaining reported)

    Agent->>Sidecar: /status
    Sidecar-->>Agent: "free_baseline_available=False"

    Agent->>Sidecar: "evaluate(commit=base_commit) [2nd call]"
    Sidecar->>Sidecar: "free_baseline = False (_free_baseline_used=True)"
    Sidecar->>Engine: "evaluate(admin=False) [metered]"
    Engine-->>Sidecar: Experiment
    Sidecar-->>Agent: EvalSummary (budget decremented)

    Agent->>Sidecar: "evaluate(commit=candidate_sha)"
    Sidecar->>Engine: "evaluate(admin=False) [always metered]"
    Engine-->>Sidecar: Experiment
    Sidecar-->>Agent: EvalSummary

Loading

%%{init: {'theme': 'base', 'themeVariables': {"darkMode": true, "background": "#0d1117", "primaryColor": "#21262d", "primaryTextColor": "#e6edf3", "primaryBorderColor": "#8b949e", "lineColor": "#8b949e", "textColor": "#e6edf3", "edgeLabelBackground": "#161b22", "actorBkg": "#21262d", "actorBorder": "#8b949e", "actorTextColor": "#e6edf3", "actorLineColor": "#8b949e", "signalColor": "#8b949e", "signalTextColor": "#e6edf3", "noteBkgColor": "#373320", "noteBorderColor": "#d4a72c", "noteTextColor": "#f0e6c0", "labelBoxBkgColor": "#21262d", "labelBoxBorderColor": "#8b949e", "labelTextColor": "#e6edf3", "loopTextColor": "#e6edf3", "activationBkgColor": "#30363d", "activationBorderColor": "#8b949e"}}}%%
sequenceDiagram
    participant Agent
    participant Sidecar as EvaluationSidecar
    participant Engine as EvaluationEngine

    Note over Sidecar: _free_baseline_used = False

    Agent->>Sidecar: "evaluate(commit=base_commit) [1st call]"
    Sidecar->>Sidecar: "_transfer_commit → sha == base_commit"
    Sidecar->>Sidecar: "free_baseline = True"
    Sidecar->>Engine: "evaluate(admin=True) [unmetered]"
    Engine-->>Sidecar: Experiment
    Sidecar->>Sidecar: "_free_baseline_used = True"
    Sidecar->>Sidecar: "_route_results(admin=False) → write to agent volume"
    Sidecar-->>Agent: EvalSummary (budget_remaining reported)

    Agent->>Sidecar: /status
    Sidecar-->>Agent: "free_baseline_available=False"

    Agent->>Sidecar: "evaluate(commit=base_commit) [2nd call]"
    Sidecar->>Sidecar: "free_baseline = False (_free_baseline_used=True)"
    Sidecar->>Engine: "evaluate(admin=False) [metered]"
    Engine-->>Sidecar: Experiment
    Sidecar-->>Agent: EvalSummary (budget decremented)

    Agent->>Sidecar: "evaluate(commit=candidate_sha)"
    Sidecar->>Engine: "evaluate(admin=False) [always metered]"
    Engine-->>Sidecar: Experiment
    Sidecar-->>Agent: EvalSummary

Loading

Prompt To Fix All With AI

Fix the following 1 code review issue. Work through them one at a time, proposing concise fixes.

---

### Issue 1 of 1
vero/src/vero/harbor/server.py:83-98
**Race window defeats the "capped at one" guarantee**

The `_free_baseline_used` read at line 87 (inside `free_baseline = (... and not self._free_baseline_used)`) is separated from the write at line 98 by `await self.engine.evaluate(...)`. If two concurrent `evaluate` coroutines arrive while none have completed yet, both will read `False`, both will set `free_baseline = True`, and both will be served as admin calls — violating the explicit "capped at exactly one" design goal.

The comment "no await runs between the check above and this write" is true about the `if free_baseline: …` assignment site alone, but that is not the relevant check. The check that controls the boolean is `not self._free_baseline_used` in the `free_baseline` tuple expression, and that read is separated from the write by a full `await`. In a single-threaded asyncio event loop two coroutines can both be suspended at `await self.engine.evaluate(...)` having already recorded `free_baseline = True`.

Fixing this requires setting the flag **before** the await (then restoring it on exception), or guarding it with a sentinel that is set synchronously before yielding — the mirror of the reasoning that justified moving the assignment *after* the await in the previous review.

_{Reviews (2): Last reviewed commit: "fix(harbor): consume the free baseline s..." | Re-trigger Greptile}

Greptile also left 1 inline comment on this PR.

[shehabyasser-scale]

Nit found by the review panel for #26 while verifying this PR's semantics: _free_baseline_used is a plain in-memory bool, while the budget ledger is persisted to <admin_volume>/ledger.json precisely so a sidecar restart cannot refund budget. After a restart the freebie re-arms: a second baseline eval is free and status reports free_baseline_available: true again. Agent-favorable (bounded free compute, never misleads the agent into wasting budget, instruction and status stay mutually consistent), so not a blocker; a follow-up could persist the flag next to ledger.json.

[greptile-apps]

Race window defeats the "capped at one" guarantee

The _free_baseline_used read at line 87 (inside free_baseline = (... and not self._free_baseline_used)) is separated from the write at line 98 by await self.engine.evaluate(...). If two concurrent evaluate coroutines arrive while none have completed yet, both will read False, both will set free_baseline = True, and both will be served as admin calls — violating the explicit "capped at exactly one" design goal.

The comment "no await runs between the check above and this write" is true about the if free_baseline: … assignment site alone, but that is not the relevant check. The check that controls the boolean is not self._free_baseline_used in the free_baseline tuple expression, and that read is separated from the write by a full await. In a single-threaded asyncio event loop two coroutines can both be suspended at await self.engine.evaluate(...) having already recorded free_baseline = True.

Fixing this requires setting the flag before the await (then restoring it on exception), or guarding it with a sentinel that is set synchronously before yielding — the mirror of the reasoning that justified moving the assignment after the await in the previous review.

Prompt To Fix With AI

This is a comment left during a code review.
Path: vero/src/vero/harbor/server.py
Line: 83-98

Comment:
**Race window defeats the "capped at one" guarantee**

The `_free_baseline_used` read at line 87 (inside `free_baseline = (... and not self._free_baseline_used)`) is separated from the write at line 98 by `await self.engine.evaluate(...)`. If two concurrent `evaluate` coroutines arrive while none have completed yet, both will read `False`, both will set `free_baseline = True`, and both will be served as admin calls — violating the explicit "capped at exactly one" design goal.

The comment "no await runs between the check above and this write" is true about the `if free_baseline: …` assignment site alone, but that is not the relevant check. The check that controls the boolean is `not self._free_baseline_used` in the `free_baseline` tuple expression, and that read is separated from the write by a full `await`. In a single-threaded asyncio event loop two coroutines can both be suspended at `await self.engine.evaluate(...)` having already recorded `free_baseline = True`.

Fixing this requires setting the flag **before** the await (then restoring it on exception), or guarding it with a sentinel that is set synchronously before yielding — the mirror of the reasoning that justified moving the assignment *after* the await in the previous review.

How can I resolve this? If you propose a fix, please make it concise.