Export editOutcome/chatFeedback dev data events as a portable EvalPort ResultSet

[adhabnr-ux]

EvalPort (https://github.com/adhabnr-ux/evalport) is an open JSON-Schema spec for portable LLM eval documents — test suites, test cases, and result sets — so eval data isn't locked to one framework's format.

I read through packages/config-yaml/src/schemas/data/ and core/data/log.ts, plus the Development Data docs — the editOutcome and chatFeedback dev data events (written to .continue/dev_data/**/*.jsonl at schema version 0.2.0) are basically eval data already: editOutcome has prompt, completion, modelName, accepted (bool), previousCode/newCode, previousCodeLines/newCodeLines, lineChange, filepath; chatFeedback has prompt, completion, modelName, feedback, sessionId. Your own blog post ("It's time to collect data on how you build software", linked from that docs page) makes basically the EvalPort argument already — that this data is valuable and shouldn't be locked away.

Right now that JSONL is Continue-specific. A converter to EvalPort's TestCase/ResultSet (spec/schemas/testcase.json, spec/schemas/resultset.json) would let a stream of real editOutcome events become a replayable regression suite — same prompts, compare accepted rate across model or prompt-template changes, as plain diffable JSON instead of ad-hoc JSONL analysis:

// editOutcome event -> EvalPort Result
// (chatFeedback maps the same way: prompt -> input, completion -> actual_output,
//  feedback === "positive" -> passed)
function editOutcomeToResult(event: EditOutcomeEvent, testCaseId: string) {
  return {
    test_case_id: testCaseId,
    actual_output: event.completion,
    grader_results: [{
      grader_id: "user_accepted",
      type: "human",
      score: event.accepted ? 1.0 : 0.0,
      passed: event.accepted,
      reason: `lineChange=${event.lineChange}, file=${event.filepath}`,
    }],
    passed: event.accepted,
    metadata: { modelName: event.modelName, filepath: event.filepath },
  };
}

This wouldn't touch the local-first dev-data pipeline at all — it'd be an optional export script/converter reading the existing JSONL and producing an EvalPort ResultSet, useful for anyone wanting to build a regression suite out of real dev-data instead of hand-written eval prompts.

Spec: https://github.com/adhabnr-ux/evalport/blob/main/SPEC.md
Schemas referenced above: https://github.com/adhabnr-ux/evalport/tree/main/spec/schemas

No pressure — happy to sketch a PR if useful, or happy to just leave this here. Thanks for Continue, it's been a great tool.