📄 Read the CoEval paper online · Download the Word version
CoEval: Ranking Language Models for Custom Tasks Without Labeled Data or Trustworthy Benchmarks — read it online (HTML with rendered math) or download the Word version.
Alexander Apartsin (Holon Institute of Technology) · Yehudit Aperstein (Afeka Tel Aviv Academic College of Engineering)
CoEval ranks models for a custom task or domain in the hardest setting: when no task-specific labeled data exists and public benchmarks cannot be trusted because their items have likely leaked into pretraining. From only a task description, a teacher model synthesizes a fresh, contamination-free benchmark and a cross-family judge ensemble ranks the candidates, with no human labels or raters.
| Result | Evidence |
|---|---|
| Recovers the true model ranking with no labeled data | Spearman ρ = 0.86 vs ground-truth correctness, 95% CI [0.77, 0.94] |
| Doubly-robust ranking recovers the true ordering and resists rogue judges | reliability and discrimination weighting lift rank-recovery to Spearman 0.95 across 13 models; an injected random judge receives weight 0.00 |
| Rankings are domain-specific, so a generic leaderboard misleads | three different models top four CoEval-generated domains; the pooled-best model is domain-best in only 1 of 4 |
| Cancels a verbosity bias no single judge avoids | ensemble r = +0.010 (CI spans zero), a 93% reduction |
| Composition over size: panel diversity, not panel size, drives reliability | ICC(3,k) peaks at two well-chosen judges, falls as low-agreement judges are added |
| Structurally precludes same-family self-preference | vendor-disjoint panel; aggregation shifts every score ≤ 0.015 |
| Contamination-free generated items | 0.0000 verbatim 13-gram overlap with five major public benchmarks |
| Inexpensive enough to re-run per model release | 7,978 evaluations for USD 5.89, fully automated |
Evaluating and selecting off-the-shelf or fine-tuned models for a specific use case is difficult.
Choosing the right LLM means navigating a minefield of hidden pitfalls:
| Challenge | Why It Hurts | |
|---|---|---|
| 🎯 | Generic benchmarks don't transfer | Public data and metrics often miss the nuances of your real-world requirements. |
| 🧩 | Custom benchmarks are hard to design | Defining representative tasks, building rubrics, and choosing robustness variations is non-trivial. |
| 💸 | Multi-model multi-task benchmarks are expensive to execute | Running every candidate model across every task and rubric quickly multiplies cost and compute. |
| 🕳️ | Leakage biases results | Public and private benchmark items (or near-duplicates) may lurk in training data, inflating scores via memorization. |
| ⚙️ | Ops and cost are complex | Running evaluations across providers, inference modes, and scoring criteria demands careful orchestration. |
Bottom line: You can't trust a leaderboard number, and building your own eval is a project in itself.
Ensemble-based synthetic self-evaluation benchmarking — let the models evaluate each other.
CoEval generates a synthetic evaluation suite spanning multiple domain-specific tasks and scoring rubrics, then assembles an ensemble of models that rotate through three roles:
┌─────────────────────────────────────────────────────────────┐
│ MODEL ENSEMBLE │
│ │
│ ┌───────────┐ ┌───────────┐ ┌───────────┐ │
│ │ Model A │ │ Model B │ │ Model C │ ... │
│ └─────┬─────┘ └─────┬─────┘ └─────┬─────┘ │
│ │ │ │ │
│ ▼ ▼ ▼ │
│ ┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓ │
│ ┃ ROTATING ROLE ASSIGNMENT ┃ │
│ ┗━━━━━━━━━━┳━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━┳━━━━┛ │
│ ▼ ▼ ▼ │
│ 🎓 TEACHER 📝 STUDENT ⚖️ JUDGE │
│ Generate synthetic Models under Score outputs │
│ challenges & evaluation take against the │
│ reference answers the challenges rubric │
│ │
└─────────────────────────────────────────────────────────────┘
Not all teachers and judges are created equal. CoEval improves signal quality by identifying:
| Role | Selection Criterion | Intuition |
|---|---|---|
| 🎓 Teacher | Differentiating — produces challenges that separate student performance | A good exam question reveals who studied. |
| ⚖️ Judge | Consensus — high agreement with ensemble majority | A reliable judge aligns with peer consensus. |
These two label-free signals (item discrimination and judge agreement) combine into CoEval's doubly-robust ranking, which recovers the true model ordering at Spearman 0.95 across 13 candidate models and assigns an injected rogue judge zero weight. Because the best model is domain-specific, CoEval ranks on your domain rather than trusting a pooled leaderboard.
Fully Automatic Semi-Automatic Manual
┌────────────┐ ┌────────────────┐ ┌──────────────┐
│ Tasks │ │ Tasks ✏️ │ │ Tasks ✏️ │
│ Rubrics │ ──► │ Rubrics │ ──► │ Rubrics ✏️ │
│ Attr. Space │ │ Attr. Space ✏️ │ │ Attr. Space ✏️│
└────────────┘ └────────────────┘ └──────────────┘
AI-generated Human-guided Human-defined
Tasks, rubrics, and diversity/attribute spaces can be provisioned fully automatically, semi-automatically (human-in-the-loop), or manually — choose the level of control that fits your workflow.
CoEval is an end-to-end system — from benchmark design to interactive reporting.
╔══════════════════════════════════════════════════════════════╗
║ C o E v a l ║
╠══════════════════════════════════════════════════════════════╣
║ ║
║ 📦 Multi-Vendor Support ║
║ ├── Multiple LLM providers & interfaces out of the box ║
║ └── Plug in proprietary / self-hosted models ║
║ ║
║ 🗺️ Benchmark Design & Planning ║
║ ├── Automated task & rubric provisioning ║
║ └── Run orchestration with cost optimization ║
║ ║
║ 📊 Interactive Visual Reports ║
║ ├── Side-by-side model comparison ║
║ └── Drill-down into tasks, rubrics & scores ║
║ ║
║ 🔄 Experiment Tracking ║
║ ├── Easy reruns & parameter sweeps ║
║ └── Repair & resume after interruptions ║
║ ║
║ 📚 Complete Documentation ║
║ ├── User guides & tutorials ║
║ └── Developer API reference ║
║ ║
╚══════════════════════════════════════════════════════════════╝
| Feature | Description |
|---|---|
| Multi-vendor | Swap providers without changing your eval pipeline. |
| Auto-provisioning | Generate tasks, rubrics, and attribute spaces from a domain description. |
| Orchestration | Schedule and parallelize runs; optimize for cost and latency. |
| Visual reports | Interactive dashboards for deep-dive analysis. |
| Resilient tracking | Resume interrupted experiments; repair partial results. |
| Docs-first | Comprehensive guides for users and contributors alike. |
OpenAI, Anthropic, Google Gemini, Azure OpenAI, Azure AI Inference, AWS Bedrock, Google Vertex AI, OpenRouter, Groq, DeepSeek, Mistral, DeepInfra, Cerebras, Cohere, HuggingFace API, HuggingFace (local), Ollama
→ Providers & Pricing — auth setup, batch discounts, pricing tables for all 18 interfaces.
# 1. Install
pip install coeval
# 2. Add your API keys (see: docs/tutorial.md § 2)
cp keys.yaml.template keys.yaml # then fill in your provider keys
# 3. Probe all models — no tokens consumed (runnable example included in the repo)
coeval probe --config examples/quickstart.yaml
# 4. Estimate cost before spending anything
coeval plan --config examples/quickstart.yaml
# 5. Run the experiment (phases 1-5: infer attributes + rubric, generate, respond, judge)
coeval run --config examples/quickstart.yaml
# 6. Generate analysis reports
coeval analyze all --run ./Runs/quickstart --out ./Runs/quickstart/reportsCoEval accepts your intent at whichever level of detail you have, from a single sentence to a fully hand-written config:
| Level | You provide | CoEval infers | How |
|---|---|---|---|
| Objective | one-line goal | everything: tasks, attributes, rubric, model roles | coeval wizard --objective "..." |
| Most-automatic | task description + models | target attributes + rubric (Phases 1-2) | hand-write a minimal YAML (below) |
| Semi-automatic | description + some attributes/rubric | the rest | partial YAML, human-in-the-loop wizard |
| Manual | full config | nothing | complete YAML |
Generate a complete, runnable config from a single high-level objective (no questions asked) and run it:
# One sentence in, a validated config out
coeval wizard \
--objective "rank LLMs on classifying customer-support tickets into urgency levels" \
--models "gpt-4o-mini, claude-3-5-haiku" \
--items 8 \
--out ticket_urgency.yaml
coeval run --config ticket_urgency.yamlThe LLM proposes the tasks, target attributes, scoring rubric, and a
cross-family judge panel; the config is auto-validated (and auto-repaired on any
validation error) before it is written. Omit --objective for the interactive,
question-by-question wizard instead.
You give only a task description and the models; CoEval infers the target attributes
and the scoring rubric. See the complete runnable file at examples/quickstart.yaml.
models:
- name: gpt-4o-mini
interface: openrouter
parameters: { model: openai/gpt-4o-mini, temperature: 0.7, max_tokens: 512 }
roles: [teacher, student, judge]
- name: claude-haiku
interface: openrouter
parameters: { model: anthropic/claude-3.5-haiku, temperature: 0.0, max_tokens: 128 }
roles: [judge] # cross-family judge
tasks:
- name: regex_explanation
description: Explain in plain English what a given regular expression matches.
output_description: A clear one-to-three sentence plain-English explanation.
sampling: { total: 6 } # target_attributes + rubric are inferred (Phases 1-2)
evaluation_mode: single
experiment:
id: quickstart
storage_folder: ./eval_runsInteractive HTML examples — click to open rendered in browser:
| Example | Description |
|---|---|
| Education Benchmark — Planning View | Full experiment plan: 3 real-dataset tasks + 10 synthetic tasks, 6 models, per-phase call budget, cost table, and attribute maps |
| Mixed Benchmark — Planning View | Mixed benchmark plan: real benchmark datasets + OpenAI models |
| Paper Dual-Track — Planning View | Paper evaluation: dual-track design with benchmark + generative teachers |
Generate your own planning view:
coeval describe --config my_experiment.yaml --out my_experiment_plan.html
| Report | Description |
|---|---|
| Dashboard | Overview dashboard — all reports in one place with top-line rankings and navigation |
| Student Performance Report | Per-student score breakdowns, task rankings, rubric factor heatmaps |
| Judge Consistency Report | Inter-judge ICC agreement, calibration drift, flagged uncertain items |
| Robust Summary Report | Final model rankings with confidence intervals and robust ensemble weights |
| Score Distribution Report | High / Medium / Low histograms filterable by task, teacher, student, and judge |
| Teacher Report | Per-teacher source quality, attribute stratum coverage, data consistency |
| Interaction Matrix | Teacher × Student pair quality heatmap — spot which combinations succeed or fail |
| Coverage Summary | Attribute Coverage Ratio (ACR) and rare-attribute recall per task |
| Judge Report | Judge-level bias rates, score calibration, inter-rater reliability |
| Annotated Report Guide | Detailed annotated screenshots of every CoEval report with explanations of every visualization and metric |
Generate all reports from a completed run:
coeval analyze all --run ./Runs/my-experiment-v1 --out ./reports
| Guide | What it covers |
|---|---|
| Concepts Glossary | Every first-class concept explained: teacher, student, judge, attributes, rubric, datapoint, slot, phases, wizard, probing, planning, resume, repair, auto interface, batch API, and more |
| Evaluation Experiment Planning and Preparation Guide | End-to-end walkthrough: installation, config design, probing, running, analysis, and benchmark export |
| Command Line Option Reference | Every coeval subcommand, flag, and exit code — run, probe, plan, generate, status, models, analyze, describe, wizard, ingest, repair |
| Running Experiments | Phase modes, --continue, batch API, quota control, cost estimation, fault recovery, use-case examples |
| Providers & Pricing | All 18 interfaces with auth, batch support, code examples, and pricing tables |
| Analytics & Reports | 11 interactive HTML dashboards, paper-quality result tables, programmatic API, Excel workbook export |
| Configuration Guide | YAML config schema: models, tasks, attributes, rubric, sampling, prompt overrides, experiment settings |
| Benchmark Datasets | Pre-ingested datasets, coeval ingest, interface: benchmark virtual teacher, reproducing published results |
| Testing Guide | All 20 test files, how to run each suite, interpreting failures, CI/CD setup |
| System Feature Wishlist | 35-item prioritized roadmap: 10 benchmark additions, 12 system features, 13 new report types |
YAML Config → Phase 1: Attribute Mapping (teachers infer task dimensions)
→ Phase 2: Rubric Mapping (teachers build evaluation criteria)
→ Phase 3: Data Generation (teachers produce benchmark items)
→ Phase 4: Response Collection (students answer benchmark prompts)
→ Phase 5: Evaluation (judges score student responses)
→ coeval analyze all (8 HTML reports + Excel workbook)
| Cloud — Async Batch ✅ | Cloud — Real-time | OpenAI-Compatible | Local / Virtual |
|---|---|---|---|
openai |
azure_openai¹ |
groq |
huggingface |
anthropic |
azure_ai |
deepseek |
ollama |
gemini² |
bedrock |
mistral |
benchmark |
vertex |
deepinfra |
||
openrouter |
cerebras |
¹
azure_openaisupports Azure Global Batch API (50% discount) — enable viabatch: azure_openai:in config. ²geminiuses concurrent requests (pseudo-batch) — no async discount.
| Capability | Detail |
|---|---|
| Cost estimation | Itemised call budget and cost table before any phases run; Batch API discounts modelled |
| Batch API | 50% async discount for OpenAI, Anthropic, and Azure OpenAI; Gemini uses concurrent mode (no discount) |
| Resume | --continue resumes at exact JSONL record; no duplicate API calls |
| Auto attributes | Teachers infer task dimensions from a description; no hand-labelling required |
| Auto rubric | Teachers propose rubric factors; merge-and-deduplicate across N teachers |
| Multi-judge ensemble | N judges → bias-resistant aggregate scores; outlier judges down-weighted |
| 8 HTML reports | Interactive charts, filterable tables, CSV export, fully self-contained (no CDN) |
| Model probe | Verify all 16 interfaces are reachable before spending a dollar |
| Virtual teachers | Pre-ingested public datasets supply zero-cost Phase 3 ground truth |
| Label accuracy | Judge-free exact-match for classification tasks (label_attributes) |
| Component | Files | LoC |
|---|---|---|
Code/runner — pipeline engine |
59 .py |
15,087 |
Code/analyzer — analysis & reports |
21 .py |
9,554 |
Public/benchmark — dataset utilities |
34 .py |
5,211 |
Tests — test suites |
41 .py |
16,845 |
docs — documentation |
35 .md |
12,521 |
CoEval · Multi-Model LLM Evaluation Framework
Designed for LLM developers, integrators, and evaluation practitioners who require robust model evaluation and ranking using custom use-case data and metrics.
Copyright (c) 2026 Alexander Apartsin. All rights reserved.