Co-Scientist Bench: Evaluation System

What the Bench Does

co-scientist bench runs the same research goal under N different (provider, model) configurations and ranks them via a single shared Elo tournament. Each candidate independently generates hypotheses through the full Generation pipeline; then every candidate-pair plays head-to-head debates, judged by a fixed judge model.

The bench is isolated from regular sessions — it does not write to tournament_matches or affect any session's leaderboard. All results are stored in dedicated bench_* SQLite tables and per-session artifact files.

How It Works

flowchart TD
    A[co-scientist bench --preset paper-aml] --> B[Parse candidates]
    B --> C[Generation runs in parallel per candidate]
    C --> D[Each candidate: full Generation pipeline<br/>literature tools + tool loop + dedup + record_hypothesis]
    D --> E[Round-robin pairings: every pair plays N matches]
    E --> F[Judge model evaluates each match<br/>forced record_verdict function call]
    F --> G[Structured verdict: no fragile text parsing]
    G --> H[Elo tournament with shared pool]
    H --> I[Gold-set scoring (if applicable)]
    I --> J[Results written to SQLite + JSON + auto-generated report]

Isolation

Bench runs use a deep-copied Config per candidate:

Each candidate gets its own cfg.llm.provider and cfg.models.*
Thinking budgets are zeroed for non-Anthropic providers
The judge model is picked separately so no candidate scores its own work

Generation

Each candidate runs the full Generation agent stack:

Literature tools (PubMed, ArXiv, Europe PMC, web search, web fetch)
Tool loop with dedup
record_hypothesis with structured output

The --n flag controls hypotheses per candidate (default varies by preset).

Tournament

Every pair of candidates plays --matches head-to-heads:

One random hypothesis from each side per match
Judge model receives both hypotheses + reviews + a forced record_verdict function call
Structured verdict via record_verdict — no fragile text parsing across providers

Cost Control

Each candidate has a --budget-per-candidate cap. The bench estimates costs via PRICE_TABLE and aborts individual candidates that exceed their budget.

Presets

Preset	Description	Candidates	Gold Set	Pipeline vs Raw
`paper`	Paper baselines via OpenRouter	Gemini 2 Flash Thinking, Gemini 2 Pro, o1, Haiku	None	No
`paper-aml`	Paper baselines + AML drug-repurposing goal + gold-set scoring	Same as `paper`	`aml-repurposing-paper-top3` (3 drugs)	No
`paper-aml-vs-raw`	`paper-aml` + raw LM comparison	Same + raw variants	`aml-repurposing-paper-top3`	Yes
`frontier-aml-vs-raw`	Current frontier models, pipeline vs raw	Claude Opus 4.7, GPT-5, Gemini 3 Pro/Flash	`aml-repurposing-paper-top3`	Yes

Custom Candidates

co-scientist bench "Identify hypotheses about X" \
  -c flash3=openrouter:google/gemini-3-flash-preview \
  -c flash3-raw=openrouter:google/gemini-3-flash-preview@direct \
  -c gpt5=openai:gpt-5 \
  -c opus=anthropic:claude-opus-4.7 \
  --judge anthropic:claude-sonnet-4-6

Format: label=provider:model[@mode]. Mode is pipeline (default) or direct (single forced-tool LM call with no tools).

Gold-Set Scoring

AML Drug Repurposing

The paper-aml* presets score recall against a curated answer key from the Co-Scientist paper.

Gold Set	Size	Drugs
`aml-repurposing-paper-top3` (default)	3	Nanvuranlat (JPH-203), KIRA6, Leflunomide
`aml-repurposing-paper-5`	5	Binimetinib, Pacritinib, Cerivastatin, Pravastatin, Dimethyl fumarate

Selection criteria for top-3: No prior published AML repurposing, no prior preclinical evidence in AML, no external inputs (no DepMap scores, no expert curation).

Matching Rules

Whole-token, case-insensitive matching
Searches every field: title, summary, full_text, entities, citation excerpts
Drug class mentions (e.g., "DHODH inhibitor") do NOT count — the candidate must name the actual compound or a registered alias
Post-hoc rescore: a bench that ran with top3 at runtime can still show results against paper-5

Swap Command

co-scientist bench --preset paper-aml --goldset aml-repurposing-paper-5   # broader list
co-scientist bench --preset paper-aml --goldset none                       # head-to-head only

Pipeline vs Raw LM Comparison

The *-vs-raw presets isolate the multi-agent harness's value-add:

Mode	What Runs	Tools
`pipeline`	Full Generation agent stack	Literature tools + tool loop + dedup
`direct`	Single forced `record_hypothesis` function call	None

Both modes use the same model + the same goal. The Elo delta between pipeline and direct mode measures how much of the output quality comes from the harness vs the underlying model.

Headline Findings (from BENCH_RESULTS.md)

After pipeline reliability fixes, pipeline mode completes for essentially every candidate. But the direct -> pipeline Elo delta is not reproducible at this sample size:

Model	Run 1 Delta	Run 2 Delta
claude-haiku-4.5	+180	-28
openai-o1	+43	+29

The delta flips sign for Haiku across two identical runs. Single-run deltas for other models are all over the map: Opus +97, Gemini 2.5 Flash +172, Gemini 3 Pro -89. Within Google alone, the 2.5 models gain and the 3.x models lose — no clean "provider" or "strength" story.

Conclusion: The harness reliably produces a hypothesis (pipeline completes), but whether it helps requires many more seeds to average out single-hypothesis variance.

Cross-Model Consistency

Across 48 AML hypotheses, agreement is at the mechanism level, not the compound level:

Recurring Theme	Count
Leukemic stem cell (LSC) targeting	28
OXPHOS / mitochondrial complex I	8
BCL-2 / MCL-1 (Venetoclax axis)	7
FLT3-ITD	6
Fatty acid oxidation	5
Ferroptosis	3

Only two compounds proposed more than once: Itraconazole (x5) and Auranofin (x2). The system converges on mechanisms, not specific drugs.

Bench Results Infrastructure

Storage

data/co_scientist.db
  bench_runs              one row per bench
  bench_candidates        one row per (bench x candidate x mode)
  bench_matches           one row per head-to-head

data/artifacts/<session_id>/
  bench/<bench_id>.json              run summary + gold_hit_detail
  hypotheses/<hyp_id>.json           every hypothesis produced
  transcripts/generation/<trn_id>.json  every LLM call

Auto-Generated Report

python scripts/build_bench_report.py reads the bench DB and generates docs/BENCH_RESULTS.md:

Headline findings section
Index of all 20+ benches
Per-bench detail with candidate tables, hypothesis statements, gold-set rescore
File pointers and ready-to-run SQL queries

SQL Queries

Every bench entry includes copy-paste SQL for inspection:

-- Per-candidate detail
SELECT label, mode, n_hypotheses, wins, losses,
       round(mean_elo,0), gold_hits, gold_hit_names,
       round(total_cost_usd, 4),
       total_input_tok, total_output_tok
  FROM bench_candidates
 WHERE bench_id='<bench_id>';

-- Every match with judge rationale
SELECT bc_a.label, bc_b.label, bm.winner,
       round(bm.judge_cost_usd, 4),
       substr(bm.rationale, 1, 200)
  FROM bench_matches bm
  JOIN bench_candidates bc_a ON bc_a.id = bm.cand_a
  JOIN bench_candidates bc_b ON bc_b.id = bm.cand_b
 WHERE bm.bench_id='<bench_id>';

Rubric Eval Bundle

co-scientist eval [agent] runs a per-agent and end-to-end evaluation suite:

Offline mode available (uses cached transcripts, no live LLM calls)
Tests structured output compliance (does record_hypothesis produce all required fields?)
Tests citation integrity (are all URLs in seen_urls?)
Tests state machine transitions (draft -> reviewed -> in_tournament)
Tests dedup behavior (does near-duplicate get skipped?)
Tests budget enforcement (does BudgetExceeded fire at the right threshold?)

Cost Summary (from Recorded Benches)

Bench Kind	Typical Cost	Candidates	Matches
Microbiome smoke test	$0.001 -$ 0.40	2-4	0-12
AML repurposing (no gold)	$0.01 -$ 0.99	2-8	0-56
AML repurposing (with gold)	$0.83 -$ 2.08	4-8	12-56

The heaviest bench ran 8 candidates with 56 matches for $2.08 — demonstrating that cross-model comparison is economically feasible.

Relationship to unit-distance-elo-ranking

The bench system uses the same Elo tournament pattern as the unit-distance project's ELO-RANKER: pairwise comparison with K-factor scheduling, debate-style evaluation, and structured verdict capture. The key difference is that the bench evaluates models while the unit-distance ELO-RANKER evaluated hypotheses.

Both systems share the insight that Elo converges faster with more matches per pair — the bench defaults to --matches 2 while the unit-distance project used higher match counts for stability.

References

See co-scientist for the system overview
See co-scientist-agents for the Ranking agent's Elo implementation
See co-scientist-infrastructure for the price table and cost estimation
See docs/BENCH_RESULTS.md for all recorded bench results