Co-Scientist Elo Tournament System

The Co-Scientist uses a standard Elo rating system to rank competing scientific hypotheses through LLM-mediated head-to-head matches. This is fundamentally different from the unit-distance-elo-ranking system (which rates hypotheses by mathematical novelty against known bounds) -- here, Elo measures relative scientific quality as judged by LLM experts in simulated peer review.

Core Elo Mathematics

Implemented in co_scientist/orchestrator/elo.py -- pure math with no I/O.

Expected Score

Standard logistic function:

E(A) = 1 / (1 + 10^((rating_B - rating_A) / 400))

K-Factor

Dynamic K-factor based on match experience:

def k_factor(matches_played, *, new_threshold=5, k_new=32, k_warm=16):
    return k_new if matches_played < new_threshold else k_warm

• K=32 for hypotheses with fewer than 5 matches (fast convergence for new entrants)
• K=16 for seasoned hypotheses (slower movement, more stability)

The K-factor is decided by the less experienced player in each match, ensuring new hypotheses can climb or fall quickly.

Elo Update

delta = K * (S_actual - S_expected)
rating_A_after = rating_A + delta
rating_B_after = rating_B - delta

Updates are zero-sum: one hypothesis gains, the other loses by the same amount. The system uses EloUpdate dataclass with fields: elo_a_after, elo_b_after, expected_a, k.

Match ID Determinism

Match IDs are computed deterministically from sha256(min(a,b) || max(a,b) || round_id) where round_id equals the task ID. This makes Elo updates idempotent: if a crash occurs mid-update, retrying with the same task produces the same match_id and the elo_journal table's UNIQUE constraint prevents double-application.

---

Tournament Data Model

Defined in co_scientist/storage/schema.sql and co_scientist/models/tournament.py:

Tables

tournament_matches -- Each head-to-head comparison:

Column	Type	Purpose
id	TEXT PK	Deterministic sha256
session_id	TEXT FK	Session scope
hyp_a	TEXT FK	First hypothesis
hyp_b	TEXT FK	Second hypothesis
mode	TEXT	pairwise or debate
winner	TEXT	a, b, or NULL (invalid)
elo_a_before/after	REAL	Elo snapshots
elo_b_before/after	REAL	Elo snapshots
rationale	TEXT	Full debate/pairwise text
similarity	REAL	Cosine similarity of hypothesis embeddings

elo_journal -- Append-only ledger. UNIQUE on match_id makes updates idempotent.

Hypothesis States

Hypotheses progress through states in the hypotheses table:

draft -> reviewed -> in_tournament -> pinned/rejected/quarantined/retired

Only in_tournament and pinned hypotheses participate in matches. Initial Elo is configurable (default 1200).

---

Pair Selection Strategy

The RankingAgent's _select_pair() method implements a three-bucket probabilistic selection system using FAISS embeddings for idea-space distance:

Bucket 1: New Arrival Pairing (probability p_new = 0.4)

Pair a new hypothesis (matches_played < 3) with the nearest-Elo warm hypothesis (matches_played >= 3). This ensures new hypotheses get compared against established ones quickly.

Bucket 2: Close-Elo Distant-Idea (probability p_close = 0.4)

Among warm hypotheses with |delta_Elo| < 200, sample pairs weighted by:

weight = exp(-delta_Elo / 200) * (1 - cosine_similarity)

This favors pairs that are close in rating but distant in idea-space -- maximizing information gain from the debate. The FAISS store provides cosine similarity via single-row reconstruction (O(2*dim) per pair, not full index scan).

Bucket 3: Random Top-Heavy (probability p_random = 0.2)

Sample randomly from the top half of hypotheses sorted by Elo. This prevents the system from only matching close competitors and allows occasional cross-tier comparisons.

Focus Mode

When a task carries a focus ID (typically a newly added hypothesis), pair selection overrides the bucket system: it pairs the focused hypothesis with its nearest-Elo opponent. This ensures every new hypothesis immediately enters the tournament.

FAISS Store Optimization

The FAISS index is loaded once per _select_pair() call (not per candidate). Previous implementations re-instantiated the embedder and reconstructed the full index for every pair evaluation, resulting in ~20 full-index reloads per match. The current implementation reconstructs only the two rows needed for cosine similarity.

---

Match Mode Selection

The _select_mode() method chooses between pairwise and debate:

def _select_mode(self, a, b):
    if min(a.matches_played, b.matches_played) < debate_when_matches_lt:  # default 2
        return "debate"
    if abs(a.elo - b.elo) < debate_when_elo_delta_lt:  # default 50
        return "debate"
    return "pairwise"

Condition	Mode	Reasoning
Either hypothesis has < 2 matches	debate	New hypotheses need thorough discussion
Elo gap < 50	debate	Close races need nuanced comparison
Otherwise	pairwise	Fast, decisive comparison

Pairwise vs. Debate

Pairwise (ranking_pairwise.md): Single LLM call, direct comparison. The model receives both hypotheses with reviews and must end with "better idea: 1" or "better idea: 2". Fast (one API call) but less nuanced.

Debate (ranking_debate.md): Multi-turn simulated expert panel (3-5 turns, max 10). The model summarizes both hypotheses, asks clarifying questions, identifies weaknesses, and delivers a final judgment. More expensive but produces better reasoning for close calls.

Both modes use the same termination format: the regex _VERDICT_DIGIT_RE = r"^[\W_]*\**\s*([12])\b" parses the trailing "better idea: X" from the response, with fallback matching for "option 1", "hypothesis 1", "hyp 1" variants.

---

Tournament Flow

flowchart TD
    A[New Hypothesis Created] --> B[Review Completed]
    B --> C[AddToTournament]
    C --> D[Initialize Elo = 1200]
    D --> E[RunTournamentBatch with focus]
    E --> F{Select Pair}
    F -->|New + Warm| G[Bucket 1]
    F -->|Close Elo, Distant Idea| H[Bucket 2]
    F -->|Random Top| I[Bucket 3]
    G --> J[Select Mode]
    H --> J
    I --> J
    J -->|matches < 2 or Elo gap < 50| K[Debate Mode]
    J -->|otherwise| L[Pairwise Mode]
    K --> M[LLM Comparison]
    L --> M
    M --> N{Parse Verdict}
    N -->|Valid a or b| O[Compute Elo Update]
    N -->|Invalid| P[Record Invalid Match]
    O --> Q[Insert Match + Journal]
    Q --> R[Emit tournament_match_complete]
    R --> S[Supervisor checks termination]
    E --> T[Idle Loop: more batches]
    T --> E

---

Convergence and Termination

Elo Stability Detection

The StabilityTracker in co_scientist/orchestrator/termination.py monitors the top-K leaderboard over time:

1. After every match_snapshot_every matches (default 10), an EloSnapshot is captured
2. The tracker maintains a sliding window of n snapshots (default 3)
3. Stability requires all of:
   • The same set of top-K hypothesis IDs across all N snapshots
   • Per-hypothesis Elo delta across the window < eps (default 25.0)
   • Minimum pool size (min_ideas_before_stable) and match count (min_matches_before_stable) guards

Stop Reasons

Reason	Condition	Action
BUDGET	Token/USD budget exhausted	Stop immediately
WALL_CLOCK	Session deadline crossed	Stop immediately
ELO_STABLE	Top-K unchanged for N snapshots	Graceful stop + final overview
EXTERNAL	User pressed pause/abort	Stop + preserve state
IDLE	Queue drained, no progress possible	Graceful stop + final overview

Leaderboard Maturity

The Supervisor's _decide_next_steps() method triggers evolution only when enough hypotheses are "mature" (at least 3 matches played). The min_mature config (default 20) gates when evolution begins. This prevents premature evolution from under-evaluated hypotheses.

---

Comparison with Unit-Distance Elo

The unit-distance-elo-ranking system and the Co-Scientist Elo system serve fundamentally different purposes:

Aspect	Unit-Distance Elo	Co-Scientist Elo
What is ranked	Mathematical hypotheses by novelty	Scientific hypotheses by quality
How judgments are made	By a human researcher (Alefita)	By LLM experts in simulated debate
K-factor	Not applicable	Dynamic (32/16 based on experience)
Comparison method	Manual scoring against benchmarks	Head-to-head pairwise matches
Purpose	Guide research direction toward higher novelty	Identify the strongest hypothesis in a pool
Termination	Human decides when to stop	Stability tracker detects convergence
Pair selection	Manual curation	Automated probabilistic selection with FAISS

The key philosophical difference: unit-distance Elo is a human calibration tool (the researcher adjusts scores based on deep mathematical understanding), while Co-Scientist Elo is a fully automated ranking mechanism where the LLM plays both the role of scientist and peer reviewer.

---

Evaluation Rubrics

The co_scientist/evals/rubrics.py module provides LLM-as-judge rubric scoring for offline evaluation (not part of the live tournament):

RANKING_RUBRIC:

• verdict_clarity (weight 1.0): Ends with "better idea: 1" or "better idea: 2"
• reasoning_quality (weight 1.0): Rationale references concrete differences, not vibes
• order_independence (weight 0.5): Verdict would not depend on listing order

The judge uses a different model than the agent under test (Sonnet judges Opus-generated content) to reduce echo-judge bias.

---

Cross-references

• co-scientist -- Project overview
• co-scientist-prompts -- The ranking prompts in detail
• co-scientist-pipeline -- How the tournament fits in the full pipeline
• unit-distance-elo-ranking -- The human-calibrated Elo system for mathematical hypotheses