---
name: co-scientist-elo-tournament
type: reference
title: "Co-Scientist Elo Tournament System"
description: "The Elo-based hypothesis ranking system: pairwise comparison, debate-driven judgment, pair selection heuristics, and leaderboard convergence"
tags: [co-scientist, elo, tournament, ranking, hypothesis, reference]
timestamp: 2026-07-21
---

# Co-Scientist Elo Tournament System

The Co-Scientist uses a standard **Elo rating system** to rank competing scientific hypotheses through LLM-mediated head-to-head matches. This is fundamentally different from the [[unit-distance-elo-ranking]] system (which rates hypotheses by mathematical novelty against known bounds) -- here, Elo measures *relative scientific quality* as judged by LLM experts in simulated peer review.

## Core Elo Mathematics

Implemented in `co_scientist/orchestrator/elo.py` -- pure math with no I/O.

### Expected Score

Standard logistic function:

```
E(A) = 1 / (1 + 10^((rating_B - rating_A) / 400))
```

### K-Factor

Dynamic K-factor based on match experience:

```python
def k_factor(matches_played, *, new_threshold=5, k_new=32, k_warm=16):
    return k_new if matches_played < new_threshold else k_warm
```

- **K=32** for hypotheses with fewer than 5 matches (fast convergence for new entrants)
- **K=16** for seasoned hypotheses (slower movement, more stability)

The K-factor is decided by the **less experienced** player in each match, ensuring new hypotheses can climb or fall quickly.

### Elo Update

```
delta = K * (S_actual - S_expected)
rating_A_after = rating_A + delta
rating_B_after = rating_B - delta
```

Updates are zero-sum: one hypothesis gains, the other loses by the same amount. The system uses `EloUpdate` dataclass with fields: `elo_a_after`, `elo_b_after`, `expected_a`, `k`.

### Match ID Determinism

Match IDs are computed deterministically from `sha256(min(a,b) || max(a,b) || round_id)` where `round_id` equals the task ID. This makes Elo updates idempotent: if a crash occurs mid-update, retrying with the same task produces the same match_id and the `elo_journal` table's UNIQUE constraint prevents double-application.

---

## Tournament Data Model

Defined in `co_scientist/storage/schema.sql` and `co_scientist/models/tournament.py`:

### Tables

**`tournament_matches`** -- Each head-to-head comparison:
| Column | Type | Purpose |
|---|---|---|
| `id` | TEXT PK | Deterministic sha256 |
| `session_id` | TEXT FK | Session scope |
| `hyp_a` | TEXT FK | First hypothesis |
| `hyp_b` | TEXT FK | Second hypothesis |
| `mode` | TEXT | `pairwise` or `debate` |
| `winner` | TEXT | `a`, `b`, or NULL (invalid) |
| `elo_a_before/after` | REAL | Elo snapshots |
| `elo_b_before/after` | REAL | Elo snapshots |
| `rationale` | TEXT | Full debate/pairwise text |
| `similarity` | REAL | Cosine similarity of hypothesis embeddings |

**`elo_journal`** -- Append-only ledger. UNIQUE on `match_id` makes updates idempotent.

### Hypothesis States

Hypotheses progress through states in the `hypotheses` table:

```
draft -> reviewed -> in_tournament -> pinned/rejected/quarantined/retired
```

Only `in_tournament` and `pinned` hypotheses participate in matches. Initial Elo is configurable (default 1200).

---

## Pair Selection Strategy

The RankingAgent's `_select_pair()` method implements a **three-bucket probabilistic selection** system using FAISS embeddings for idea-space distance:

### Bucket 1: New Arrival Pairing (probability `p_new = 0.4`)
Pair a **new hypothesis** (`matches_played < 3`) with the nearest-Elo **warm hypothesis** (`matches_played >= 3`). This ensures new hypotheses get compared against established ones quickly.

### Bucket 2: Close-Elo Distant-Idea (probability `p_close = 0.4`)
Among warm hypotheses with `|delta_Elo| < 200`, sample pairs weighted by:

```
weight = exp(-delta_Elo / 200) * (1 - cosine_similarity)
```

This favors pairs that are **close in rating but distant in idea-space** -- maximizing information gain from the debate. The FAISS store provides cosine similarity via single-row reconstruction (O(2*dim) per pair, not full index scan).

### Bucket 3: Random Top-Heavy (probability `p_random = 0.2`)
Sample randomly from the top half of hypotheses sorted by Elo. This prevents the system from only matching close competitors and allows occasional cross-tier comparisons.

### Focus Mode

When a task carries a `focus` ID (typically a newly added hypothesis), pair selection overrides the bucket system: it pairs the focused hypothesis with its nearest-Elo opponent. This ensures every new hypothesis immediately enters the tournament.

### FAISS Store Optimization

The FAISS index is loaded **once per `_select_pair()` call** (not per candidate). Previous implementations re-instantiated the embedder and reconstructed the full index for every pair evaluation, resulting in ~20 full-index reloads per match. The current implementation reconstructs only the two rows needed for cosine similarity.

---

## Match Mode Selection

The `_select_mode()` method chooses between `pairwise` and `debate`:

```python
def _select_mode(self, a, b):
    if min(a.matches_played, b.matches_played) < debate_when_matches_lt:  # default 2
        return "debate"
    if abs(a.elo - b.elo) < debate_when_elo_delta_lt:  # default 50
        return "debate"
    return "pairwise"
```

| Condition | Mode | Reasoning |
|---|---|---|
| Either hypothesis has < 2 matches | debate | New hypotheses need thorough discussion |
| Elo gap < 50 | debate | Close races need nuanced comparison |
| Otherwise | pairwise | Fast, decisive comparison |

### Pairwise vs. Debate

**Pairwise** (`ranking_pairwise.md`): Single LLM call, direct comparison. The model receives both hypotheses with reviews and must end with "better idea: 1" or "better idea: 2". Fast (one API call) but less nuanced.

**Debate** (`ranking_debate.md`): Multi-turn simulated expert panel (3-5 turns, max 10). The model summarizes both hypotheses, asks clarifying questions, identifies weaknesses, and delivers a final judgment. More expensive but produces better reasoning for close calls.

Both modes use the same termination format: the regex `_VERDICT_DIGIT_RE = r"^[\W_]*\**\s*([12])\b"` parses the trailing "better idea: X" from the response, with fallback matching for "option 1", "hypothesis 1", "hyp 1" variants.

---

## Tournament Flow

```mermaid
flowchart TD
    A[New Hypothesis Created] --> B[Review Completed]
    B --> C[AddToTournament]
    C --> D[Initialize Elo = 1200]
    D --> E[RunTournamentBatch with focus]
    E --> F{Select Pair}
    F -->|New + Warm| G[Bucket 1]
    F -->|Close Elo, Distant Idea| H[Bucket 2]
    F -->|Random Top| I[Bucket 3]
    G --> J[Select Mode]
    H --> J
    I --> J
    J -->|matches < 2 or Elo gap < 50| K[Debate Mode]
    J -->|otherwise| L[Pairwise Mode]
    K --> M[LLM Comparison]
    L --> M
    M --> N{Parse Verdict}
    N -->|Valid a or b| O[Compute Elo Update]
    N -->|Invalid| P[Record Invalid Match]
    O --> Q[Insert Match + Journal]
    Q --> R[Emit tournament_match_complete]
    R --> S[Supervisor checks termination]
    E --> T[Idle Loop: more batches]
    T --> E
```

---

## Convergence and Termination

### Elo Stability Detection

The `StabilityTracker` in `co_scientist/orchestrator/termination.py` monitors the top-K leaderboard over time:

1. After every `match_snapshot_every` matches (default 10), an `EloSnapshot` is captured
2. The tracker maintains a sliding window of `n` snapshots (default 3)
3. Stability requires **all** of:
   - The same set of top-K hypothesis IDs across all N snapshots
   - Per-hypothesis Elo delta across the window < `eps` (default 25.0)
   - Minimum pool size (`min_ideas_before_stable`) and match count (`min_matches_before_stable`) guards

### Stop Reasons

| Reason | Condition | Action |
|---|---|---|
| `BUDGET` | Token/USD budget exhausted | Stop immediately |
| `WALL_CLOCK` | Session deadline crossed | Stop immediately |
| `ELO_STABLE` | Top-K unchanged for N snapshots | Graceful stop + final overview |
| `EXTERNAL` | User pressed pause/abort | Stop + preserve state |
| `IDLE` | Queue drained, no progress possible | Graceful stop + final overview |

### Leaderboard Maturity

The Supervisor's `_decide_next_steps()` method triggers evolution only when enough hypotheses are "mature" (at least 3 matches played). The `min_mature` config (default 20) gates when evolution begins. This prevents premature evolution from under-evaluated hypotheses.

---

## Comparison with Unit-Distance Elo

The [[unit-distance-elo-ranking]] system and the Co-Scientist Elo system serve fundamentally different purposes:

| Aspect | Unit-Distance Elo | Co-Scientist Elo |
|---|---|---|
| What is ranked | Mathematical hypotheses by novelty | Scientific hypotheses by quality |
| How judgments are made | By a human researcher (Alefita) | By LLM experts in simulated debate |
| K-factor | Not applicable | Dynamic (32/16 based on experience) |
| Comparison method | Manual scoring against benchmarks | Head-to-head pairwise matches |
| Purpose | Guide research direction toward higher novelty | Identify the strongest hypothesis in a pool |
| Termination | Human decides when to stop | Stability tracker detects convergence |
| Pair selection | Manual curation | Automated probabilistic selection with FAISS |

The key philosophical difference: unit-distance Elo is a **human calibration tool** (the researcher adjusts scores based on deep mathematical understanding), while Co-Scientist Elo is a **fully automated ranking mechanism** where the LLM plays both the role of scientist and peer reviewer.

---

## Evaluation Rubrics

The `co_scientist/evals/rubrics.py` module provides LLM-as-judge rubric scoring for offline evaluation (not part of the live tournament):

**RANKING_RUBRIC:**
- `verdict_clarity` (weight 1.0): Ends with "better idea: 1" or "better idea: 2"
- `reasoning_quality` (weight 1.0): Rationale references concrete differences, not vibes
- `order_independence` (weight 0.5): Verdict would not depend on listing order

The judge uses a **different model** than the agent under test (Sonnet judges Opus-generated content) to reduce echo-judge bias.

---

## Cross-references

- [[co-scientist]] -- Project overview
- [[co-scientist-prompts]] -- The ranking prompts in detail
- [[co-scientist-pipeline]] -- How the tournament fits in the full pipeline
- [[unit-distance-elo-ranking]] -- The human-calibrated Elo system for mathematical hypotheses