---
name: co-scientist-pipeline
type: reference
title: "Co-Scientist Full Pipeline"
description: "Complete data flow from goal input to final overview: parsing, generation, reflection, ranking, evolution, meta-review, and termination"
tags: [co-scientist, pipeline, orchestrator, supervisor, task-queue, reference]
timestamp: 2026-07-21
---

# Co-Scientist Full Pipeline

The Co-Scientist pipeline is a **durable, bounded-concurrency multi-agent system** orchestrated by a Supervisor that manages a SQLite-backed task queue. A researcher provides a natural-language goal; the system autonomously generates hypotheses, reviews them, runs tournaments, evolves top performers, and produces a final research overview.

## High-Level Architecture

```mermaid
flowchart TB
    subgraph Entry
        A[Scientist's Goal] --> B[parse_goal]
        B --> C[ResearchPlan]
    end

    subgraph Core Loop
        C --> D[Generation Agent]
        D -->|record_hypothesis| E[hypothesis row]
        E --> F[Reflection Agent]
        F -->|record_review| G[review row]
        G --> H[AddToTournament]
        H --> I[Ranking Agent]
        I -->|tournament_match_complete| J[Elo Update]
    end

    subgraph Idle Refinement
        J --> K{Queue Empty?}
        K -->|No| I
        K -->|Yes| L[decide_next_steps]
        L -->|tournament batch| I
        L -->|evolution needed| M[Evolution Agent]
        L -->|periodic feedback| N[MetaReview Agent]
        M -->|record_hypothesis| E
        N -->|record_system_feedback| O[SystemFeedback]
        O -->|injected into next generation| D
    end

    subgraph Termination
        K -->|StopReason| P[Finalize]
        P --> Q[MetaReviewAgent Final Overview]
        Q --> R[Markdown Research Report]
    end
```

## Phase 1: Goal Parsing

**Entry point:** `Supervisor.run_session()` in `co_scientist/agents/supervisor.py`

1. A session row is inserted into SQLite with the goal, budget limits, wall-clock deadline, and a frozen config snapshot
2. The `parse_goal` prompt is rendered and sent to Sonnet (single call, forced `record_research_plan`)
3. The `ResearchPlan` is extracted and stored:
   - `objective` -- atomic research question
   - `preferences` -- what makes a good hypothesis for this goal
   - `constraints` -- scope/methodology limits
   - `idea_attributes` -- adjectives for strong candidates
   - `domain_hint` -- discipline classification
4. Initial generation tasks are enqueued: `n_initial` (default 3) tasks with `agent="generation"`, `action="CreateInitialHypotheses"`, each with a unique idempotency key

## Phase 2: Generation

**Agent:** `GenerationAgent` in `co_scientist/agents/generation.py`
**Prompt:** `generation.literature` or `generation.debate`
**Model:** Opus with 4000 thinking tokens

### Tool Loop Execution

Generation uses the tool loop (`co_scientist/llm/tool_loop.py`) with up to 8 iterations:

1. The model receives the prompt with access to search tools: `web_search`, `pubmed_search`, `arxiv_search`, `europe_pmc_search`, `web_fetch`
2. It performs literature searches, reads abstracts, and synthesizes understanding
3. All URLs from tool results are tracked in `seen_urls`
4. The model calls `record_hypothesis` with a structured hypothesis record
5. The tool loop detects the terminal tool call and returns immediately
6. Citation URLs are filtered against `seen_urls` (anti-hallucination)
7. The hypothesis is persisted with a deterministic ID: `sha256(session_id || origin || statement)`

### Deduplication

Before inserting, the hypothesis is embedded (Voyage-3-large, 1024-dim) and checked against the session's FAISS index. If cosine similarity exceeds `dedup_cosine_threshold` (default 0.92), the duplicate is rejected and the existing hypothesis ID is returned.

### Follow-Up Scheduling

After `hypothesis_created`, the Supervisor enqueues:
```
Task(agent="reflection", action="ReviewHypothesis", target_id=hid, kind="full")
```

## Phase 3: Reflection

**Agent:** `ReflectionAgent` in `co_scientist/agents/reflection.py`
**Prompt:** `reflection.full` (M3); `reflection.verification` and `reflection.observation` in later milestones
**Model:** Opus

### Tool Loop Execution

Reflection uses the tool loop with up to 8 iterations:

1. The model receives the hypothesis wrapped in `<HYPOTHESIS_TEXT>` tags
2. It searches for supporting/contradicting evidence using available tools
3. It calls `record_review` with a structured review including verdict, scores, evidence, and assumptions
4. Evidence entries are filtered against `seen_urls` (same anti-hallucination guard)

### State Transition

After review, the hypothesis state transitions from `draft` to `reviewed` (only if currently in `draft` state -- never drags back evolved/pinned hypotheses).

### Follow-Up Scheduling

After `review_completed`, the Supervisor enqueues:
```
Task(agent="ranking", action="AddToTournament", target_id=hid)
```

## Phase 4: Tournament Ranking

**Agent:** `RankingAgent` in `co_scientist/agents/ranking.py`
**Model:** Sonnet (intentionally weaker to avoid echo-judge bias)

### AddToTournament

No LLM call. Initializes the hypothesis's Elo to 1200 and sets state to `in_tournament`.

### RunTournamentBatch

The core ranking loop:

1. **Pair selection** via three-bucket probabilistic strategy (see [[co-scientist-elo-tournament]])
2. **Mode selection** (pairwise vs. debate based on match count and Elo gap)
3. **LLM comparison** -- single call for pairwise, multi-turn for debate
4. **Verdict parsing** via regex extraction of "better idea: 1|2"
5. **Elo update** via `update_elo()` with deterministic match IDs
6. **Match persistence** to `tournament_matches` and `elo_journal` tables

### Follow-Up Scheduling

After `tournament_match_complete`, the Supervisor periodically re-clusters the proximity graph (every `full_recluster_every_matches` matches, default 20).

## Phase 5: Evolution

**Agent:** `EvolutionAgent` in `co_scientist/agents/evolution.py`
**Model:** Opus with 6000 thinking tokens

### When Evolution Triggers

The Supervisor's `_decide_next_steps()` method triggers evolution when:
- The queue is idle (no pending or in-progress tasks)
- At least `min_mature` hypotheses (default 20) have `matches_played >= 3`
- This is the idle-refinement path, not a reactive trigger

### Strategies

| Strategy | Selection | Input | Output |
|---|---|---|---|
| `combine` | Most idea-distant pair in top-K | Two hypotheses + reviews | Synthesized hypothesis |
| `simplify` | Top-1 hypothesis | One hypothesis + review | Simplified hypothesis |
| `out_of_box` | Top-5 hypotheses | Five hypotheses | Novel hypothesis inspired by analogies |
| `feasibility` | Top-1 hypothesis | One hypothesis | Technology-refined hypothesis |

### Dedup and Persistence

Same dedup + FAISS pattern as Generation. Evolution hypotheses carry `parent_ids` for lineage tracking. They enter the same `draft -> reviewed -> in_tournament` pipeline.

## Phase 6: Meta-Review (Periodic)

**Agent:** `MetaReviewAgent` in `co_scientist/agents/metareview.py`
**Prompt:** `metareview.system`
**Model:** Sonnet with 8000 thinking tokens

### Trigger

Scheduled every ~50 matches (approximated by: `match_count >= (feedback_count + 1) * 50`).

### What It Synthesizes

- All session reviews (up to 50, truncated to 3000 chars each)
- Recent tournament debate rationales (up to 50, truncated to 1500 chars each)

### Output

`record_system_feedback` with:
- `common_weaknesses[]` -- recurring critique patterns
- `common_strengths[]` -- what reviewers consistently praise
- `suggested_focus_areas[]` -- directions the system should explore
- `narrative` -- 1-2 paragraph synthesis

### Feedback Loop

The narrative is stored as a `SystemFeedback` row and **auto-injected** into the next Generation and Evolution prompts via `_latest_system_feedback()`. This creates a closed learning signal:

```
reviews -> meta-review -> system_feedback -> generation/evolution -> new hypotheses -> reviews -> ...
```

## Phase 7: Final Overview

**Agent:** `MetaReviewAgent` (final action)
**Prompt:** `metareview.final`
**Model:** Opus with 16000 thinking tokens (maximum)
**No tools:** Produces plain markdown

### Trigger

Runs once in `Supervisor._finalize()` after the main loop exits on any `StopReason`.

### Content

The final overview is structured as:
1. **Executive summary** -- what the tournament converged on
2. **Main research directions** -- per-direction: claim, supporting hypotheses, open questions, first experiment
3. **Convergence and divergence** -- which hypotheses overlap, which are orthogonal
4. **Caveats and limitations** -- what was not explored, where domain experts might disagree

If MetaReviewAgent fails, a fallback overview is generated from raw hypothesis metadata.

---

## The Supervisor Main Loop

The Supervisor (`co_scientist/agents/supervisor.py`) is the heart of the system:

```mermaid
flowchart TD
    A[Main Loop Start] --> B{Check External Stop}
    B -->|aborted| C[Wait for Inflight to Drain]
    B -->|paused| D[Sleep 1s, Retry]
    B -->|running| E{should_stop?}
    E -->|BUDGET/WALL_CLOCK/ELO_STABLE| C
    E -->|None| F{Claim Tasks}
    F --> G[Semaphore-Limited Workers]
    G --> H[Run Task via Agent]
    H --> I[Apply Follow-Up Rules]
    I --> J[Complete Task]
    J --> K[Emit Event to Bus]
    K --> L{Update Elo Snapshot?}
    L -->|match threshold crossed| M[Push Snapshot to Tracker]
    L --> N{Queue Empty?}
    M --> N
    N -->|tasks pending| A
    N -->|empty + inflight| O[Wait for First Complete]
    O --> A
    N -->|empty + idle| P{decide_next_steps}
    P -->|enqueued > 0| A
    P -->|enqueued = 0| Q[Return IDLE]
    C --> R[Finalize]
```

### Bounded Concurrency

Concurrency is controlled by `asyncio.Semaphore(cfg.run.concurrency)` (default 4). Workers claim tasks from the SQLite queue with optimistic locking (lease-based with configurable timeout and max attempts).

### Task Queue Schema

The `tasks` table in `co_scientist/storage/schema.sql`:

| Column | Purpose |
|---|---|
| `agent` | Which agent handles this task |
| `action` | Specific action (e.g., `CreateInitialHypotheses`, `ReviewHypothesis`) |
| `target_id` | Related entity (hypothesis_id, etc.) |
| `priority` | Lower = higher priority (100 default, 80 for ranking, 120 for tournament batch) |
| `status` | `pending -> leased -> in_progress -> done/failed/dead/cancelled` |
| `lease_owner` | Worker ID holding the lease |
| `lease_expires_at` | Lease TTL (default 300s, 600s for reflection, 1800s for final overview) |
| `idempotency_key` | UNIQUE constraint prevents duplicate task enqueueing |
| `attempts` | Retry counter, max 3 by default |

### Lease Management

Tasks are claimed by setting `lease_owner` and `lease_expires_at`. If a worker crashes, the Supervisor reclaims expired leases on resume via `task_repo.reclaim_expired_leases()`. The `max_attempts` config (default 3) prevents infinite retries.

---

## Token Budget System

Implemented in `co_scientist/llm/budgets.py`.

### Architecture

A single `TokenBudget` per session, shared across all concurrent agents. Admission is serialized by `asyncio.Lock` to prevent race conditions.

### Per-Agent Shares

| Agent | Share | Reasoning |
|---|---|---|
| generation | 20% | Multiple hypotheses, each needing search + synthesis |
| reflection | 20% | Each hypothesis needs thorough review |
| ranking | 25% | Multiple tournament matches per session |
| evolution | 15% | Runs less frequently, on mature hypotheses |
| metareview | 10% | Periodic + final, relatively rare |
| proximity | 2% | Embedding updates, lightweight |
| reserve | 8% | Buffer for overages |

### Admission Flow

1. `admit(agent, est_tokens, est_usd)` -- blocks if session budget or agent share would be exceeded
2. Agent executes the LLM call
3. `settle(agent, est_tokens, est_usd, actual_usd, ...)` -- releases reservation and credits actual usage

### Cost Estimation

The routing module (`co_scientist/llm/routing.py`) maintains a comprehensive `PRICE_TABLE` covering Anthropic, OpenAI, Google, Meta, and Mistral models. Unknown models fall back to family-hint matching (e.g., any model containing "flash" gets flash-tier pricing) or a conservative Sonnet-class default.

---

## Model Routing

The `route()` function in `co_scientist/llm/routing.py` maps `(agent, mode)` pairs to specific model configurations:

| Agent | Mode | Default Model | Thinking Tokens |
|---|---|---|---|
| parse_goal | -- | claude-sonnet-4-6 | 0 |
| generation | literature | claude-opus-4-7 | 4000 |
| generation | debate | claude-opus-4-7 | 8000 |
| reflection | full | claude-opus-4-7 | 0 |
| reflection | verification | claude-opus-4-7 | 12000 |
| reflection | observation | claude-opus-4-7 | 6000 |
| ranking | pairwise | claude-sonnet-4-6 | 4000 |
| ranking | debate | claude-sonnet-4-6 | 8000 |
| evolution | combine | claude-opus-4-7 | 6000 |
| evolution | out_of_box | claude-opus-4-7 | 6000 |
| evolution | simplify | claude-opus-4-7 | 0 |
| evolution | feasibility | claude-opus-4-7 | 0 |
| metareview | system | claude-sonnet-4-6 | 8000 |
| metareview | final | claude-opus-4-7 | 16000 |
| classifier | -- | claude-haiku-4-5-20251001 | 0 |
| judge | -- | claude-sonnet-4-6 | 0 |

### Degradation Chain

When a model is unavailable, the system walks a soft fallback chain:
```
claude-opus-4-7 -> claude-sonnet-4-6 -> claude-haiku-4-5-20251001
```

Certain modes are **never-degraded**: `reflection.verification` and `metareview.final` -- these require maximum capability.

### Thinking Budgets

Thinking tokens are only allocated for Opus-class models. Non-Opus models always get `thinking_tokens=0`. The thinking budget for each mode is configurable in `ThinkingCfg`.

---

## Event Bus

The `EventBus` in `co_scientist/orchestrator/events.py` provides in-process pub/sub for the web UI:

- Agents publish events via `bus.publish(session_id, event_name, payload)`
- SSE handlers subscribe via `bus.subscribe(session_id)` for async iteration
- Buffer limit: 256 events per subscriber; oldest events are dropped for slow subscribers
- The bus is in-process only; on restart, the UI snapshots from the `events` table

### Event Types

| Event | Source | Payload |
|---|---|---|
| `session_started` | Supervisor | goal, n_initial, budget_usd |
| `task_started` | Supervisor | task_id, agent, action, target |
| `task_completed` | Supervisor | task_id, kind, follow_hypothesis_ids |
| `task_failed` | Supervisor | task_id, err |
| `match_complete` | RankingAgent | mode, hyp_a, hyp_b, winner, elo_applied |
| `session_done` | Supervisor | stop_reason |

---

## Database Schema

The full SQLite schema in `co_scientist/storage/schema.sql` (WAL mode, foreign keys enabled):

| Table | Purpose | Key Relationships |
|---|---|---|
| `sessions` | Session config, budget, status, final_overview | Parent of all other tables |
| `hypotheses` | Hypothesis records with state machine | FK to sessions |
| `reviews` | Review records with verdicts and scores | FK to hypotheses + sessions |
| `tournament_matches` | Head-to-head comparison results | FK to hypotheses + sessions |
| `elo_journal` | Append-only Elo update ledger | UNIQUE on match_id |
| `tasks` | Durable task queue with lease management | FK to sessions |
| `transcripts` | LLM call logs with token counts and costs | FK to sessions + tasks |
| `system_feedback` | Meta-review narratives + human preferences | FK to sessions |
| `embeddings_meta` | FAISS vector index metadata | FK to hypotheses |
| `spans` | OpenTelemetry-style observability spans | FK to sessions + tasks |
| `events` | Event log for UI replay | FK to sessions |
| `schema_migrations` | Linear migration tracking | Standalone |

### Migrations

Migrations are in `co_scientist/storage/migrations/`. The system uses a `schema_migrations` table to track applied versions. Migration v1 uses the canonical `schema.sql`; subsequent migrations use individual files. Duplicate-column errors during fresh init are gracefully handled (ALTER TABLE becomes a no-op).

---

## Complete Data Flow Summary

```mermaid
sequenceDiagram
    participant S as Scientist
    participant Sup as Supervisor
    participant Gen as Generation
    participant Ref as Reflection
    participant Rank as Ranking
    participant Evo as Evolution
    participant MR as MetaReview
    participant DB as SQLite

    S->>Sup: goal + preferences
    Sup->>DB: INSERT session
    Sup->>DB: parse_goal -> ResearchPlan
    loop n_initial times
        Sup->>DB: enqueue CreateInitialHypotheses
    end

    loop Main Loop
        Sup->>DB: claim task
        alt Generation task
            Sup->>Gen: execute(task)
            Gen->>Gen: tool loop (search -> record_hypothesis)
            Gen->>DB: INSERT hypothesis (state=draft)
            Gen->>Sup: hypothesis_created
            Sup->>DB: enqueue ReviewHypothesis
        else Reflection task
            Sup->>Ref: execute(task)
            Ref->>Ref: tool loop (search -> record_review)
            Ref->>DB: INSERT review
            Ref->>DB: UPDATE hypothesis state -> reviewed
            Ref->>Sup: review_completed
            Sup->>DB: enqueue AddToTournament
        else Ranking task
            Sup->>Rank: execute(task)
            alt AddToTournament
                Rank->>DB: UPDATE hypothesis state -> in_tournament
            else RunTournamentBatch
                Rank->>Rank: select pair, run debate/pairwise
                Rank->>DB: INSERT match + elo_journal
                Rank->>DB: UPDATE hypothesis elos
            end
            Rank->>Sup: tournament_match_complete
        else Evolution task
            Sup->>Evo: execute(task)
            Evo->>Evo: tool loop (combine/simplify/out_of_box)
            Evo->>DB: INSERT hypothesis (state=draft)
            Evo->>Sup: hypothesis_created
            Sup->>DB: enqueue ReviewHypothesis
        else MetaReview task
            Sup->>MR: execute(task)
            alt System Feedback
                MR->>DB: INSERT system_feedback
                Note right of DB: Injected into next Gen/Evo prompts
            else Final Overview
                MR->>MR: synthesize top hypotheses
                MR->>DB: UPDATE session.final_overview
            end
        end
        Sup->>DB: check termination
    end

    Sup->>MR: GenerateFinalResearchOverview
    MR->>DB: UPDATE session.status -> done
    Sup->>S: session_id
```

---

## Default Configuration

Key defaults from `co_scientist/config.py`:

| Parameter | Value | Purpose |
|---|---|---|
| `run.concurrency` | 4 | Max parallel agent workers |
| `run.max_ideas` | 60 | Cap on total hypotheses |
| `run.wall_clock_seconds` | 7200 (2h) | Session time limit |
| `run.budget_usd` | $25.00 | Session USD cap |
| `ranking.elo_initial` | 1200 | Starting Elo for new hypotheses |
| `ranking.k_factor_new` | 32 | Fast convergence for new hypotheses |
| `ranking.k_factor_warm` | 16 | Stable Elo for experienced hypotheses |
| `termination.elo_stability_k` | 5 | Top-K to monitor for convergence |
| `termination.elo_stability_n` | 3 | Snapshots needed for stability |
| `termination.elo_stability_eps` | 25.0 | Max Elo drift within stable window |
| `evolution.min_mature` | 20 | Min hypotheses with >= 3 matches before evolution |
| `evolution.top_k` | 5 | Top hypotheses to evolve |
| `vectors.dedup_cosine_threshold` | 0.92 | Cosine similarity threshold for dedup |
| `lease.default_seconds` | 300 | Task lease TTL |

---

## Cross-references

- [[co-scientist]] -- Project overview
- [[co-scientist-prompts]] -- All 14 prompt templates in detail
- [[co-scientist-elo-tournament]] -- The Elo ranking subsystem