Co-Scientist Full Pipeline

The Co-Scientist pipeline is a durable, bounded-concurrency multi-agent system orchestrated by a Supervisor that manages a SQLite-backed task queue. A researcher provides a natural-language goal; the system autonomously generates hypotheses, reviews them, runs tournaments, evolves top performers, and produces a final research overview.

High-Level Architecture

flowchart TB
    subgraph Entry
        A[Scientist's Goal] --> B[parse_goal]
        B --> C[ResearchPlan]
    end

    subgraph Core Loop
        C --> D[Generation Agent]
        D -->|record_hypothesis| E[hypothesis row]
        E --> F[Reflection Agent]
        F -->|record_review| G[review row]
        G --> H[AddToTournament]
        H --> I[Ranking Agent]
        I -->|tournament_match_complete| J[Elo Update]
    end

    subgraph Idle Refinement
        J --> K{Queue Empty?}
        K -->|No| I
        K -->|Yes| L[decide_next_steps]
        L -->|tournament batch| I
        L -->|evolution needed| M[Evolution Agent]
        L -->|periodic feedback| N[MetaReview Agent]
        M -->|record_hypothesis| E
        N -->|record_system_feedback| O[SystemFeedback]
        O -->|injected into next generation| D
    end

    subgraph Termination
        K -->|StopReason| P[Finalize]
        P --> Q[MetaReviewAgent Final Overview]
        Q --> R[Markdown Research Report]
    end

Phase 1: Goal Parsing

Entry point: Supervisor.run_session() in co_scientist/agents/supervisor.py

A session row is inserted into SQLite with the goal, budget limits, wall-clock deadline, and a frozen config snapshot
The parse_goal prompt is rendered and sent to Sonnet (single call, forced record_research_plan)
The ResearchPlan is extracted and stored:
- objective -- atomic research question
- preferences -- what makes a good hypothesis for this goal
- constraints -- scope/methodology limits
- idea_attributes -- adjectives for strong candidates
- domain_hint -- discipline classification
Initial generation tasks are enqueued: n_initial (default 3) tasks with agent="generation", action="CreateInitialHypotheses", each with a unique idempotency key

Phase 2: Generation

Agent: GenerationAgent in co_scientist/agents/generation.py Prompt: generation.literature or generation.debate Model: Opus with 4000 thinking tokens

Tool Loop Execution

Generation uses the tool loop (co_scientist/llm/tool_loop.py) with up to 8 iterations:

The model receives the prompt with access to search tools: web_search, pubmed_search, arxiv_search, europe_pmc_search, web_fetch
It performs literature searches, reads abstracts, and synthesizes understanding
All URLs from tool results are tracked in seen_urls
The model calls record_hypothesis with a structured hypothesis record
The tool loop detects the terminal tool call and returns immediately
Citation URLs are filtered against seen_urls (anti-hallucination)
The hypothesis is persisted with a deterministic ID: sha256(session_id || origin || statement)

Deduplication

Before inserting, the hypothesis is embedded (Voyage-3-large, 1024-dim) and checked against the session's FAISS index. If cosine similarity exceeds dedup_cosine_threshold (default 0.92), the duplicate is rejected and the existing hypothesis ID is returned.

Follow-Up Scheduling

After hypothesis_created, the Supervisor enqueues:

Task(agent="reflection", action="ReviewHypothesis", target_id=hid, kind="full")

Phase 3: Reflection

Agent: ReflectionAgent in co_scientist/agents/reflection.py Prompt: reflection.full (M3); reflection.verification and reflection.observation in later milestones Model: Opus

Tool Loop Execution

Reflection uses the tool loop with up to 8 iterations:

The model receives the hypothesis wrapped in <HYPOTHESIS_TEXT> tags
It searches for supporting/contradicting evidence using available tools
It calls record_review with a structured review including verdict, scores, evidence, and assumptions
Evidence entries are filtered against seen_urls (same anti-hallucination guard)

State Transition

After review, the hypothesis state transitions from draft to reviewed (only if currently in draft state -- never drags back evolved/pinned hypotheses).

Follow-Up Scheduling

After review_completed, the Supervisor enqueues:

Task(agent="ranking", action="AddToTournament", target_id=hid)

Phase 4: Tournament Ranking

Agent: RankingAgent in co_scientist/agents/ranking.py Model: Sonnet (intentionally weaker to avoid echo-judge bias)

AddToTournament

No LLM call. Initializes the hypothesis's Elo to 1200 and sets state to in_tournament.

RunTournamentBatch

The core ranking loop:

Pair selection via three-bucket probabilistic strategy (see co-scientist-elo-tournament)
Mode selection (pairwise vs. debate based on match count and Elo gap)
LLM comparison -- single call for pairwise, multi-turn for debate
Verdict parsing via regex extraction of "better idea: 1|2"
Elo update via update_elo() with deterministic match IDs
Match persistence to tournament_matches and elo_journal tables

Follow-Up Scheduling

After tournament_match_complete, the Supervisor periodically re-clusters the proximity graph (every full_recluster_every_matches matches, default 20).

Phase 5: Evolution

Agent: EvolutionAgent in co_scientist/agents/evolution.py Model: Opus with 6000 thinking tokens

When Evolution Triggers

The Supervisor's _decide_next_steps() method triggers evolution when:

The queue is idle (no pending or in-progress tasks)
At least min_mature hypotheses (default 20) have matches_played >= 3
This is the idle-refinement path, not a reactive trigger

Strategies

Strategy	Selection	Input	Output
`combine`	Most idea-distant pair in top-K	Two hypotheses + reviews	Synthesized hypothesis
`simplify`	Top-1 hypothesis	One hypothesis + review	Simplified hypothesis
`out_of_box`	Top-5 hypotheses	Five hypotheses	Novel hypothesis inspired by analogies
`feasibility`	Top-1 hypothesis	One hypothesis	Technology-refined hypothesis

Dedup and Persistence

Same dedup + FAISS pattern as Generation. Evolution hypotheses carry parent_ids for lineage tracking. They enter the same draft -> reviewed -> in_tournament pipeline.

Phase 6: Meta-Review (Periodic)

Agent: MetaReviewAgent in co_scientist/agents/metareview.py Prompt: metareview.system Model: Sonnet with 8000 thinking tokens

Trigger

Scheduled every ~50 matches (approximated by: match_count >= (feedback_count + 1) * 50).

What It Synthesizes

All session reviews (up to 50, truncated to 3000 chars each)
Recent tournament debate rationales (up to 50, truncated to 1500 chars each)

Output

record_system_feedback with:

common_weaknesses[] -- recurring critique patterns
common_strengths[] -- what reviewers consistently praise
suggested_focus_areas[] -- directions the system should explore
narrative -- 1-2 paragraph synthesis

Feedback Loop

The narrative is stored as a SystemFeedback row and auto-injected into the next Generation and Evolution prompts via _latest_system_feedback(). This creates a closed learning signal:

reviews -> meta-review -> system_feedback -> generation/evolution -> new hypotheses -> reviews -> ...

Phase 7: Final Overview

Agent: MetaReviewAgent (final action) Prompt: metareview.final Model: Opus with 16000 thinking tokens (maximum) No tools: Produces plain markdown

Trigger

Runs once in Supervisor._finalize() after the main loop exits on any StopReason.

Content

The final overview is structured as:

Executive summary -- what the tournament converged on
Main research directions -- per-direction: claim, supporting hypotheses, open questions, first experiment
Convergence and divergence -- which hypotheses overlap, which are orthogonal
Caveats and limitations -- what was not explored, where domain experts might disagree

If MetaReviewAgent fails, a fallback overview is generated from raw hypothesis metadata.

The Supervisor Main Loop

The Supervisor (co_scientist/agents/supervisor.py) is the heart of the system:

flowchart TD
    A[Main Loop Start] --> B{Check External Stop}
    B -->|aborted| C[Wait for Inflight to Drain]
    B -->|paused| D[Sleep 1s, Retry]
    B -->|running| E{should_stop?}
    E -->|BUDGET/WALL_CLOCK/ELO_STABLE| C
    E -->|None| F{Claim Tasks}
    F --> G[Semaphore-Limited Workers]
    G --> H[Run Task via Agent]
    H --> I[Apply Follow-Up Rules]
    I --> J[Complete Task]
    J --> K[Emit Event to Bus]
    K --> L{Update Elo Snapshot?}
    L -->|match threshold crossed| M[Push Snapshot to Tracker]
    L --> N{Queue Empty?}
    M --> N
    N -->|tasks pending| A
    N -->|empty + inflight| O[Wait for First Complete]
    O --> A
    N -->|empty + idle| P{decide_next_steps}
    P -->|enqueued > 0| A
    P -->|enqueued = 0| Q[Return IDLE]
    C --> R[Finalize]

Bounded Concurrency

Concurrency is controlled by asyncio.Semaphore(cfg.run.concurrency) (default 4). Workers claim tasks from the SQLite queue with optimistic locking (lease-based with configurable timeout and max attempts).

Task Queue Schema

The tasks table in co_scientist/storage/schema.sql:

Column	Purpose
`agent`	Which agent handles this task
`action`	Specific action (e.g., `CreateInitialHypotheses`, `ReviewHypothesis`)
`target_id`	Related entity (hypothesis_id, etc.)
`priority`	Lower = higher priority (100 default, 80 for ranking, 120 for tournament batch)
`status`	`pending -> leased -> in_progress -> done/failed/dead/cancelled`
`lease_owner`	Worker ID holding the lease
`lease_expires_at`	Lease TTL (default 300s, 600s for reflection, 1800s for final overview)
`idempotency_key`	UNIQUE constraint prevents duplicate task enqueueing
`attempts`	Retry counter, max 3 by default

Lease Management

Tasks are claimed by setting lease_owner and lease_expires_at. If a worker crashes, the Supervisor reclaims expired leases on resume via task_repo.reclaim_expired_leases(). The max_attempts config (default 3) prevents infinite retries.

Token Budget System

Implemented in co_scientist/llm/budgets.py.

Architecture

A single TokenBudget per session, shared across all concurrent agents. Admission is serialized by asyncio.Lock to prevent race conditions.

Per-Agent Shares

Agent	Share	Reasoning
generation	20%	Multiple hypotheses, each needing search + synthesis
reflection	20%	Each hypothesis needs thorough review
ranking	25%	Multiple tournament matches per session
evolution	15%	Runs less frequently, on mature hypotheses
metareview	10%	Periodic + final, relatively rare
proximity	2%	Embedding updates, lightweight
reserve	8%	Buffer for overages

Admission Flow

admit(agent, est_tokens, est_usd) -- blocks if session budget or agent share would be exceeded
Agent executes the LLM call
settle(agent, est_tokens, est_usd, actual_usd, ...) -- releases reservation and credits actual usage

Cost Estimation

The routing module (co_scientist/llm/routing.py) maintains a comprehensive PRICE_TABLE covering Anthropic, OpenAI, Google, Meta, and Mistral models. Unknown models fall back to family-hint matching (e.g., any model containing "flash" gets flash-tier pricing) or a conservative Sonnet-class default.

Model Routing

The route() function in co_scientist/llm/routing.py maps (agent, mode) pairs to specific model configurations:

Agent	Mode	Default Model	Thinking Tokens
parse_goal	--	claude-sonnet-4-6	0
generation	literature	claude-opus-4-7	4000
generation	debate	claude-opus-4-7	8000
reflection	full	claude-opus-4-7	0
reflection	verification	claude-opus-4-7	12000
reflection	observation	claude-opus-4-7	6000
ranking	pairwise	claude-sonnet-4-6	4000
ranking	debate	claude-sonnet-4-6	8000
evolution	combine	claude-opus-4-7	6000
evolution	out_of_box	claude-opus-4-7	6000
evolution	simplify	claude-opus-4-7	0
evolution	feasibility	claude-opus-4-7	0
metareview	system	claude-sonnet-4-6	8000
metareview	final	claude-opus-4-7	16000
classifier	--	claude-haiku-4-5-20251001	0
judge	--	claude-sonnet-4-6	0

Degradation Chain

When a model is unavailable, the system walks a soft fallback chain:

claude-opus-4-7 -> claude-sonnet-4-6 -> claude-haiku-4-5-20251001

Certain modes are never-degraded: reflection.verification and metareview.final -- these require maximum capability.

Thinking Budgets

Thinking tokens are only allocated for Opus-class models. Non-Opus models always get thinking_tokens=0. The thinking budget for each mode is configurable in ThinkingCfg.

Event Bus

The EventBus in co_scientist/orchestrator/events.py provides in-process pub/sub for the web UI:

Agents publish events via bus.publish(session_id, event_name, payload)
SSE handlers subscribe via bus.subscribe(session_id) for async iteration
Buffer limit: 256 events per subscriber; oldest events are dropped for slow subscribers
The bus is in-process only; on restart, the UI snapshots from the events table

Event Types

Event	Source	Payload
`session_started`	Supervisor	goal, n_initial, budget_usd
`task_started`	Supervisor	task_id, agent, action, target
`task_completed`	Supervisor	task_id, kind, follow_hypothesis_ids
`task_failed`	Supervisor	task_id, err
`match_complete`	RankingAgent	mode, hyp_a, hyp_b, winner, elo_applied
`session_done`	Supervisor	stop_reason

Database Schema

The full SQLite schema in co_scientist/storage/schema.sql (WAL mode, foreign keys enabled):

Table	Purpose	Key Relationships
`sessions`	Session config, budget, status, final_overview	Parent of all other tables
`hypotheses`	Hypothesis records with state machine	FK to sessions
`reviews`	Review records with verdicts and scores	FK to hypotheses + sessions
`tournament_matches`	Head-to-head comparison results	FK to hypotheses + sessions
`elo_journal`	Append-only Elo update ledger	UNIQUE on match_id
`tasks`	Durable task queue with lease management	FK to sessions
`transcripts`	LLM call logs with token counts and costs	FK to sessions + tasks
`system_feedback`	Meta-review narratives + human preferences	FK to sessions
`embeddings_meta`	FAISS vector index metadata	FK to hypotheses
`spans`	OpenTelemetry-style observability spans	FK to sessions + tasks
`events`	Event log for UI replay	FK to sessions
`schema_migrations`	Linear migration tracking	Standalone

Migrations

Migrations are in co_scientist/storage/migrations/. The system uses a schema_migrations table to track applied versions. Migration v1 uses the canonical schema.sql; subsequent migrations use individual files. Duplicate-column errors during fresh init are gracefully handled (ALTER TABLE becomes a no-op).

Complete Data Flow Summary

sequenceDiagram
    participant S as Scientist
    participant Sup as Supervisor
    participant Gen as Generation
    participant Ref as Reflection
    participant Rank as Ranking
    participant Evo as Evolution
    participant MR as MetaReview
    participant DB as SQLite

    S->>Sup: goal + preferences
    Sup->>DB: INSERT session
    Sup->>DB: parse_goal -> ResearchPlan
    loop n_initial times
        Sup->>DB: enqueue CreateInitialHypotheses
    end

    loop Main Loop
        Sup->>DB: claim task
        alt Generation task
            Sup->>Gen: execute(task)
            Gen->>Gen: tool loop (search -> record_hypothesis)
            Gen->>DB: INSERT hypothesis (state=draft)
            Gen->>Sup: hypothesis_created
            Sup->>DB: enqueue ReviewHypothesis
        else Reflection task
            Sup->>Ref: execute(task)
            Ref->>Ref: tool loop (search -> record_review)
            Ref->>DB: INSERT review
            Ref->>DB: UPDATE hypothesis state -> reviewed
            Ref->>Sup: review_completed
            Sup->>DB: enqueue AddToTournament
        else Ranking task
            Sup->>Rank: execute(task)
            alt AddToTournament
                Rank->>DB: UPDATE hypothesis state -> in_tournament
            else RunTournamentBatch
                Rank->>Rank: select pair, run debate/pairwise
                Rank->>DB: INSERT match + elo_journal
                Rank->>DB: UPDATE hypothesis elos
            end
            Rank->>Sup: tournament_match_complete
        else Evolution task
            Sup->>Evo: execute(task)
            Evo->>Evo: tool loop (combine/simplify/out_of_box)
            Evo->>DB: INSERT hypothesis (state=draft)
            Evo->>Sup: hypothesis_created
            Sup->>DB: enqueue ReviewHypothesis
        else MetaReview task
            Sup->>MR: execute(task)
            alt System Feedback
                MR->>DB: INSERT system_feedback
                Note right of DB: Injected into next Gen/Evo prompts
            else Final Overview
                MR->>MR: synthesize top hypotheses
                MR->>DB: UPDATE session.final_overview
            end
        end
        Sup->>DB: check termination
    end

    Sup->>MR: GenerateFinalResearchOverview
    MR->>DB: UPDATE session.status -> done
    Sup->>S: session_id

Default Configuration

Key defaults from co_scientist/config.py:

Parameter	Value	Purpose
`run.concurrency`	4	Max parallel agent workers
`run.max_ideas`	60	Cap on total hypotheses
`run.wall_clock_seconds`	7200 (2h)	Session time limit
`run.budget_usd`	$25.00	Session USD cap
`ranking.elo_initial`	1200	Starting Elo for new hypotheses
`ranking.k_factor_new`	32	Fast convergence for new hypotheses
`ranking.k_factor_warm`	16	Stable Elo for experienced hypotheses
`termination.elo_stability_k`	5	Top-K to monitor for convergence
`termination.elo_stability_n`	3	Snapshots needed for stability
`termination.elo_stability_eps`	25.0	Max Elo drift within stable window
`evolution.min_mature`	20	Min hypotheses with >= 3 matches before evolution
`evolution.top_k`	5	Top hypotheses to evolve
`vectors.dedup_cosine_threshold`	0.92	Cosine similarity threshold for dedup
`lease.default_seconds`	300	Task lease TTL

Cross-references

co-scientist -- Project overview
co-scientist-prompts -- All 14 prompt templates in detail
co-scientist-elo-tournament -- The Elo ranking subsystem