---
name: co-scientist-infrastructure
type: reference
title: "Co-Scientist Infrastructure: Providers, Tools, Storage, and Budgets"
description: "LLM provider abstraction, tool registry, FAISS vector store, SQLite schema, token budgets, event bus, retry logic, and batch API."
tags: [infrastructure, llm-providers, tool-registry, faiss, sqlite, token-budgets, batch-api, event-bus]
timestamp: 2026-07-21
---

# Co-Scientist Infrastructure

## LLM Provider Abstraction

### Architecture

The provider system uses a **Protocol-based** design. `LLMProvider` is a runtime-checkable protocol with a single method: `async def call(spec, ctx) -> AnthropicResponse`. All agents are coded against this protocol — they never import vendor-specific SDKs.

```
Agent -> AgentCallSpec -> LLMProvider.call() -> AnthropicResponse
                                  |
                    +-------------+-------------+
                    |             |             |
            AnthropicClient  OpenAIClient  (future)
            (native SDK)     (OpenAI SDK)
```

The intermediate request shape (`AgentCallSpec`) is Anthropic-flavored: `system_blocks`, `user_blocks`, `tools`, `tool_choice`, `max_output_tokens`, `thinking` budgets. Each provider translates this to its vendor's format. The response is normalized back to an `AnthropicResponse` with `.raw` exposing a Message-like object with `.content`, `.stop_reason`, `.usage`.

### The 9 Providers

| Provider | Endpoint | API Key Env Var | SDK | Notes |
|---|---|---|---|---|
| `anthropic` | api.anthropic.com | `ANTHROPIC_API_KEY` | `anthropic` | Full feature set: cache, thinking, batch |
| `openai` | api.openai.com | `OPENAI_API_KEY` | `openai` | Reasoning effort for o-series models |
| `openai_compatible` | Any endpoint | `OPENAI_API_KEY` | `openai` | Set `[llm.openai] base_url` |
| `openrouter` | openrouter.ai/api/v1 | `OPENROUTER_API_KEY` | `openai` | 200+ models, attribution headers |
| `gemini` | generativelanguage.googleapis.com | `GEMINI_API_KEY` | `openai` | OpenAI-compat endpoint |
| `google` | (alias for gemini) | `GEMINI_API_KEY` | `openai` | Alias |
| `groq` | api.groq.com | `GROQ_API_KEY` | `openai` | Fast Llama/Mixtral |
| `together` | api.together.xyz | `TOGETHER_API_KEY` | `openai` | Together AI |
| `mistral` | api.mistral.ai | `MISTRAL_API_KEY` | `openai` | Mistral la-plateforme |
| `ollama` | localhost:11434/v1 | (none) | `openai` | Local models |

**Key precedence:** For every OpenAI-compatible preset, `OPENAI_API_KEY` is used first if set. The provider-specific var is only the fallback.

### Provider Feature Support

| Feature | Anthropic | Everything Else |
|---|---|---|
| Tool / function call (required) | Native | Native OpenAI; must be supported on compat endpoints |
| Extended reasoning | `thinking` budgets | `reasoning_effort` for o-series models only |
| Prompt-cache breakpoints | `cache_control` | Stripped before sending |
| Batch API (50% off ranking) | Native | Not supported |

### Multi-Vendor Routing

For mixing vendors in a single session, use `provider = "openrouter"` and point each agent's model at a different vendor:

```toml
[llm]
provider = "openrouter"

[models]
generation = "openai/gpt-5"
reflection = "anthropic/claude-3.5-sonnet"
ranking_pairwise = "google/gemini-2.5-flash"
metareview_final = "meta-llama/llama-3.3-70b-instruct"
```

### Model Routing

The `route()` function maps `(agent, mode)` to a `ModelRoute` containing the model string and thinking budget:

| Agent.Mode | Default Model | Thinking |
|---|---|---|
| `generation.literature` | claude-opus-4-7 | 4000 |
| `generation.debate` | claude-opus-4-7 | 8000 |
| `reflection.full` | claude-opus-4-7 | 0 |
| `reflection.verification` | claude-opus-4-7 | 12000 |
| `reflection.observation` | claude-opus-4-7 | 6000 |
| `ranking.pairwise` | claude-sonnet-4-6 | 4000 |
| `ranking.debate` | claude-sonnet-4-6 | 8000 |
| `ranking.priority` | claude-opus-4-7 | 0 |
| `evolution.combine` | claude-opus-4-7 | 6000 |
| `evolution.out_of_box` | claude-opus-4-7 | 6000 |
| `evolution.feasibility` | claude-opus-4-7 | 0 |
| `evolution.simplify` | claude-opus-4-7 | 0 |
| `metareview.system` | claude-sonnet-4-6 | 8000 |
| `metareview.final` | claude-opus-4-7 | 16000 |
| `parse_goal` | claude-sonnet-4-6 | 0 |
| `classifier` | claude-haiku-4-5-20251001 | 0 |
| `judge` | claude-sonnet-4-6 | 0 |

**Degradation chain:** `claude-opus-4-7` -> `claude-sonnet-4-6` -> `claude-haiku-4-5-20251001`. Never-degrade modes: `reflection.verification`, `metareview.final`.

### Price Table

Cost estimation uses `PRICE_TABLE` (USD per 1M tokens) with family-hint fallbacks for unknown models. The table covers 40+ model identifiers across Anthropic, OpenAI, Google, Mistral, Meta, and OpenRouter.

Family hints: `flash-lite` < `flash` < `haiku` < `sonnet` < `opus`. Unknown models default to conservative sonnet-class pricing ($3/$15 input/output per 1M tokens).

### Retry Logic

| Condition | Max Attempts | Base Delay | Cap |
|---|---|---|---|
| HTTP 429 (rate limit) | 6 | 1000ms | 60000ms |
| HTTP 529 (overloaded) | 8 | 1000ms | 60000ms |
| HTTP 5xx | 5 | 1000ms | 60000ms |
| Timeout | 3 | 1000ms | 60000ms |

Per-call timeout: 120s (300s for thinking calls).

---

## Tool Registry

### Architecture

`ToolRegistry` discovers and indexes all available tools at session start. Tools are filtered per-agent via `AGENT_TOOLS` allowlists:

| Agent | Available Tools |
|---|---|
| `generation` | web_search, web_fetch, pubmed_search, arxiv_search, europe_pmc_search, literature_* |
| `reflection` | web_search, web_fetch, pubmed_search, arxiv_search, europe_pmc_search, literature_* |
| `ranking` | (none — no tools mid-debate) |
| `evolution` | web_search, web_fetch, pubmed_search, arxiv_search, europe_pmc_search, literature_* |
| `proximity` | (none) |
| `metareview` | (none) |

### Built-in Tools

| Tool | Description | Source |
|---|---|---|
| `web_fetch` | Fetch URL content, extract text via trafilatura | `co_scientist/tools/web_fetch.py` |
| `web_search` | Web search via Tavily or Brave API | `co_scientist/tools/web_search.py` |
| `pubmed_search` | PubMed article search | `co_scientist/tools/builtins/pubmed.py` |
| `arxiv_search` | ArXiv paper search | `co_scientist/tools/builtins/arxiv.py` |
| `europe_pmc_search` | Europe PMC search | `co_scientist/tools/builtins/europe_pmc.py` |

**Conditional registration:** `web_search` only registers if a backing API key (Tavily or Brave) is set. Otherwise the model sees a tool it cannot use, which causes smaller models to abort.

### Science Skills

The `ScienceSkillTool` discovers skills from `vendor/science-skills/` (a pinned clone of google-deepmind/science-skills). Skills are discovered via `SKILL.md` frontmatter and registered with `literature_*` wildcard matching.

### Tool Execution

Tools run via `asyncio.wait_for()` with configurable timeout (default 30s). Parallel dispatch is capped at `parallel_cap=4` concurrent tool calls per assistant turn.

---

## FAISS Vector Store

### Configuration

```toml
[embeddings]
provider = "voyage"          # voyage | openai
model = "voyage-3-large"
dim = 1024

[vectors]
backend = "faiss"
dedup_cosine_threshold = 0.92
cluster_threshold = 0.15
full_recluster_every_matches = 20
```

### Embedder Chain

1. **Voyage AI** (`voyage-3-large`, 1024 dim) — primary
2. **OpenAI** — fallback if Voyage key unavailable
3. **Hash-fallback** — deterministic embedding from text hash (degraded but functional)

### FAISS Index

- Type: `IndexFlatIP` (inner product = cosine on L2-normalized vectors)
- One index per session, stored at `data/artifacts/<session_id>/vectors/index.faiss`
- Metadata stored in `embeddings_meta` SQLite table
- Asyncio-locked for concurrent access
- Atomic save/load

### Dedup Flow

```
New hypothesis text
  -> embed (1024-dim vector)
  -> FAISS search (k=1)
  -> if cosine_similarity >= 0.92: return existing ID (skip insert)
  -> else: insert into FAISS + SQLite
```

### Clustering Flow

```
All session embeddings
  -> cosine similarity matrix (N x N)
  -> distance = 1.0 - similarity
  -> AgglomerativeClustering(threshold=0.15, linkage="average")
  -> assign cluster labels to hypotheses.dedup_cluster
```

---

## SQLite Schema (15 Tables)

The database uses WAL mode, foreign keys, busy_timeout=5000ms, and idempotent migration runner.

### Core Tables

| Table | Purpose | Key Fields |
|---|---|---|
| `sessions` | Session metadata + budget tracking | id, status, research_goal, research_plan (JSON), budget_tokens, budget_usd, wall_deadline, final_overview |
| `hypotheses` | All hypotheses | id, session_id, strategy, parent_ids (JSON), elo, matches_played, state, dedup_cluster |
| `reviews` | Reflection outputs | id, hypothesis_id, kind, verdict, novelty/correctness/testability/feasibility scores |
| `tournament_matches` | Head-to-head comparisons | id, hyp_a, hyp_b, mode, winner, elo_before/after, rationale |
| `elo_journal` | Append-only Elo ledger | Unique on match_id for idempotent updates |
| `tasks` | Durable task queue | id, agent, action, status, lease_owner, lease_expires_at, idempotency_key |
| `transcripts` | LLM call logs | agent, action, model, input/output tokens, cache_read/write, cost_usd |
| `system_feedback` | Human + meta-review feedback | source (human/meta_review), kind (directive/preference/rejection/pin/system_feedback) |
| `embeddings_meta` | FAISS index metadata | hypothesis_id, model, dim, faiss_offset, text_hash |
| `spans` | Observability traces | trace_id, parent_span_id, name, status |
| `events` | Event log for SSE + audit | session_id, agent, event, payload (JSON) |
| `schema_migrations` | Version tracking | version, applied_at |

### Hypothesis State Machine

```
draft -> reviewed -> in_tournament -> pinned
                \-> quarantined (safety)
                \-> rejected (manual)
                \-> retired (superseded by evolution)
```

### Task Queue

Tasks are claimed via `claim_one()` with lease-based locking:
- Default lease: 300s
- Reflection lease: 600s (longer tool loops)
- Meta-review final lease: 1800s (longest)
- Heartbeat: 60s
- Max attempts: 3 (then dead-letter)

Idempotency keys prevent duplicate task creation on retry (e.g., `{session_id}::generation::initial::{i}`).

---

## Token Budgets

### Per-Agent Shares

```toml
[budget_shares]
generation = 0.20
reflection = 0.20
ranking    = 0.25
evolution  = 0.15
metareview = 0.10
proximity  = 0.02
reserve    = 0.08
```

### Budget Enforcement

The `TokenBudget` class uses a reservation system:

1. **Admit** — Before each LLM call, check if the agent's share has headroom. Raise `BudgetExceeded` if not.
2. **Settle** — After the call, release the reservation and credit actual usage.

Concurrent agents share the same `TokenBudget` via `asyncio.Lock`. The session-wide cap (including reserve) is checked first, then the per-agent share.

### Cost Estimation

`estimate_cost_usd()` converts token usage to USD using `PRICE_TABLE`. The `co-scientist estimate` command runs a pre-flight cost estimate and warns if the projected cost exceeds 1.2x the configured budget.

---

## Event Bus

### Architecture

`EventBus` is an in-process pub/sub system for SSE (Server-Sent Events) delivery to the web UI.

```python
GLOBAL_BUS = EventBus()  # module-level singleton

# Publisher (agents)
await GLOBAL_BUS.publish(session_id, "match_complete", payload)

# Subscriber (SSE handler)
async for event in bus.subscribe(session_id):
    yield event.to_json()
```

### Design Decisions

- **Non-blocking publish** — Never awaits on slow subscribers. Uses `put_nowait()` with drop-oldest policy (max buffer 256).
- **Per-session queues** — Each subscriber gets its own `asyncio.Queue[Event]`.
- **Memory only** — On restart, the UI snapshots from the `events` table and reconnects for live updates.
- **Context-managed subscription** — `async with contextlib.aclosing(bus.subscribe(...)) as gen:` guarantees deterministic unregister.

### Event Types

| Event | Emitted By |
|---|---|
| `session_started` | Supervisor |
| `task_started` | Supervisor |
| `task_completed` | Supervisor |
| `task_failed` | Supervisor |
| `elo_snapshot` | Supervisor |
| `match_complete` | Ranking |
| `hypothesis_created` | Generation, Evolution |
| `session_done` | Supervisor |

---

## Batch API

### Purpose

Anthropic's Batch API provides ~50% cost reduction for tournament matches that don't need real-time verdicts. The system uses it for sub-decile matches — low-Elo pairs where the freshness of the verdict doesn't matter.

### Flow

```
RankingAgent detects sub-decile pair
  -> enqueue BatchedMatch into BatchPool
  -> Supervisor periodically calls submit_batch(min_size=4)
  -> BatchPool drains pending -> one Anthropic Batch API request
  -> poll_handles() checks for completion
  -> Reconcile completed results into tournament_matches + elo_journal
```

### Constraints

- Anthropic-only (other providers run all matches synchronously)
- Up to 24h latency (acceptable for sub-decile matches)
- High-Elo head of leaderboard always runs synchronously
- Minimum batch size of 4 to amortize overhead

---

## Web UI

### Stack

- **FastAPI** + **htmx** + **SSE** (Server-Sent Events)
- **Sanitized markdown renderer** for hypothesis/review display
- Default: `localhost:7878`

### Commands

```bash
co-scientist serve              # Start the dashboard
co-scientist report <id>        # Print final overview
co-scientist status <id>        # Session metadata + counts
co-scientist pause <id>         # Pause a running session
co-scientist resume <id>        # Resume a paused session
co-scientist abort <id>         # Abort a session
co-scientist feedback <id> --kind directive --text "focus on X"
```

---

## Configuration Layering

```
config/default.toml       <- shipped defaults
~/.co-scientist/config.toml  <- user global
./co-scientist.toml       <- project-local
--config <path>           <- CLI override
```

Layers are deep-merged (not replaced). Secrets come from environment variables only, never from config files.

## References

- See [[co-scientist]] for the system overview
- See [[co-scientist-agents]] for agent details
- See [[co-scientist-safety]] for injection defense and classifier
- See [[co-scientist-evaluation]] for the bench system