WikifitaGitHub live67e8de5
outro · co-scientist/co-scientist-infrastructure

Co-Scientist Infrastructure: Providers, Tools, Storage, and Budgets

LLM provider abstraction, tool registry, FAISS vector store, SQLite schema, token budgets, event bus, retry logic, and batch API.

Baixar raw

Co-Scientist Infrastructure

LLM Provider Abstraction

Architecture

The provider system uses a Protocol-based design. LLMProvider is a runtime-checkable protocol with a single method: async def call(spec, ctx) -> AnthropicResponse. All agents are coded against this protocol — they never import vendor-specific SDKs.

Agent -> AgentCallSpec -> LLMProvider.call() -> AnthropicResponse
                                  |
                    +-------------+-------------+
                    |             |             |
            AnthropicClient  OpenAIClient  (future)
            (native SDK)     (OpenAI SDK)

The intermediate request shape (AgentCallSpec) is Anthropic-flavored: system_blocks, user_blocks, tools, tool_choice, max_output_tokens, thinking budgets. Each provider translates this to its vendor's format. The response is normalized back to an AnthropicResponse with .raw exposing a Message-like object with .content, .stop_reason, .usage.

The 9 Providers

ProviderEndpointAPI Key Env VarSDKNotes
anthropicapi.anthropic.comANTHROPIC_API_KEYanthropicFull feature set: cache, thinking, batch
openaiapi.openai.comOPENAI_API_KEYopenaiReasoning effort for o-series models
openai_compatibleAny endpointOPENAI_API_KEYopenaiSet [llm.openai] base_url
openrouteropenrouter.ai/api/v1OPENROUTER_API_KEYopenai200+ models, attribution headers
geminigenerativelanguage.googleapis.comGEMINI_API_KEYopenaiOpenAI-compat endpoint
google(alias for gemini)GEMINI_API_KEYopenaiAlias
groqapi.groq.comGROQ_API_KEYopenaiFast Llama/Mixtral
togetherapi.together.xyzTOGETHER_API_KEYopenaiTogether AI
mistralapi.mistral.aiMISTRAL_API_KEYopenaiMistral la-plateforme
ollamalocalhost:11434/v1(none)openaiLocal models

Key precedence: For every OpenAI-compatible preset, OPENAI_API_KEY is used first if set. The provider-specific var is only the fallback.

Provider Feature Support

FeatureAnthropicEverything Else
Tool / function call (required)NativeNative OpenAI; must be supported on compat endpoints
Extended reasoningthinking budgetsreasoning_effort for o-series models only
Prompt-cache breakpointscache_controlStripped before sending
Batch API (50% off ranking)NativeNot supported

Multi-Vendor Routing

For mixing vendors in a single session, use provider = "openrouter" and point each agent's model at a different vendor:

[llm]
provider = "openrouter"

[models]
generation = "openai/gpt-5"
reflection = "anthropic/claude-3.5-sonnet"
ranking_pairwise = "google/gemini-2.5-flash"
metareview_final = "meta-llama/llama-3.3-70b-instruct"

Model Routing

The route() function maps (agent, mode) to a ModelRoute containing the model string and thinking budget:

Agent.ModeDefault ModelThinking
generation.literatureclaude-opus-4-74000
generation.debateclaude-opus-4-78000
reflection.fullclaude-opus-4-70
reflection.verificationclaude-opus-4-712000
reflection.observationclaude-opus-4-76000
ranking.pairwiseclaude-sonnet-4-64000
ranking.debateclaude-sonnet-4-68000
ranking.priorityclaude-opus-4-70
evolution.combineclaude-opus-4-76000
evolution.out_of_boxclaude-opus-4-76000
evolution.feasibilityclaude-opus-4-70
evolution.simplifyclaude-opus-4-70
metareview.systemclaude-sonnet-4-68000
metareview.finalclaude-opus-4-716000
parse_goalclaude-sonnet-4-60
classifierclaude-haiku-4-5-202510010
judgeclaude-sonnet-4-60

Degradation chain: claude-opus-4-7 -> claude-sonnet-4-6 -> claude-haiku-4-5-20251001. Never-degrade modes: reflection.verification, metareview.final.

Price Table

Cost estimation uses PRICE_TABLE (USD per 1M tokens) with family-hint fallbacks for unknown models. The table covers 40+ model identifiers across Anthropic, OpenAI, Google, Mistral, Meta, and OpenRouter.

Family hints: flash-lite < flash < haiku < sonnet < opus. Unknown models default to conservative sonnet-class pricing (3/3/15 input/output per 1M tokens).

Retry Logic

ConditionMax AttemptsBase DelayCap
HTTP 429 (rate limit)61000ms60000ms
HTTP 529 (overloaded)81000ms60000ms
HTTP 5xx51000ms60000ms
Timeout31000ms60000ms

Per-call timeout: 120s (300s for thinking calls).

Tool Registry

Architecture

ToolRegistry discovers and indexes all available tools at session start. Tools are filtered per-agent via AGENT_TOOLS allowlists:

AgentAvailable Tools
generationweb_search, web_fetch, pubmed_search, arxiv_search, europe_pmc_search, literature_*
reflectionweb_search, web_fetch, pubmed_search, arxiv_search, europe_pmc_search, literature_*
ranking(none — no tools mid-debate)
evolutionweb_search, web_fetch, pubmed_search, arxiv_search, europe_pmc_search, literature_*
proximity(none)
metareview(none)

Built-in Tools

ToolDescriptionSource
web_fetchFetch URL content, extract text via trafilaturaco_scientist/tools/web_fetch.py
web_searchWeb search via Tavily or Brave APIco_scientist/tools/web_search.py
pubmed_searchPubMed article searchco_scientist/tools/builtins/pubmed.py
arxiv_searchArXiv paper searchco_scientist/tools/builtins/arxiv.py
europe_pmc_searchEurope PMC searchco_scientist/tools/builtins/europe_pmc.py

Conditional registration: web_search only registers if a backing API key (Tavily or Brave) is set. Otherwise the model sees a tool it cannot use, which causes smaller models to abort.

Science Skills

The ScienceSkillTool discovers skills from vendor/science-skills/ (a pinned clone of google-deepmind/science-skills). Skills are discovered via SKILL.md frontmatter and registered with literature_* wildcard matching.

Tool Execution

Tools run via asyncio.wait_for() with configurable timeout (default 30s). Parallel dispatch is capped at parallel_cap=4 concurrent tool calls per assistant turn.

FAISS Vector Store

Configuration

[embeddings]
provider = "voyage"          # voyage | openai
model = "voyage-3-large"
dim = 1024

[vectors]
backend = "faiss"
dedup_cosine_threshold = 0.92
cluster_threshold = 0.15
full_recluster_every_matches = 20

Embedder Chain

  1. Voyage AI (voyage-3-large, 1024 dim) — primary
  2. OpenAI — fallback if Voyage key unavailable
  3. Hash-fallback — deterministic embedding from text hash (degraded but functional)

FAISS Index

  • Type: IndexFlatIP (inner product = cosine on L2-normalized vectors)
  • One index per session, stored at data/artifacts/<session_id>/vectors/index.faiss
  • Metadata stored in embeddings_meta SQLite table
  • Asyncio-locked for concurrent access
  • Atomic save/load

Dedup Flow

New hypothesis text
  -> embed (1024-dim vector)
  -> FAISS search (k=1)
  -> if cosine_similarity >= 0.92: return existing ID (skip insert)
  -> else: insert into FAISS + SQLite

Clustering Flow

All session embeddings
  -> cosine similarity matrix (N x N)
  -> distance = 1.0 - similarity
  -> AgglomerativeClustering(threshold=0.15, linkage="average")
  -> assign cluster labels to hypotheses.dedup_cluster

SQLite Schema (15 Tables)

The database uses WAL mode, foreign keys, busy_timeout=5000ms, and idempotent migration runner.

Core Tables

TablePurposeKey Fields
sessionsSession metadata + budget trackingid, status, research_goal, research_plan (JSON), budget_tokens, budget_usd, wall_deadline, final_overview
hypothesesAll hypothesesid, session_id, strategy, parent_ids (JSON), elo, matches_played, state, dedup_cluster
reviewsReflection outputsid, hypothesis_id, kind, verdict, novelty/correctness/testability/feasibility scores
tournament_matchesHead-to-head comparisonsid, hyp_a, hyp_b, mode, winner, elo_before/after, rationale
elo_journalAppend-only Elo ledgerUnique on match_id for idempotent updates
tasksDurable task queueid, agent, action, status, lease_owner, lease_expires_at, idempotency_key
transcriptsLLM call logsagent, action, model, input/output tokens, cache_read/write, cost_usd
system_feedbackHuman + meta-review feedbacksource (human/meta_review), kind (directive/preference/rejection/pin/system_feedback)
embeddings_metaFAISS index metadatahypothesis_id, model, dim, faiss_offset, text_hash
spansObservability tracestrace_id, parent_span_id, name, status
eventsEvent log for SSE + auditsession_id, agent, event, payload (JSON)
schema_migrationsVersion trackingversion, applied_at

Hypothesis State Machine

draft -> reviewed -> in_tournament -> pinned
                \-> quarantined (safety)
                \-> rejected (manual)
                \-> retired (superseded by evolution)

Task Queue

Tasks are claimed via claim_one() with lease-based locking:

  • Default lease: 300s
  • Reflection lease: 600s (longer tool loops)
  • Meta-review final lease: 1800s (longest)
  • Heartbeat: 60s
  • Max attempts: 3 (then dead-letter)

Idempotency keys prevent duplicate task creation on retry (e.g., {session_id}::generation::initial::{i}).

Token Budgets

Per-Agent Shares

[budget_shares]
generation = 0.20
reflection = 0.20
ranking    = 0.25
evolution  = 0.15
metareview = 0.10
proximity  = 0.02
reserve    = 0.08

Budget Enforcement

The TokenBudget class uses a reservation system:

  1. Admit — Before each LLM call, check if the agent's share has headroom. Raise BudgetExceeded if not.
  2. Settle — After the call, release the reservation and credit actual usage.

Concurrent agents share the same TokenBudget via asyncio.Lock. The session-wide cap (including reserve) is checked first, then the per-agent share.

Cost Estimation

estimate_cost_usd() converts token usage to USD using PRICE_TABLE. The co-scientist estimate command runs a pre-flight cost estimate and warns if the projected cost exceeds 1.2x the configured budget.

Event Bus

Architecture

EventBus is an in-process pub/sub system for SSE (Server-Sent Events) delivery to the web UI.

GLOBAL_BUS = EventBus()  # module-level singleton

# Publisher (agents)
await GLOBAL_BUS.publish(session_id, "match_complete", payload)

# Subscriber (SSE handler)
async for event in bus.subscribe(session_id):
    yield event.to_json()

Design Decisions

  • Non-blocking publish — Never awaits on slow subscribers. Uses put_nowait() with drop-oldest policy (max buffer 256).
  • Per-session queues — Each subscriber gets its own asyncio.Queue[Event].
  • Memory only — On restart, the UI snapshots from the events table and reconnects for live updates.
  • Context-managed subscription — async with contextlib.aclosing(bus.subscribe(...)) as gen: guarantees deterministic unregister.

Event Types

EventEmitted By
session_startedSupervisor
task_startedSupervisor
task_completedSupervisor
task_failedSupervisor
elo_snapshotSupervisor
match_completeRanking
hypothesis_createdGeneration, Evolution
session_doneSupervisor

Batch API

Purpose

Anthropic's Batch API provides ~50% cost reduction for tournament matches that don't need real-time verdicts. The system uses it for sub-decile matches — low-Elo pairs where the freshness of the verdict doesn't matter.

Flow

RankingAgent detects sub-decile pair
  -> enqueue BatchedMatch into BatchPool
  -> Supervisor periodically calls submit_batch(min_size=4)
  -> BatchPool drains pending -> one Anthropic Batch API request
  -> poll_handles() checks for completion
  -> Reconcile completed results into tournament_matches + elo_journal

Constraints

  • Anthropic-only (other providers run all matches synchronously)
  • Up to 24h latency (acceptable for sub-decile matches)
  • High-Elo head of leaderboard always runs synchronously
  • Minimum batch size of 4 to amortize overhead

Web UI

Stack

  • FastAPI + htmx + SSE (Server-Sent Events)
  • Sanitized markdown renderer for hypothesis/review display
  • Default: localhost:7878

Commands

co-scientist serve              # Start the dashboard
co-scientist report <id>        # Print final overview
co-scientist status <id>        # Session metadata + counts
co-scientist pause <id>         # Pause a running session
co-scientist resume <id>        # Resume a paused session
co-scientist abort <id>         # Abort a session
co-scientist feedback <id> --kind directive --text "focus on X"

Configuration Layering

config/default.toml       <- shipped defaults
~/.co-scientist/config.toml  <- user global
./co-scientist.toml       <- project-local
--config <path>           <- CLI override

Layers are deep-merged (not replaced). Secrets come from environment variables only, never from config files.

References