Elo as an Experimental Protocol
Cross-project comparison of Elo-like ratings as relative-evidence protocols across Pokémon TCG, Antigravity and Co-Scientist research systems.
Elo as an Experimental Protocol
Question
Several Wikifita projects use Elo-like updates to summarize pairwise comparisons. The shared idea is useful: a result changes a relative estimate, and the estimate is meaningful only with its comparison population, update rule and evidence source. The shared name does not make the resulting numbers interoperable.
This page registers the methodological relationship. It does not create a common leaderboard or claim that a Pokémon deck, a mathematical hypothesis and a research model occupy one latent scale.
Comparison map
| Project | Rated object | Initial or anchor convention | Evidence boundary |
|---|---|---|---|
| Pokémon TCG AI Battle | Agents, decks and cards; submission/model lineages remain partly planned | Current database bootstrap is 600; local and remote evidence are separated | Live SQLite tables, tournament cohorts and ResultsDB; invariant-deck calibration is experimental and source/cohort dependent |
| Antigravity 2.0 | Mathematical hypotheses compared with published or human reference results | Research page records a 1200 reset and a hypothesis-oriented scale | Handoff/research ledger; Elo expresses relative evaluation, not proof of a theorem |
| Co-Scientist | Candidate research systems or hypotheses in head-to-head evaluation | Its page defines its own initial scale, K-factor choices, match IDs and pair selection | Gold-set and comparison protocol defined by the Co-Scientist project |
| Red Team Arena | Safety-system evaluation pools | Separate arena conventions and rating pools | Attack, defense and judge outcomes; not a substitute for the other project ledgers |
What transfers between projects
The following abstractions transfer safely:
- every rating needs an explicit object, population and update rule;
- a rating is relational evidence, not an intrinsic essence of the rated object;
- resets, source partitions and sampling change comparability;
- deterministic match identifiers and provenance make repeated comparisons auditable;
- a rating table should be accompanied by uncertainty, cohort and evidence status;
- a strong score does not prove the mechanism or theory that produced it.
These principles connect the Pokémon TCG database work to the broader evaluation methodology in co-scientist-elo-tournament, antigravity-2.0 and red-team-arena-elo.
What does not transfer
The following operations would be invalid without a new calibration study:
- averaging scores across projects;
- treating the 600 bootstrap in Pokémon TCG as equivalent to a 1200 reset in Antigravity or Co-Scientist;
- comparing K-factors or draw handling without matching the update equations;
- interpreting a deck's invariant rating as a model or submission rating;
- using a PageRank score as an Elo score merely because both are rank vectors;
- using a tournament result as proof of the proposed RoPEND, MoE or PageRank-Abelian architecture when the corresponding runtime is absent.
Pokémon TCG as the detailed case
The Pokémon TCG work shows why this distinction matters. The repository has ordinary source-separated daily ratings, an invariant-deck calculation that combines a clipped win-rate inversion, MD10 shrinkage and a local-to-remote translation, and a research proposal connecting translation invariance to a sampled graph. These are three related layers, not one finished metric.
The current implementation also exposes a draw-handling difference: the
invariant-deck path uses wins / games_played, while the ordinary online update
uses a 0.5 score for a draw. A future comparison must state which path was used
before interpreting a number. See pokemon_tcg_submissions_and_elo and
pokemon_tcg_elo_pagerank_invariance.
Research use
For future experiments, record the following tuple with every Elo-like result:
project / rated object / source / cohort or reset / update rule /
match identifier scheme / sample size / uncertainty / claim class
The claim class should distinguish an observed result, a code-validated calculation, a source-reported handoff result, an inference and a proposal. This is especially important when an orchestration transcript or a manuscript uses completion language that the live source tree does not confirm.
Primary sources
- pokemon_tcg_elo_pagerank_invariance — Pokémon TCG rating and graph bridge.
- pokemon_tcg_submissions_and_elo — source-separated rating lineages and current implementation boundary.
- co-scientist-elo-tournament — Co-Scientist comparison protocol.
- antigravity-2.0 — Antigravity hypothesis-ranking protocol.
- red-team-arena-elo — safety-evaluation rating pools.
Cross-references
- pokemon_tcg_orchestration_artifact_ledger — handoff artifacts and acceptance gates.
- pokemon_tcg_cross_harness_provenance — transcript and cross-harness provenance.
- multi-agent-methodology — broader agent-evaluation and review methodology.