WikifitaGitHub live67e8de5
pesquisa · kaggle/pokemon_tcg_tournament_system

Pokémon TCG AI Battle — Tournament System (sweep + round-robin)

scripts/tournament.py CLI, sweep semantics, sweep-source contract, JSON reports, per-run tournament design, intra-suite round-robin as proto-self-play, and how the driver script wires everything for the ablation suite.

Baixar raw

Pokémon TCG AI Battle — Tournament System

Boundary

Documents scripts/tournament.py (uv run tcg-tournament) as it stands after the 2026-08-07 CLI enhancements. The tournament driver remains current, but the FP16 artifact names in the historical search and round-robin examples are not the current tensor-dtype authority; the live converter now enforces strict FP32. Complements pokemon_tcg_dashboard_and_tournament (dashboard-side arena and configuration), pokemon_tcg_evaluation_results (evaluation provenance) and pokemon_tcg_current_state_reconciliation. Terminology (sweep, sweep-source, baselines, round-robin) in pokemon_tcg_glossary.

What the tournament runs

For a single invocation:

  • OUR agent is loaded from agent/main.py, which reads a checkpoint by priority-ordered search. The first two paths retain the historical fp16 filename, but the current converter validates the loaded floating tensors as strict FP32:

    1. model/bc_model/bc_best_torch_fp16.pt
    2. model/checkpoint/bc_best_torch_fp16.pt
    3. model/bc_model/bc_best_mlx_final.pkl
    4. model/checkpoint/bc_best_mlx.pkl
    5. model/bc_model/bc_best_final.pkl
    6. model/checkpoint/bc_best.pkl

    In practice bc_best_torch_fp16.pt wins when that legacy-named file exists and was converted by tcg-build; the filename is not the current dtype authority. See pokemon_tcg_torch_inference.

  • Opponents are loaded from paths — either a submission .tar.gz or a directory containing main.py. Every game alternates sides internally (run_matchup handles this).

  • Games per opponent = --games N. If sweep is on, this is per deck per opponent.

Results are persisted to model/results.db (matches, match_steps, match_participants) so local Elo builds up over time.

CLI

usage: tcg-tournament [-h] [--games GAMES] [--opponent OPPONENT] [--note NOTE]
                      [--no-sweep] [--sweep-source {remote,local}]
                      [--skip-baselines] [--report-json PATH]
                      [--txt-backup] [--smoke]
flagpurpose
--games NGames per opponent (per deck when sweep is on). Default 20.
--opponent PATHAdd opponent. May be repeated to select a custom subset. Without any --opponent, iterates everything under public_agents/.
--no-sweepDisable the OUR-agent deck sweep. Only agent/deck.csv is used.
--sweep-source {remote,local}Elo tier the sweep pulls its top decks from. Default remote.
--skip-baselinesDrop the built-in random + first opponents.
--report-json PATHEmit the structured per-opponent/per-deck rows + overall summary as JSON.
--note STRAnnotation for this run (stored in SQLite).
--txt-backupAlso append results to eval_results.txt.
--smokeUse only public_agents/submissions/smoke/submission_smoke.tar.gz as OUR agent and force --no-sweep.

Baselines

Always prepended to the opponents list unless --skip-baselines:

  • random — picks a uniformly random legal option every decision. Zero reasoning, absolute noise floor.
  • first — picks the first legal option every decision. Deterministic, sometimes surprisingly hard to beat because the first legal option may be "pass turn."

If OUR agent does not beat random and first by a wide margin (>85% typical), something is broken structurally — encoding, checkpoint conversion, or the deck. This is the smoke test at every tournament run.

Deck sweep

When sweep is on (--no-sweep absent) and OUR agent is not the smoke bundle:

  1. Read up to top-3 decks from deck_elo_daily WHERE source = <sweep-source> on the most recent day with rows.
  2. Dedup against the current agent/deck.csv composition (90% quantity-aware similarity).
  3. Result is a list of up to 3–4 decks: default + unique top decks.
  4. For each opponent, iterate through the deck list, rewriting agent/deck.csv and calling our_module.reload_deck() between rotations. The agent module caches its deck at import, so reload_deck is required for the swap to actually take effect.
  5. original_deck is captured up front and always restored in a finally block, so a crash mid-sweep does not leave agent/deck.csv mutated.

Only OUR agent's deck rotates. Opponent agents are loaded once and their decks stay fixed. The sweep measures "how well does our model play each of OUR top decks against a fixed opposition," not archetype matchups.

The sweep is defined for OUR agent only because rewriting an opponent's deck would require rewriting an artifact from public_agents/, which is out of contract (opponents are read-only artifacts).

JSON report

--report-json PATH writes a single JSON with the full structured result:

{
  "our_agent": "/Users/alefita/workdir/pokemon-tcg/agent/main.py",
  "sweep": true,
  "sweep_source": "remote",
  "games_per_opponent": 20,
  "note": "suite 5d_10ep_OFF per-run",
  "rows": [
    {"opponent_label": "random", "opponent_path": "random",
     "deck_id": 1, "wins": 15, "losses": 5, "draws": 0,
     "wr_pct": 75.0, "elapsed_s": 4.32, "error": null},
    ...
  ],
  "overall": {"wins": 113, "losses": 426, "draws": 1,
              "wr_pct": 20.96, "elapsed_s": 275.4}
}

This is the contract callers depend on — no stdout scraping. The suite aggregator (experiments/bc_curriculum_suite.sh phase 3) reads these files to build the aggregated markdown at experiments/bc_curriculum_suite/results.md.

Per-run tournament (in the ablation suite)

For each of the 10 checkpoints, the suite runs a per-run tournament immediately after training:

uv run tcg-tournament \
  --games 20 --sweep-source remote \
  --opponent public_agents/starters/lb510_mega_abomasnow_ex \
  --opponent public_agents/starters/lb526_iono \
  --opponent public_agents/starters/lb600_dragapult_ex \
  --opponent public_agents/starters/lb600_mega_lucario_ex \
  --opponent public_agents/lb826_alakazam_seok \
  --opponent public_agents/lb945_multiply_ivan \
  --opponent public_agents/lb1009_mega_lucario_ex_islet \
  --note "suite <tag> per-run" \
  --report-json experiments/bc_curriculum_suite/reports/per_run/<tag>_tourn.json

Opponent shape: 2 baselines (auto) + 4 starters + 3 strong public agents. Sweep on → 3 decks per opponent. Total ≈ 9 opps × 3 decks × 20 = 540 games per run. Full findings in pokemon_tcg_bc_curriculum_ablation.

Round-robin (proto-self-play)

After all 10 checkpoints exist, the suite runs the intra-suite round-robin — 45 unique pairs, --no-sweep --skip-baselines, 30 games per pair. For each pair (i, j):

  1. Copy model/checkpoint/suite_<i>/<i>.pkl to model/checkpoint/bc_best_mlx.pkl.
  2. Run uv run tcg-build --checkpoint model/checkpoint/bc_best_mlx.pkl --out experiments/bc_curriculum_suite/models/<i>.tar.gz to convert the MLX checkpoint into the legacy-named model/bc_model/bc_best_torch_fp16.pt path (current converter contents are strict FP32, despite the filename) and refresh the <i>.tar.gz tarball.
  3. Run uv run tcg-tournament --games 30 --no-sweep --skip-baselines --opponent experiments/bc_curriculum_suite/models/<j>.tar.gz --report-json experiments/bc_curriculum_suite/reports/round_robin/<i>_vs_<j>.json.

The result is a 10×10 WR matrix, with the diagonal empty (a model does not play itself in an unordered round-robin). The mirror WR (j vs i from row j in the matrix) is computed as 100 - (i vs j) for cells that were played — draws are folded implicitly.

Why this is a step toward self-play. In classical self-play the current model plays against a snapshot of itself (or a lagged version, or a pool of past checkpoints). The round-robin here matches each of the 10 checkpoints against every other one as an opponent — which is exactly the "played against past selves" pattern, generalized to a hyperparameter fan-out. No RL loop is closed yet (no policy updates from these games), but the mechanics — model-vs-model, WR matrix, no external baselines contaminating the signal — are the same substrate an RL self-play loop would use to sample opponents.

Empirically the peer round-robin distribution sits at 44–53% across all 10 models, a useful within-cohort sanity signal. It is not a general validation of the later objective or validation contract; the Stage 3/4 postmortem records a separate training failure. See pokemon_tcg_bc_curriculum_ablation and pokemon_tcg_stage3_training_failure_postmortem.

Anti-correlation: vs publics vs vs peers

From the same 10-model matrix vs the per-run tournament results:

modelvs publicsvs peers
5d_10ep_OFF21.0% (1st)47.8% (8th)
5d_10ep_ON20.0% (2nd)44.1% (10th)
3d_1ep_ON20.4%53.2% (1st tie)
5d_1ep_ON17.8%53.2% (1st tie)

Models that specialize enough to beat public agents (fewer generalization moves, more dataset-specific patterns) become predictable to peers trained on the same distribution. Models that stay generic (1ep) beat peers but do not beat publics.

Consequence for submission choice. vs publics is the ladder-Elo proxy — the Kaggle ladder is not populated by BC clones of this suite. The intra-suite round-robin is a diagnostic (for architecture sanity and specialization), not a submission signal.

Where results live

  • SQLite — every game creates rows in matches, match_steps, match_participants, match_card_usage. This is what enables local Elo to grow over time via compute_agent_elo(source='local').
  • JSON reports — --report-json writes one per invocation. The suite lays them out under experiments/bc_curriculum_suite/reports/{per_run,round_robin}/.
  • stdout — human-readable table (Opponent / W / L / D / WinRate / Time), one row per (opponent, deck) with a per-deck performance summary when sweep is on.
  • eval_results.txt — text log, optional with --txt-backup.

Cross-references