WikifitaGitHub live67e8de5
pesquisa · kaggle/pokemon_tcg_top_elo_curriculum_filter

Pokémon TCG AI Battle — Top-Elo Curriculum Filter

Semantics of --top-elo N: how the filter queries agent_elo_daily, what it prunes at load time, coverage cost, and evidence-based findings from the BC curriculum ablation.

Baixar raw

Pokémon TCG AI Battle — Top-Elo Curriculum Filter

Purpose

Documents the --top-elo N flag on the MLX trainer: its exact query semantics, what it prunes at load time, its coverage cost, and what the pokemon_tcg_bc_curriculum_ablation revealed about when it helps versus hurts. See pokemon_tcg_train_config_reference for the flag in the full flag list and pokemon_tcg_glossary for base vocabulary.

Flag

--top-elo N     Filter to episodes played by top-N agents by daily remote elo
                (source=remote). Both player_name and opponent_name must be
                in the top-N set for that day. 0=off.

CLI-only; no config counterpart. 0 (or unset) disables the filter. There is no config key for it in TrainConfig; the flag is checkpoint-recorded in run_config.top_elo for provenance.

Fails closed, not open. Two failure modes prune to zero rather than fall through:

  • If the resolved Parquet corpus lacks player_name / opponent_name sidecar columns, the trainer hard-exits with --top-elo requires player_name/opponent_name columns in the parquet; rebuild the dataset (scripts/bc/bc_train_mlx.py:1985-1989).
  • If a resolved day has zero agent_elo_daily WHERE source='remote' rows, every episode of that day is dropped (allowed=None → continue → keep[i]=False, scripts/bc/bc_train_mlx.py:2011-2013). The filter does not silently pass those episodes through.

If the AND of the two _name in allowed checks empties the whole corpus, the trainer hard-exits with --top-elo filter left no episodes; lower N or check agent_elo_daily coverage.

Query semantics (exact)

The filter runs one single global query over all resolved days, not one query per day. Windowed ranking builds the per-day top-N set in a single scan (scripts/bc/bc_train_mlx.py:1995-2004):

SELECT day_id, name FROM (
  SELECT aed.day_id, a.name, aed.elo,
         ROW_NUMBER() OVER (PARTITION BY aed.day_id ORDER BY aed.elo DESC) AS rk
  FROM agent_elo_daily aed
  JOIN agents a ON a.id = aed.agent_id
  WHERE aed.source = 'remote'
) WHERE rk <= ?

? is int(a.top_elo). The result populates top_names_by_day: dict[int, set[str]] — one set per day, keyed by day_id.

An episode e on day d is kept iff both:

e.player_name   ∈ top_names_by_day[d]
e.opponent_name ∈ top_names_by_day[d]

The comparison is exact string match on the raw values (str(ep_player[i]) in allowed, scripts/bc/bc_train_mlx.py:2015). No case-folding, no whitespace normalization — the Parquet strings must equal the agents.name strings byte-for-byte.

Both-sides AND is intentional. A top-Elo agent playing against an unknown opponent is not kept: the opposing decisions in that episode belong to the unknown side and would poison the learned policy with weak counterfactuals. The side effect is that asymmetric games (e.g. top-1 vs top-500) also disappear even at moderate N — both endpoints have to make the cut simultaneously.

Implementation trace

  1. Argparse (scripts/bc/bc_train_mlx.py:1608-1617): --top-elo N, default None, help text points at agent_elo_daily WHERE source='remote'.
  2. Normalization (scripts/bc/bc_train_mlx.py:1786): a.top_elo = int(a.top_elo) if a.top_elo else 0.
  3. Column scan (scripts/bc/bc_train_mlx.py:1944-1966): when top_elo > 0, the pre-split scan adds player_name / opponent_name to the Parquet columns pulled with pa_dataset.to_batches.
  4. Per-episode canonicalization (scripts/bc/bc_train_mlx.py:1967-1979): np.unique(all_eids, return_index=True) picks one first_index per episode_id; ep_player[i] / ep_opponent[i] are the canonical name pair for episode i. Every row of a given episode carries identical values, so first-appearance is the cheapest representative.
  5. SQL scan (scripts/bc/bc_train_mlx.py:1994-2009): the single windowed query above, materialized into top_names_by_day.
  6. Filter mask (scripts/bc/bc_train_mlx.py:2010-2016): boolean keep[i] populated by day-scoped AND check, then unique_eids = unique_eids[keep] and downstream counts, ep_day_ids sliced identically.
  7. Log line (scripts/bc/bc_train_mlx.py:2019-2023): [bc-train-mlx] --top-elo N: kept K/T episode(s) played by top-N agents on both sides.
  8. Provenance (scripts/bc/bc_train_mlx.py:3772): run_config.top_elo is written into the checkpoint payload; a downstream reader always knows whether a model was trained with the filter and at which N.

Name provenance

player_name and opponent_name are string sidecar columns emitted by the Parquet builder, not derived at training time. Anchor: scripts/bc/build_bc_from_zips.py:262 — _META_STR = ("player_name", "opponent_name", "player_deck_hash", "opponent_deck_hash"). The names come straight from the Kaggle replay JSON via m.get("player_name") / m.get("opponent_name") (scripts/bc/build_bc_from_zips.py:404-405) and are the exact strings that also populate agents.name in the SQLite catalog during the same ingest. That is why the string join works without normalization.

What the filter does not touch

Being explicit about scope, since the filter has been confused with adjacent Elo-driven machinery:

  • Not deck-driven. deck_elo_daily and deck_cards exist in the SQLite schema and are populated (see pokemon_tcg_sqlite_schema_current), but the filter never reads them. Deck identity, deck Elo, deck archetype, and player_deck_hash play no role in what stays or leaves.
  • Not row-level. The filter never subsets to specific decisions within an episode; whole episodes stay or leave. Chunking / TBPTT and multi-select behavior are unchanged.
  • Not tied to what the agent will play against. The filter picks whose recorded replays we learn from. Whether those agents are also executable locally (as public agents, starters, or submissions) is unrelated — nothing in this code path checks for local runnability.
  • Not linked to --sweep-source. The tournament flag --sweep-source {remote,local} in scripts/tournament.py decides which decks rotate on the OUR-agent side during evaluation; it queries deck_elo_daily, not agent_elo_daily, in a different phase of the pipeline. The two share only the SQLite file.

Population reality at the August 7 analysis snapshot

At the snapshot used for this dated ablation analysis, the agent_elo_daily table held 6,552 rows over 22 days and 551 agents. source='remote' was the dominant population; source='local' required local tournaments and was thinly populated (and, again, unused by this filter).

The August 15 read-only revalidation reports 28,745 agent_elo_daily rows, 1,117 agents and 30 days in the live database. That later population must not be substituted into the historical ablation tables: the exact training corpus, run configuration and checkpoint remain the authority for each result. See pokemon_tcg_sqlite_schema_current and pokemon_tcg_current_state_reconciliation.

The trainer prints the exact retained-vs-total episode count per run in the [bc-train-mlx] --top-elo N: kept K/T episode(s)... line — that is the authoritative concentration number for a specific configuration, not a modelled estimate. Per-day breakdowns are not currently instrumented; if that ever matters, add it there rather than back-computing from downstream metrics.

What the filter is trying to do

The intent is a curriculum: train only on games where both sides played reasonably well, so the model learns competent decisions rather than noise from weak agents.

The intuition is correct in the limit of large datasets. The finding from the ablation is that in the current data-limited regime, the intent is defeated by coverage cost — see below.

Coverage cost is real

Weak-Elo agents disproportionately play with weak-Elo decks — starter decks (lb526_iono, lb510_mega_abomasnow_ex, lb600_dragapult_ex, lb600_mega_lucario_ex) and off-meta archetypes. Filter them out, and the training set contains almost no games with those decks on the board. The model never learns to counter them, so it plays terribly against public agents built on those archetypes.

Ablation evidence

From pokemon_tcg_bc_curriculum_ablation — 10 configs on (1d, 3d, 5d) × (1ep, 10ep) × top-Elo OFF/ON at N=50), each evaluated in a sweep-ON per-run tournament vs baselines + 4 starters + 3 strong public agents.

WR vs lb526_iono (canonical low-Elo starter deck):

runfiltervs lb526
1d_10ep_OFFoff5.0%
1d_10ep_ONon0.0%
3d_1ep_OFFoff1.7%
3d_1ep_ONon5.0%
3d_10ep_OFFoff8.3%
3d_10ep_ONon0.0%
5d_1ep_OFFoff5.0%
5d_1ep_ONon6.7%
5d_10ep_OFFoff1.7%
5d_10ep_ONon10.0%

The 0% zeros are the signal of no exposure: the filtered training set literally contained no episodes with lb526's archetype on the board, so the model has no policy for it. The signal closes at 5d_10ep_ON because five filtered days carry enough deck diversity for the archetype to sneak back in as a side effect.

Overall vs publics (all 9 chosen opponents)

pairOFFON
1d_10ep18.5%18.1%
3d_1ep18.3%20.4%
3d_10ep19.6%16.9%
5d_1ep18.0%17.8%
5d_10ep21.0%20.0%

The filter never wins in matched (days, epochs) pairs — OFF ties or beats ON in every quadrant on vs publics. The one ON improvement (3d_1ep) is inside the run-to-run noise floor.

Overfit signal on the filtered set

3d_10ep_ON produced the worst overall in the whole 10-run suite: 16.9%. Compare to 3d_1ep_ON = 20.4% overall (same days, filter on): 10 epochs on the small filtered set regressed 3.5 points. The filtered set at 3 days is small enough that repeated passes overspecialize. This is a training-regime bug (too many passes over too little data), not a filter bug per se.

Interpretation

The --top-elo filter is the right idea applied at the wrong granularity. It filters by agent Elo, which correlates with deck skill but not with deck diversity. What the ablation wants is a filter that concentrates on high-signal games while preserving archetype coverage. The SQLite schema already supports it — the deck_elo_daily and deck_cards tables let you stratify by deck top-N per archetype, or fill weak matchup cells first, or quota on (agent_top, deck_archetype) pairs. That is the next natural experiment; not scope for this page.

When to use --top-elo

Do use it when:

  • You genuinely want a small, high-quality subset for a smoke test.
  • You are OK with the model being blind to non-top-Elo decks (e.g. sparring against strong opponents only).

Do not use it when:

  • Comparing full-corpus regimes on vs publics — it will underperform OFF at matched scale.
  • Training a submission candidate — coverage cost is not worth the concentration gain in the current data-limited regime.

Cross-references