Pokémon TCG AI Battle — BC Curriculum Ablation (2026-08-06/07)
Dated 10-config behavioral-cloning ablation over (days × epochs × top-Elo curriculum) with per-run tournaments and intra-suite round-robin; its architecture sanity result is cohort-scoped and does not supersede the later Stage 3/4 training-contract incident.
Pokémon TCG AI Battle — BC Curriculum Ablation (2026-08-06/07)
Scope
Ablation study run over develop in ~/workdir/pokemon-tcg on 2026-08-06 to 2026-08-07. Driver script: experiments/bc_curriculum_suite.sh. Full JSON reports at experiments/bc_curriculum_suite/reports/, aggregated markdown at experiments/bc_curriculum_suite/results.md. Ten checkpoints were trained on the same base pipeline described in pokemon_tcg_training_overhaul_2026_07_29 and pokemon_tcg_tbptt_training_contract — MLX FP16 training, PyTorch FP16 arena inference, TBPTT always on. This is a dated empirical cohort; its FP16 runtime is preserved as provenance and is not the current source contract.
Runtime split for this historical phase was fixed as training in MLX, inference in PyTorch FP16 and recurrent TBPTT. The current strict-FP32 correction is recorded in pokemon_tcg_current_state_reconciliation and pokemon_tcg_training_pipeline.
The phrase “data-limited, not architecture-limited” below is a conclusion within this dated suite. It means that the suite did not isolate a broad encoding, conversion, cache or model-collapse failure under its own logs. It does not certify the later training contract: the Stage 3/4 reconstruction found an auxiliary-loss scale mismatch and an in-pool validation boundary. Read pokemon_tcg_stage3_training_failure_postmortem before extending this sanity result to the current architecture.
Experiment matrix
(1d, 3d, 5d) × (1ep, 10ep) × (top-elo OFF, ON) = 10 runs
Fixed knobs across all 10: batch 1024, TBPTT chunk 16, val batch 1024, lr 2.46e-4, warmup 20 steps, aux_return_weight 1.0, --max-rows-per-day 30000 (uniform cap per calendar day). --top-elo 50 on the ON side filters episodes to those played by top-50 agents (source='remote' from Kaggle replays) on both sides.
The 30k/day cap was chosen for suite parity — not because we believe it is the right training-time data volume. Larger runs will drop it.
Per-run tournament — sweep ON, --sweep-source remote (3 decks: default + top 2 by remote deck Elo, deduplicated), 20 games/opponent/deck, 9 opponents = 2 baselines (random, first) + 4 starters (lb510_mega_abomasnow_ex, lb526_iono, lb600_dragapult_ex, lb600_mega_lucario_ex) + 3 strong public agents (lb826_alakazam_seok, lb945_multiply_ivan, lb1009_mega_lucario_ex_islet). ~540 games per run.
Intra-suite round-robin — no sweep, no baselines, 30 games per pair, all C(10,2) = 45 pairs. For each pair (i, j) the OUR-agent slot is loaded from model/checkpoint/suite_${i}/${i}.pkl (via bc_best_mlx.pkl swap → tcg-build → model/bc_model/bc_best_torch_fp16.pt), and the opponent slot is the pre-packaged experiments/bc_curriculum_suite/models/${j}.tar.gz.
Per-run tournament — vs public agents
Aggregated across sweep decks. Numbers are win rates over wins + losses (draws excluded from the denominator).
| run | overall | random | first | lb510 | lb526 | lb600d | lb600m | lb826 | lb945 | lb1009 |
|---|---|---|---|---|---|---|---|---|---|---|
| 1d_10ep_OFF | 18.5% | 63.3% | 38.3% | 10.0% | 5.0% | 20.0% | 6.7% | 18.3% | 3.3% | 1.7% |
| 1d_10ep_ON | 18.1% | 70.0% | 28.3% | 5.0% | 0.0% | 5.0% | 23.3% | 23.3% | 5.0% | 3.3% |
| 3d_1ep_OFF | 18.3% | 71.7% | 21.7% | 13.3% | 1.7% | 16.7% | 11.7% | 18.3% | 5.0% | 5.0% |
| 3d_1ep_ON | 20.4% | 76.7% | 43.3% | 11.7% | 5.0% | 11.7% | 15.0% | 16.7% | 3.3% | 0.0% |
| 3d_10ep_OFF | 19.6% | 70.0% | 33.3% | 15.0% | 8.3% | 11.7% | 6.7% | 20.0% | 1.7% | 10.0% |
| 3d_10ep_ON | 16.9% | 60.0% | 38.3% | 8.3% | 0.0% | 15.0% | 6.7% | 20.0% | 3.3% | 0.0% |
| 5d_1ep_OFF | 18.0% | 71.7% | 31.7% | 13.3% | 5.0% | 8.3% | 8.3% | 20.0% | 3.3% | 0.0% |
| 5d_1ep_ON | 17.8% | 68.3% | 35.0% | 13.3% | 6.7% | 3.3% | 5.0% | 21.7% | 3.3% | 3.3% |
| 5d_10ep_OFF | 21.0% | 75.0% | 40.0% | 10.0% | 1.7% | 8.3% | 13.3% | 30.0% | 3.3% | 6.8% |
| 5d_10ep_ON | 20.0% | 73.3% | 31.7% | 16.7% | 10.0% | 5.0% | 6.7% | 33.3% | 1.7% | 1.7% |
Intra-suite round-robin — WR of row-model against column-model
| our \ opp | 1d_10ep_OFF | 1d_10ep_ON | 3d_10ep_OFF | 3d_10ep_ON | 3d_1ep_OFF | 3d_1ep_ON | 5d_10ep_OFF | 5d_10ep_ON | 5d_1ep_OFF | 5d_1ep_ON | avg |
|---|---|---|---|---|---|---|---|---|---|---|---|
| 1d_10ep_OFF | — | 70.0 | 60.0 | 60.0 | 43.3 | 34.5 | 53.3 | 40.0 | 50.0 | 53.3 | 51.6 |
| 1d_10ep_ON | 30.0 | — | 56.7 | 50.0 | 43.3 | 40.0 | 66.7 | 53.3 | 53.3 | 48.3 | 49.1 |
| 3d_10ep_OFF | 40.0 | 43.3 | — | 50.0 | 53.3 | 50.0 | 53.3 | 63.3 | 53.3 | 50.0 | 50.7 |
| 3d_10ep_ON | 40.0 | 50.0 | 50.0 | — | 50.0 | 46.7 | 40.0 | 70.0 | 46.7 | 46.7 | 48.9 |
| 3d_1ep_OFF | 56.7 | 56.7 | 46.7 | 50.0 | — | 43.3 | 46.7 | 66.7 | 63.3 | 46.7 | 53.0 |
| 3d_1ep_ON | 65.5 | 60.0 | 50.0 | 53.3 | 56.7 | — | 66.7 | 43.3 | 36.7 | 46.7 | 53.2 |
| 5d_10ep_OFF | 46.7 | 33.3 | 46.7 | 60.0 | 53.3 | 33.3 | — | 73.3 | 40.0 | 43.3 | 47.8 |
| 5d_10ep_ON | 60.0 | 46.7 | 36.7 | 30.0 | 33.3 | 56.7 | 26.7 | — | 56.7 | 50.0 | 44.1 |
| 5d_1ep_OFF | 50.0 | 46.7 | 46.7 | 53.3 | 36.7 | 63.3 | 60.0 | 43.3 | — | 36.7 | 48.5 |
| 5d_1ep_ON | 46.7 | 51.7 | 50.0 | 53.3 | 53.3 | 53.3 | 56.7 | 50.0 | 63.3 | — | 53.2 |
Findings
1. Coverage cost of the Elo filter is real and closes at scale
Every ON run except 5d_10ep_ON scored 0.0% vs lb526_iono — a low-Elo starter deck that top-50 filtering excludes. The model literally never trains on episodes that involve that archetype, so it does not know how to play against it. This is the assignment predicted by the coverage hypothesis.
5d_10ep_ON finally reaches 10.0% vs lb526 because 5 filtered days carry enough deck diversity for the archetype to reappear in the training set. Curriculum-by-agent-Elo does not fundamentally exclude decks; it excludes them until the filtered day count is high enough that they reappear as a side effect.
2. Top-elo filter never wins in matched pairs
For every (days, epochs) quadrant on vs publics, the ON side ties or loses to OFF:
| pair | OFF | ON |
|---|---|---|
| 1d_10ep | 18.5% | 18.1% |
| 3d_1ep | 18.3% | 20.4% |
| 3d_10ep | 19.6% | 16.9% |
| 5d_1ep | 18.0% | 17.8% |
| 5d_10ep | 21.0% | 20.0% |
The only ON improvement (3d_1ep) is inside the noise floor. On the largest scale (5d_10ep), OFF wins by 1.0 point. Curriculum by agent-Elo is the wrong cut while the model is data-limited. See discussion in pokemon_tcg_data_pipeline for the alternate cuts the sqlite structure already supports.
3. Overfit on the filtered set
3d_10ep_ON = 16.9% overall, worst of the ten. Compare to 3d_1ep_ON = 20.4%: 10 epochs on the same filtered set regresses by 3.5 points. The filtered set at 3 days is small enough that repeated passes overspecialize. This is a training-regime bug, not a filter bug — it repeats on random too (60% vs 76.7%).
4. Anti-correlation between vs-publics and vs-peers
| run | vs publics | vs peers | rank swap |
|---|---|---|---|
| 5d_10ep_OFF | 21.0% (1st) | 47.8% (8th) | ↓ 7 |
| 5d_10ep_ON | 20.0% (2nd) | 44.1% (10th) | ↓ 8 |
| 3d_1ep_ON | 20.4% | 53.2% (1st tie) | stable |
| 5d_1ep_ON | 17.8% | 53.2% (1st tie) | ↑ |
| 3d_1ep_OFF | 18.3% | 53.0% | ↑ |
Models that peel off the peer distribution to specialize on public agents (5d_10ep_*) become predictable to other peers trained on similar data. Models that stay generic (1ep) beat peers but do not beat publics as reliably.
Consequence for submission choice: vs publics is the proxy for real ladder Elo, since the ladder does not contain BC clones of this suite. The intra-suite ranking is a diagnostic, not a submission signal. See pokemon_tcg_evaluation_results for the general framing on how eval numbers compare and do not compare.
5. Architecture sanity within this historical suite
The suite did not expose the following broad failure modes in its logged runs:
| failure mode | evidence it did not happen |
|---|---|
| broken encoding | training loss decreased monotonically on every run (5d_10ep_OFF: 3.49 → 1.12) |
| MLX → PyTorch conversion drift | tournaments emitted zero ERR opponents in ~5,400 games; agent always chose legal actions |
| loss misalignment | val_acc improved with data and epochs (5d_10ep_OFF: 0.53 → 0.61) |
| FP16 overflow/underflow | no NaN observed in any run; grad_norm bounded |
| KV cache corruption | val_acc improved monotonically; no erratic regression |
| optimizer contract broken | aux heads (ko_bce, prize_mse, terminal_bce, return_mse) all decreased on every run |
| collapsed model | intra-suite peer WR sits in 44–53% across all 10 — a normal Elo dispersion; no model in 15–25% |
The peer round-robin distribution being centered at ~50% with 44–53% spread is a useful architecture sanity signal for this cohort: 10 models drawn from the same distribution split games evenly. It is not an independent proof that the shared trunk, objective and validation contract were healthy in every later stage.
Within this suite, the overall ceiling of ~20% vs public agents was interpreted as data- or regime-limited rather than a broad architecture failure. Increasing epochs on the suite's 30k/day slice hurt (see finding 3). The later Stage 3/4 evidence prevents that observation from becoming a current universal claim about the Transformer or the training pipeline.
Operational notes from the run
Two failures during the suite that are worth recording so they do not repeat:
- Session teardown killed a background training run: the first launch attempted
setsid, which is unavailable on macOS. Later/compactreparented the shell process group and SIGHUP'd the training. Fix in place:run_in_background: truefrom the Bash tool keeps the process attached to the harness lifetime, which is what the user wants. - KV cache SSD spill dirs filled the disk:
.cache_spill/under eachmodel/checkpoint/suite_*/grew unbounded (25 GiB alone forsuite_5d_1ep_ON), accumulating ~70 GiB across nine completed runs. Training crashed withOSError 28mid-epoch-1 of the 10th run. Fix landed in the suite script:rm -rf "$OUT_DIR/.cache_spill"right after training completes, before packaging. Long-term this belongs inside the trainer's cache destructor; not scoped to this experiment.
Tournament CLI enhancements committed for this suite
Changes to scripts/tournament.py made in support of the suite, kept for future experiments:
--sweep-source {remote,local}(default remote): Elo tier the OUR-side deck sweep pulls its top decks from.remotereadsdeck_elo_dailypopulated from Kaggle replays;localrequires local tournaments to have populated the local Elo tier first.--opponent PATHaccepts multiple paths (action=append) — pick a custom opponent subset instead of iterating everything underpublic_agents/.--skip-baselines: drop the built-inrandom+firstopponents. Used by the round-robin phase where the noise floor was already measured elsewhere.--report-json PATH: emit the structured per-opponent / per-deck rows + overall summary as JSON so downstream code stops parsing stdout.
Reproduction
uv run tcg-train --days 2026-07-28,2026-07-29,2026-07-30,2026-07-31,2026-08-01 \
--max-rows-per-day 30000 --epochs 10 --batch 1024 --tbptt-chunk 16 \
--val-batch-size 1024 --lr 2.46e-4 --warmup-steps 20 --aux-return-weight 1.0 \
--out model/checkpoint/suite_5d_10ep_OFF/5d_10ep_OFF.pkl \
--checkpoint-every-epochs 999
bash experiments/bc_curriculum_suite.sh # full suite, resume-aware
Cross-references
- pokemon_tcg_ai_battle — public project hub
- pokemon_tcg_agent_architecture — architecture and ceiling framing this experiment tests against
- pokemon_tcg_mlx_migration — MLX implementation contract this ablation runs on top of
- pokemon_tcg_training_overhaul_2026_07_29 — temporal baseline this suite extends
- pokemon_tcg_tbptt_training_contract — exact TBPTT accounting the trainer honors
- pokemon_tcg_evaluation_results — general eval framing and provenance boundaries
- pokemon_tcg_data_pipeline — the sqlite structure that supports stratified selection beyond agent-Elo
- pokemon_tcg_submissions_and_elo — where the
5d_10ep_OFFsubmission candidate goes - pokemon_tcg_current_state_reconciliation — current runtime and database boundary for interpreting this historical cohort