---
type: analysis
title: "Pokémon TCG AI Battle — BC Curriculum Ablation (2026-08-06/07)"
description: "Dated 10-config behavioral-cloning ablation over (days × epochs × top-Elo curriculum) with per-run tournaments and intra-suite round-robin; its architecture sanity result is cohort-scoped and does not supersede the later Stage 3/4 training-contract incident."
tags: [pokemon-tcg, mlx, behavioral-cloning, tbptt, curriculum, ablation, tournament, elo, architecture-validation]
timestamp: "2026-08-15T17:24:00-03:00"
---

# Pokémon TCG AI Battle — BC Curriculum Ablation (2026-08-06/07)

## Scope

Ablation study run over `develop` in `~/workdir/pokemon-tcg` on 2026-08-06 to 2026-08-07. Driver script: `experiments/bc_curriculum_suite.sh`. Full JSON reports at `experiments/bc_curriculum_suite/reports/`, aggregated markdown at `experiments/bc_curriculum_suite/results.md`. Ten checkpoints were trained on the same base pipeline described in [[pokemon_tcg_training_overhaul_2026_07_29]] and [[pokemon_tcg_tbptt_training_contract]] — MLX FP16 training, PyTorch FP16 arena inference, TBPTT always on. This is a dated empirical cohort; its FP16 runtime is preserved as provenance and is not the current source contract.

Runtime split for this historical phase was fixed as training in MLX,
inference in PyTorch FP16 and recurrent TBPTT. The current strict-FP32
correction is recorded in [[pokemon_tcg_current_state_reconciliation]] and
[[pokemon_tcg_training_pipeline]].

The phrase “data-limited, not architecture-limited” below is a conclusion
within this dated suite. It means that the suite did not isolate a broad
encoding, conversion, cache or model-collapse failure under its own logs. It
does not certify the later training contract: the Stage 3/4 reconstruction
found an auxiliary-loss scale mismatch and an in-pool validation boundary.
Read [[pokemon_tcg_stage3_training_failure_postmortem]] before extending this
sanity result to the current architecture.

## Experiment matrix

```text
(1d, 3d, 5d) × (1ep, 10ep) × (top-elo OFF, ON) = 10 runs
```

Fixed knobs across all 10: batch 1024, TBPTT chunk 16, val batch 1024, lr 2.46e-4, warmup 20 steps, aux_return_weight 1.0, `--max-rows-per-day 30000` (uniform cap per calendar day). `--top-elo 50` on the ON side filters episodes to those played by top-50 agents (source='remote' from Kaggle replays) on both sides.

The 30k/day cap was chosen for suite parity — not because we believe it is the right training-time data volume. Larger runs will drop it.

Per-run tournament — sweep ON, `--sweep-source remote` (3 decks: default + top 2 by remote deck Elo, deduplicated), 20 games/opponent/deck, 9 opponents = 2 baselines (`random`, `first`) + 4 starters (`lb510_mega_abomasnow_ex`, `lb526_iono`, `lb600_dragapult_ex`, `lb600_mega_lucario_ex`) + 3 strong public agents (`lb826_alakazam_seok`, `lb945_multiply_ivan`, `lb1009_mega_lucario_ex_islet`). ~540 games per run.

Intra-suite round-robin — no sweep, no baselines, 30 games per pair, all C(10,2) = 45 pairs. For each pair `(i, j)` the OUR-agent slot is loaded from `model/checkpoint/suite_${i}/${i}.pkl` (via `bc_best_mlx.pkl` swap → `tcg-build` → `model/bc_model/bc_best_torch_fp16.pt`), and the opponent slot is the pre-packaged `experiments/bc_curriculum_suite/models/${j}.tar.gz`.

## Per-run tournament — vs public agents

Aggregated across sweep decks. Numbers are win rates over `wins + losses` (draws excluded from the denominator).

| run | overall | random | first | lb510 | lb526 | lb600d | lb600m | lb826 | lb945 | lb1009 |
|---|---|---|---|---|---|---|---|---|---|---|
| 1d_10ep_OFF | 18.5% | 63.3% | 38.3% | 10.0% | 5.0% | 20.0% | 6.7% | 18.3% | 3.3% | 1.7% |
| 1d_10ep_ON | 18.1% | 70.0% | 28.3% | 5.0% | **0.0%** | 5.0% | 23.3% | 23.3% | 5.0% | 3.3% |
| 3d_1ep_OFF | 18.3% | 71.7% | 21.7% | 13.3% | 1.7% | 16.7% | 11.7% | 18.3% | 5.0% | 5.0% |
| 3d_1ep_ON | 20.4% | 76.7% | 43.3% | 11.7% | 5.0% | 11.7% | 15.0% | 16.7% | 3.3% | **0.0%** |
| 3d_10ep_OFF | 19.6% | 70.0% | 33.3% | 15.0% | 8.3% | 11.7% | 6.7% | 20.0% | 1.7% | 10.0% |
| 3d_10ep_ON | **16.9%** | 60.0% | 38.3% | 8.3% | **0.0%** | 15.0% | 6.7% | 20.0% | 3.3% | **0.0%** |
| 5d_1ep_OFF | 18.0% | 71.7% | 31.7% | 13.3% | 5.0% | 8.3% | 8.3% | 20.0% | 3.3% | **0.0%** |
| 5d_1ep_ON | 17.8% | 68.3% | 35.0% | 13.3% | 6.7% | 3.3% | 5.0% | 21.7% | 3.3% | 3.3% |
| **5d_10ep_OFF** | **21.0%** | 75.0% | 40.0% | 10.0% | 1.7% | 8.3% | 13.3% | **30.0%** | 3.3% | 6.8% |
| 5d_10ep_ON | 20.0% | 73.3% | 31.7% | **16.7%** | **10.0%** | 5.0% | 6.7% | **33.3%** | 1.7% | 1.7% |

## Intra-suite round-robin — WR of row-model against column-model

| our \ opp | 1d_10ep_OFF | 1d_10ep_ON | 3d_10ep_OFF | 3d_10ep_ON | 3d_1ep_OFF | 3d_1ep_ON | 5d_10ep_OFF | 5d_10ep_ON | 5d_1ep_OFF | 5d_1ep_ON | avg |
|---|---|---|---|---|---|---|---|---|---|---|---|
| 1d_10ep_OFF | — | 70.0 | 60.0 | 60.0 | 43.3 | 34.5 | 53.3 | 40.0 | 50.0 | 53.3 | **51.6** |
| 1d_10ep_ON | 30.0 | — | 56.7 | 50.0 | 43.3 | 40.0 | 66.7 | 53.3 | 53.3 | 48.3 | **49.1** |
| 3d_10ep_OFF | 40.0 | 43.3 | — | 50.0 | 53.3 | 50.0 | 53.3 | 63.3 | 53.3 | 50.0 | **50.7** |
| 3d_10ep_ON | 40.0 | 50.0 | 50.0 | — | 50.0 | 46.7 | 40.0 | 70.0 | 46.7 | 46.7 | **48.9** |
| 3d_1ep_OFF | 56.7 | 56.7 | 46.7 | 50.0 | — | 43.3 | 46.7 | 66.7 | 63.3 | 46.7 | **53.0** |
| 3d_1ep_ON | 65.5 | 60.0 | 50.0 | 53.3 | 56.7 | — | 66.7 | 43.3 | 36.7 | 46.7 | **53.2** |
| 5d_10ep_OFF | 46.7 | 33.3 | 46.7 | 60.0 | 53.3 | 33.3 | — | 73.3 | 40.0 | 43.3 | **47.8** |
| 5d_10ep_ON | 60.0 | 46.7 | 36.7 | 30.0 | 33.3 | 56.7 | 26.7 | — | 56.7 | 50.0 | **44.1** |
| 5d_1ep_OFF | 50.0 | 46.7 | 46.7 | 53.3 | 36.7 | 63.3 | 60.0 | 43.3 | — | 36.7 | **48.5** |
| 5d_1ep_ON | 46.7 | 51.7 | 50.0 | 53.3 | 53.3 | 53.3 | 56.7 | 50.0 | 63.3 | — | **53.2** |

## Findings

### 1. Coverage cost of the Elo filter is real and closes at scale

Every `ON` run except `5d_10ep_ON` scored **0.0% vs `lb526_iono`** — a low-Elo starter deck that top-50 filtering excludes. The model literally never trains on episodes that involve that archetype, so it does not know how to play against it. This is the assignment predicted by the coverage hypothesis.

`5d_10ep_ON` finally reaches **10.0% vs `lb526`** because 5 filtered days carry enough deck diversity for the archetype to reappear in the training set. Curriculum-by-agent-Elo does not fundamentally exclude decks; it excludes them until the filtered day count is high enough that they reappear as a side effect.

### 2. Top-elo filter never wins in matched pairs

For every (days, epochs) quadrant on `vs publics`, the ON side ties or loses to OFF:

| pair | OFF | ON |
|---|---|---|
| 1d_10ep | 18.5% | 18.1% |
| 3d_1ep | 18.3% | 20.4% |
| 3d_10ep | 19.6% | 16.9% |
| 5d_1ep | 18.0% | 17.8% |
| 5d_10ep | 21.0% | 20.0% |

The only ON improvement (`3d_1ep`) is inside the noise floor. On the largest scale (`5d_10ep`), OFF wins by 1.0 point. Curriculum by agent-Elo is the wrong cut while the model is data-limited. See discussion in [[pokemon_tcg_data_pipeline]] for the alternate cuts the sqlite structure already supports.

### 3. Overfit on the filtered set

`3d_10ep_ON` = **16.9% overall**, worst of the ten. Compare to `3d_1ep_ON` = 20.4%: 10 epochs on the same filtered set regresses by 3.5 points. The filtered set at 3 days is small enough that repeated passes overspecialize. This is a training-regime bug, not a filter bug — it repeats on `random` too (60% vs 76.7%).

### 4. Anti-correlation between vs-publics and vs-peers

| run | vs publics | vs peers | rank swap |
|---|---|---|---|
| 5d_10ep_OFF | 21.0% (1st) | 47.8% (8th) | ↓ 7 |
| 5d_10ep_ON | 20.0% (2nd) | 44.1% (10th) | ↓ 8 |
| 3d_1ep_ON | 20.4% | 53.2% (1st tie) | stable |
| 5d_1ep_ON | 17.8% | 53.2% (1st tie) | ↑ |
| 3d_1ep_OFF | 18.3% | 53.0% | ↑ |

Models that peel off the peer distribution to specialize on public agents (`5d_10ep_*`) become predictable to other peers trained on similar data. Models that stay generic (`1ep`) beat peers but do not beat publics as reliably.

Consequence for submission choice: **vs publics is the proxy for real ladder Elo**, since the ladder does not contain BC clones of this suite. The intra-suite ranking is a diagnostic, not a submission signal. See [[pokemon_tcg_evaluation_results]] for the general framing on how eval numbers compare and do not compare.

### 5. Architecture sanity within this historical suite

The suite did not expose the following broad failure modes in its logged runs:

| failure mode | evidence it did not happen |
|---|---|
| broken encoding | training loss decreased monotonically on every run (5d_10ep_OFF: 3.49 → 1.12) |
| MLX → PyTorch conversion drift | tournaments emitted zero `ERR` opponents in ~5,400 games; agent always chose legal actions |
| loss misalignment | val_acc improved with data and epochs (5d_10ep_OFF: 0.53 → 0.61) |
| FP16 overflow/underflow | no NaN observed in any run; grad_norm bounded |
| KV cache corruption | val_acc improved monotonically; no erratic regression |
| optimizer contract broken | aux heads (`ko_bce`, `prize_mse`, `terminal_bce`, `return_mse`) all decreased on every run |
| collapsed model | intra-suite peer WR sits in 44–53% across all 10 — a normal Elo dispersion; no model in 15–25% |

The peer round-robin distribution being centered at ~50% with 44–53% spread is
a useful architecture sanity signal for this cohort: 10 models drawn from the
same distribution split games evenly. It is not an independent proof that the
shared trunk, objective and validation contract were healthy in every later
stage.

Within this suite, the overall ceiling of ~20% vs public agents was
interpreted as **data- or regime-limited rather than a broad architecture
failure**. Increasing epochs on the suite's 30k/day slice hurt (see finding
3). The later Stage 3/4 evidence prevents that observation from becoming a
current universal claim about the Transformer or the training pipeline.

## Operational notes from the run

Two failures during the suite that are worth recording so they do not repeat:

- **Session teardown killed a background training run**: the first launch attempted `setsid`, which is unavailable on macOS. Later `/compact` reparented the shell process group and SIGHUP'd the training. Fix in place: `run_in_background: true` from the Bash tool keeps the process attached to the harness lifetime, which is what the user wants.
- **KV cache SSD spill dirs filled the disk**: `.cache_spill/` under each `model/checkpoint/suite_*/` grew unbounded (25 GiB alone for `suite_5d_1ep_ON`), accumulating ~70 GiB across nine completed runs. Training crashed with `OSError 28` mid-epoch-1 of the 10th run. Fix landed in the suite script: `rm -rf "$OUT_DIR/.cache_spill"` right after training completes, before packaging. Long-term this belongs inside the trainer's cache destructor; not scoped to this experiment.

## Tournament CLI enhancements committed for this suite

Changes to `scripts/tournament.py` made in support of the suite, kept for future experiments:

- `--sweep-source {remote,local}` (default remote): Elo tier the OUR-side deck sweep pulls its top decks from. `remote` reads `deck_elo_daily` populated from Kaggle replays; `local` requires local tournaments to have populated the local Elo tier first.
- `--opponent PATH` accepts multiple paths (`action=append`) — pick a custom opponent subset instead of iterating everything under `public_agents/`.
- `--skip-baselines`: drop the built-in `random` + `first` opponents. Used by the round-robin phase where the noise floor was already measured elsewhere.
- `--report-json PATH`: emit the structured per-opponent / per-deck rows + overall summary as JSON so downstream code stops parsing stdout.

## Reproduction

```bash
uv run tcg-train --days 2026-07-28,2026-07-29,2026-07-30,2026-07-31,2026-08-01 \
  --max-rows-per-day 30000 --epochs 10 --batch 1024 --tbptt-chunk 16 \
  --val-batch-size 1024 --lr 2.46e-4 --warmup-steps 20 --aux-return-weight 1.0 \
  --out model/checkpoint/suite_5d_10ep_OFF/5d_10ep_OFF.pkl \
  --checkpoint-every-epochs 999

bash experiments/bc_curriculum_suite.sh   # full suite, resume-aware
```

## Cross-references

- [[pokemon_tcg_ai_battle]] — public project hub
- [[pokemon_tcg_agent_architecture]] — architecture and ceiling framing this experiment tests against
- [[pokemon_tcg_mlx_migration]] — MLX implementation contract this ablation runs on top of
- [[pokemon_tcg_training_overhaul_2026_07_29]] — temporal baseline this suite extends
- [[pokemon_tcg_tbptt_training_contract]] — exact TBPTT accounting the trainer honors
- [[pokemon_tcg_evaluation_results]] — general eval framing and provenance boundaries
- [[pokemon_tcg_data_pipeline]] — the sqlite structure that supports stratified selection beyond agent-Elo
- [[pokemon_tcg_submissions_and_elo]] — where the `5d_10ep_OFF` submission candidate goes
- [[pokemon_tcg_current_state_reconciliation]] — current runtime and database boundary for interpreting this historical cohort