Pokemon TCG - Evaluation Results

August 14 tournament ledger update

The current model/results.db snapshot contains 128 tournament summaries and 1,621 matchup summaries. The comparable FP32 Stage 1-4 cohort is tournament IDs 119-122: 14.29%, 15.24%, 13.81% and 17.14% overall respectively over 420 games each. Within Stage 4, the Yan / Deck #633 vehicle recorded 39/140 (27.9%) and the starter/default Deck #251 recorded 18/140 (12.9%). These are deck-conditioned local observations, not ladder ratings and not a proof of causal deck superiority.

The current ledger preserves tournament summaries, but only 1,760 local match rows are present in the normalized matches table. The snapshot therefore does not establish whether underlying replay rows were reset, partially rebuilt or selectively retained between runs. Keep this uncertainty attached to historical tournament comparisons. Full cohort tables are in pokemon_tcg_aug14_ablations_tournaments.

Evidence Boundary

Evaluation records are source-bound. Local tournament numbers, Kaggle ladder ratings, transcript analysis, and dashboard-derived Elo are not interchangeable. Every result should be read with its model, deck, opponent set, date, and submission lineage.

Historical Local Tournaments

model/eval_results.txt records local tournaments from July 21-23, 2026 in the inspected worktree. These are historical local observations, not current Prospective V2 claims.

Examples include:

July 21, 2026 13:26: 195-25, 88.6% overall across 220 games.
July 22, 2026 15:46: 198-42, 82.5% overall across 240 games.
July 23, 2026 13:18: 208-32, 86.7% overall across 240 games.

Those numbers are useful regression context, but they depend on the bundled opponent set, deck state, checkpoint lineage, and local harness behavior.

One-Epoch Prospective Result

The source task transcript records a July 29, 2026 analysis of a one-epoch checkpoint trained on the 2026-07-28 BC corpus with true TBPTT64, scratch32, would-KO, and prospective support. The local tournament completed 320/320 games without apparent integration failure.

The aggregate including smoke was 33/320, approximately 10.3%. That total is misleading because 18/20 wins came from the isolated smoke opponent:

Group	Wins	Games	Win rate
Random + first	6	40	15.0%
Public agents	8	180	4.44%
Historical submissions	1	80	1.25%
Smoke	18	20	90.0%
Total without smoke	15	300	5.0%

The historical operational interpretation was: MLX-to-PyTorch FP16 conversion, runtime loading, would-KO/prospective checkpoint metadata, and local tournament execution worked end to end; the one-epoch policy itself was still undertrained. The current converter is strict FP32, as recorded in pokemon_tcg_current_state_reconciliation.

Reported Ladder Context

The project memory preserves a user-reported earlier Kaggle submission around the 900-930 Elo/rating range. That is a provenance note, not a reproduced measurement from the current worktree. It should not be compared directly with the one-epoch local tournament.

Kaggle ladder results need their own submission id, deck, timestamp, visible rating, and rules/population context. A rating on one day is population-relative and can drift as teams, decks, and submissions change.

Comparison Rules

Do not compare:

local smoke win rate against public-agent tournament win rate as if they were the same population;
historical local PyTorch/BC checkpoints against Prospective V2 without deck, dataset, and opponent controls;
user-reported ladder ratings against local Elo;
validation accuracy against tournament win rate without noting objective, dataset, and architecture differences.

Use matchup matrices and release records when deciding whether a model/deck pair is a candidate for submission.

BC Curriculum Ablation (2026-08-06 – 2026-08-07)

Ten checkpoints trained across (1d, 3d, 5d) × (1ep, 10ep) × (top-Elo OFF, ON), each followed by a sweep-ON tournament against baselines + starters + strong public agents (~540 games per run), and a final no-sweep intra-suite round-robin (45 pairs × 30 games). The historical best vs publics was 5d_10ep_OFF at 21.0% overall; it was the submission candidate for that dated suite, not a current acceptance claim. Best vs peers: 3d_1ep_ON and 5d_1ep_ON tied at 53.2%. Anti-correlation between the two rankings: models that specialize enough to beat public agents lose to peers trained on the same distribution.

Within this dated suite, the absence of a broad encoding, conversion, cache or collapse failure supported a data- or regime-limited interpretation rather than a broad architecture failure. That interpretation is cohort-scoped: the later Stage 3/4 loss-scale and validation incident prevents it from serving as a universal current architecture claim. Full methodology, tables and the reconciliation are in pokemon_tcg_bc_curriculum_ablation and pokemon_tcg_stage3_training_failure_postmortem.