---
type: analysis
title: "Pokemon TCG - Evaluation Results"
description: "Observed local and reported ladder results with provenance, limitations, and comparison boundaries."
tags: [pokemon-tcg, evaluation, tournament, ladder, elo, provenance]
timestamp: "2026-08-15T17:24:00-03:00"
---

# Pokemon TCG - Evaluation Results

## August 14 tournament ledger update

The current `model/results.db` snapshot contains 128 tournament summaries and
1,621 matchup summaries. The comparable FP32 Stage 1-4 cohort is tournament
IDs 119-122: 14.29%, 15.24%, 13.81% and 17.14% overall respectively over 420
games each. Within Stage 4, the Yan / Deck #633 vehicle recorded 39/140
(27.9%) and the starter/default Deck #251 recorded 18/140 (12.9%). These are
deck-conditioned local observations, not ladder ratings and not a proof of
causal deck superiority.

The current ledger preserves tournament summaries, but only 1,760 local match
rows are present in the normalized `matches` table. The snapshot therefore
does not establish whether underlying replay rows were reset, partially
rebuilt or selectively retained between runs. Keep this uncertainty attached
to historical tournament comparisons. Full cohort tables are in
[[pokemon_tcg_aug14_ablations_tournaments]].

## Evidence Boundary

Evaluation records are source-bound. Local tournament numbers, Kaggle ladder
ratings, transcript analysis, and dashboard-derived Elo are not interchangeable.
Every result should be read with its model, deck, opponent set, date, and
submission lineage.

## Historical Local Tournaments

`model/eval_results.txt` records local tournaments from July 21-23, 2026 in the
inspected worktree. These are historical local observations, not current
Prospective V2 claims.

Examples include:

- July 21, 2026 13:26: `195-25`, `88.6%` overall across 220 games.
- July 22, 2026 15:46: `198-42`, `82.5%` overall across 240 games.
- July 23, 2026 13:18: `208-32`, `86.7%` overall across 240 games.

Those numbers are useful regression context, but they depend on the bundled
opponent set, deck state, checkpoint lineage, and local harness behavior.

## One-Epoch Prospective Result

The source task transcript records a July 29, 2026 analysis of a one-epoch
checkpoint trained on the 2026-07-28 BC corpus with true TBPTT64, scratch32,
would-KO, and prospective support. The local tournament completed `320/320`
games without apparent integration failure.

The aggregate including smoke was `33/320`, approximately `10.3%`. That total
is misleading because `18/20` wins came from the isolated smoke opponent:

| Group | Wins | Games | Win rate |
|---|---:|---:|---:|
| Random + first | 6 | 40 | 15.0% |
| Public agents | 8 | 180 | 4.44% |
| Historical submissions | 1 | 80 | 1.25% |
| Smoke | 18 | 20 | 90.0% |
| Total without smoke | 15 | 300 | 5.0% |

The historical operational interpretation was: MLX-to-PyTorch FP16 conversion,
runtime loading, would-KO/prospective checkpoint metadata, and local tournament
execution worked end to end; the one-epoch policy itself was still
undertrained. The current converter is strict FP32, as recorded in
[[pokemon_tcg_current_state_reconciliation]].

## Reported Ladder Context

The project memory preserves a user-reported earlier Kaggle submission around
the `900-930` Elo/rating range. That is a provenance note, not a reproduced
measurement from the current worktree. It should not be compared directly with
the one-epoch local tournament.

Kaggle ladder results need their own submission id, deck, timestamp, visible
rating, and rules/population context. A rating on one day is population-relative
and can drift as teams, decks, and submissions change.

## Comparison Rules

Do not compare:

- local smoke win rate against public-agent tournament win rate as if they were
  the same population;
- historical local PyTorch/BC checkpoints against Prospective V2 without deck,
  dataset, and opponent controls;
- user-reported ladder ratings against local Elo;
- validation accuracy against tournament win rate without noting objective,
  dataset, and architecture differences.

Use matchup matrices and release records when deciding whether a model/deck
pair is a candidate for submission.

## BC Curriculum Ablation (2026-08-06 – 2026-08-07)

Ten checkpoints trained across `(1d, 3d, 5d) × (1ep, 10ep) × (top-Elo OFF, ON)`, each followed by a sweep-ON tournament against baselines + starters + strong public agents (~540 games per run), and a final no-sweep intra-suite round-robin (45 pairs × 30 games). The historical best `vs publics` was `5d_10ep_OFF` at 21.0% overall; it was the submission candidate for that dated suite, not a current acceptance claim. Best `vs peers`: `3d_1ep_ON` and `5d_1ep_ON` tied at 53.2%. Anti-correlation between the two rankings: models that specialize enough to beat public agents lose to peers trained on the same distribution.

Within this dated suite, the absence of a broad encoding, conversion, cache or
collapse failure supported a data- or regime-limited interpretation rather
than a broad architecture failure. That interpretation is cohort-scoped: the
later Stage 3/4 loss-scale and validation incident prevents it from serving as
a universal current architecture claim. Full methodology, tables and the
reconciliation are in [[pokemon_tcg_bc_curriculum_ablation]] and
[[pokemon_tcg_stage3_training_failure_postmortem]].

## Related Pages

- [[pokemon_tcg_ladder_and_research]]
- [[pokemon_tcg_training_overhaul_2026_07_29]]
- [[pokemon_tcg_bc_curriculum_ablation]]
- [[pokemon_tcg_prospective_v2]]
- [[pokemon_tcg_submissions_and_elo]]
- [[pokemon_tcg_deck_strategy]]