Pokémon TCG AI Battle — Parquet Dataset Format

Boundary

This page documents the current on-disk dataset format used by pokemon_tcg_training_pipeline. The historical .npy shard format lives in pokemon_tcg_data_pipeline and is superseded by everything on this page.

Terminology on this page (row, episode, row_group, aux_valid, manifest, source='remote', etc.) is defined in pokemon_tcg_glossary.

Layout

One Parquet file per calendar day, plus a JSON manifest sidecar per file:

data/bc_data/
  2026-07-28.parquet
  2026-07-28.manifest.json
  2026-07-29.parquet
  2026-07-29.manifest.json
  ...
  2026-08-01.parquet
  2026-08-01.manifest.json

Each Parquet file is discovered and validated through the SQLite catalog table datasets (see pokemon_tcg_sqlite_schema_current). A row in datasets carries the file path, schema_version, row count, SHA256, and aux_targets flag.

Row semantics

One Parquet row is one classified decision by one side of one match. Every row carries:

the full encoded observation (state entities, options, masks),
the action label y and its legality mask action_mask,
four aux targets (aux_ko, aux_prize_delta, aux_terminal, aux_return) with aux_valid gating them,
metadata sufficient to reconstruct temporal order and provenance (episode_id, side, step_id, decision_id, substep, new_episode, terminal, is_self, player_name, opponent_name, day_id, deck hashes).

Currently the largest days sit at ~813k rows. Data volume in the SQLite catalog:

day	rows
2026-08-01	813,124
2026-07-31	810,573
2026-07-30	812,765
2026-07-29	811,539
2026-07-28	813,079

Column schema

90 columns total. Types (pyarrow):

Scalar state

column	type	notes
`select_type`	int32	wire select-type ordinal
`select_context`	int32	wire select-context ordinal
`day_index_norm`	float	normalized competition-day scalar
`opponent_agent_bucket`	int32	0–10 meta bucket (Elo decile + UNKNOWN)
`opponent_deck_bucket`	int32	0–10 meta bucket

State entity vectors (fixed_size_list<float>[K] or <int32>[K])

column	length	purpose
`cls_scalars`	19	13 board/turn + 5 select-dynamics + 1 our-turn offensive buff
`effect_id` / `effect_mask`	2 / 2	active effect tokens
`self_deck_id`/`mask`/`flag`	60 / 60 / 60	our decklist tokens (drawable flag)
`opp_deck_id`/`mask`/`flag`	60 / 60 / 60	opponent's decklist
`self_prize_id`/`mask`	6 / 6	our prize row
`opp_prize_id`/`mask`	6 / 6	opponent prize row
`self_hand_id`/`mask`	30 / 30	our hand (public+hidden)
`opp_hand_id`/`mask`/`flag`	30 / 30 / 30	opponent hand (only visible cards concrete)
`self_discard_id`/`mask`	60 / 60	public discard
`opp_discard_id`/`mask`	60 / 60	public discard
`stadium_id`/`mask`	2 / 2	active stadium slot per side
`self_unit_top_id`/`preevo_id`/`tool_id`/`energy_id`/`attr`/`mask`	9 / 18 / 36 / 36 / 216 / 9	1 active + 8 bench units (attr has 24 floats/unit)
`opp_unit_*`	(same shapes)	opponent field

Option stream (up to MAX_OPTIONS = 192 legal actions per decision)

column	length	notes
`opt_attr`	6912	192 × 36 float feats per option (would_ko trio, structural, already-picked flag)
`opt_verb`	192	option-type ordinal (0..16)
`opt_src_pos` / `opt_tgt_pos`	192 / 192	global state-token positions the option references
`opt_src_card` / `opt_tgt_card`	192 / 192	resolved card IDs
`opt_attack_id`	192	attack identity for attack options; 0 = no-attack padding
`action_mask`	193	legal-action mask, index 192 = `SUBMIT_ACTION`
`opt_group`	193	equivalence-class ids used by the `--dedup` collapse

Meta buckets (Elo decile + UNKNOWN, 0..10; applied per token slot)

self_deck_meta_bucket, self_hand_meta_bucket, self_discard_meta_bucket, self_unit_top_meta_bucket, self_unit_preevo_meta_bucket, self_unit_tool_meta_bucket, self_unit_energy_meta_bucket, and their opp_* twins — each fixed_size_list<int32>[K] matching the corresponding _id column.

Label + supervision

column	type	notes
`y`	int32	true action index (0..192)
`is_attack`	int8	val-metric flag
`outcome`	int8	game outcome sign in {-1,0,1}
`reward`	float	normalized episode reward
`is_self`	int8	row belongs to our-side agent (per bc_self_aliases)

Aux targets

column	type	purpose
`aux_ko`	int8	did the decision produce an immediate KO? BCE target
`aux_prize_delta`	float	prize delta at the decision boundary. MSE target
`aux_terminal`	int8	is this the terminal row? BCE target
`aux_return`	float	normalized return-to-go. MSE target
`aux_valid`	int8	0/1 mask: whether targets exist and should contribute to loss

Sequential metadata (needed by TBPTT and val split)

column	type	notes
`episode_id`	int64	one match
`side`	int8	0 or 1
`step_id`	int32	raw engine step
`decision_id`	int32	groups substeps of the same multi-select
`substep`	int8	order inside a multi-select decision
`new_episode`	int8	reset boundary flag for scratch memory
`terminal`	int8	final row flag
`day_id`	int32	FK into `sqlite:days`
`player_name`	string	agent name from Kaggle replay
`opponent_name`	string	opponent name
`player_deck_hash`	string	deck fingerprint
`opponent_deck_hash`	string	deck fingerprint

Row groups

Row groups are the physical read unit inside a Parquet file. They matter operationally because the trainer's cache retention, SSD spill, and streaming projection are all row-group-scoped, not row-scoped.

For a representative day (2026-08-01):

n_row_groups   23
n_rows_total   813,124
rows / row_group   ~35,000 (34,613,862 bytes for row_group 0)

Every day currently sits around ~800k rows and ~23 row groups. The row-group boundary is chosen by the writer at build time and is invariant afterwards — the loader cannot re-chunk without rewriting the file.

Because the KV cache retains row groups (not rows), the effective working-set size is (bytes_per_row_group × resident_row_groups). With _HOT_ZONE_FRACTION=0.6 and 5 row groups resident (typical steady-state), the cache holds ~5 × 35k = ~175k rows warm at any time.

Streaming reads

The trainer opens each Parquet file with pyarrow.dataset.dataset(paths, format="parquet") and iterates via a Scanner that:

projects only the columns the current training step needs (drops everything else),
reads by row group in order (or in a scan-key sequence when --top-elo / --max-rows-per-day prune episodes),
reads ahead internally (no separate manual prefetch thread — pyarrow's Scanner overlaps I/O and decode on its own).

Because reads are column-projected, dropping unused metadata columns from a run is essentially free. The exception is the TBPTT path, which projects everything the model needs plus episode_id, side, step_id for lane construction.

Row-group-level operations (retention, promotion, eviction, SSD spill) live in the KV cache — see pokemon_tcg_kv_cache_hierarchical.

Aux target contract

The trainer requires aux_valid, aux_ko, aux_prize_delta, aux_terminal, aux_return to be present. It refuses to start if any is missing. When aux_valid[i]==0 for a row, the four losses on that row are ignored (masked); the row still contributes to the primary cross-entropy on y.

A dataset's aux_targets column in sqlite:datasets records 1 when the builder emitted aux columns, 0 otherwise. Days built without aux cannot be used by the current trainer.

Manifest sidecar

data/bc_data/<date>.manifest.json records the provenance of the corresponding Parquet:

source zip SHA256 (the Kaggle replay archive)
encoder version and card-catalog SHA256
aux-target contract version
writer parameters (bc_would_ko, bc_wk_nvar, bc_both_sides, bc_self_aliases, bc_flush, bc_ep_timeout)
row count
build timestamp

The trainer does not read the manifest at run time; it is used by the dataset builder for idempotency (already-built days skip) and by tcg-data for the catalog view.

SQLite catalog

Every Parquet file appears exactly once in the SQLite table datasets. The trainer resolves --days 2026-07-28,2026-08-01 (or --last-n-days N, or --all-days) by selecting rows from datasets joined against days, then opening the resolved path values. If a Parquet exists on disk but is missing from datasets, the trainer will not see it.

Population and full schema in pokemon_tcg_sqlite_schema_current.

Historical: NPY shards

Before 2026-08 the pipeline used .npy shards per episode, described in pokemon_tcg_data_pipeline. That page is preserved for context; do not build against it. The current builder writes Parquet directly.

Provenance / anchor commits

277a5b5 (2026-07-25) — feat(D): option compaction, episode metadata, episode-level val split — landed the sequential metadata contract (originally as episode_meta.npy).
a942373 (2026-08-03) — sidecar removed, aux heads + meta features + parquet pipeline — the pivot: dataset writer moved from .npy shards to day-partitioned Parquet; aux target columns and meta buckets integrated as first-class row columns.
504118d (2026-08-03) — smoke pipeline validated: strict semantics, competition_day, streaming TBPTT — first end-to-end validation of the Parquet path.
e772fe0 (2026-08-06) — train loader: KV-style parquet cache — row groups become the physical retention unit downstream, cementing the row-group boundary as a load-time contract.

Full timeline in pokemon_tcg_repository_timeline.

Cross-references

pokemon_tcg_glossary — terminology
pokemon_tcg_kv_cache_hierarchical — how row groups are retained across microbatches
pokemon_tcg_training_pipeline — what the trainer does with these rows
pokemon_tcg_sqlite_schema_current — the datasets and days tables the trainer consults
pokemon_tcg_data_pipeline — historical NPY shard pipeline
pokemon_tcg_agent_architecture — the token layout the columns encode into
pokemon_tcg_top_elo_curriculum_filter — how the SQLite catalog is queried at load time to filter episodes