---
type: reference
title: "Pokémon TCG AI Battle — Parquet Dataset Format"
description: "Day-partitioned Parquet dataset: schema (90 columns, fixed_size_list vectors), row groups as the physical I/O unit, streaming column projection, aux targets, per-row metadata, manifest sidecar and SQLite catalog integration."
tags: [pokemon-tcg, parquet, dataset, pyarrow, streaming, row-group, aux-targets, manifest]
timestamp: "2026-08-07T11:00:00-03:00"
---

# Pokémon TCG AI Battle — Parquet Dataset Format

## Boundary

This page documents the current on-disk dataset format used by [[pokemon_tcg_training_pipeline]]. The historical `.npy` shard format lives in [[pokemon_tcg_data_pipeline]] and is superseded by everything on this page.

Terminology on this page (`row`, `episode`, `row_group`, `aux_valid`, `manifest`, `source='remote'`, etc.) is defined in [[pokemon_tcg_glossary]].

## Layout

One Parquet file per calendar day, plus a JSON manifest sidecar per file:

```text
data/bc_data/
  2026-07-28.parquet
  2026-07-28.manifest.json
  2026-07-29.parquet
  2026-07-29.manifest.json
  ...
  2026-08-01.parquet
  2026-08-01.manifest.json
```

Each Parquet file is discovered and validated through the SQLite catalog table `datasets` (see [[pokemon_tcg_sqlite_schema_current]]). A row in `datasets` carries the file path, `schema_version`, row count, SHA256, and `aux_targets` flag.

## Row semantics

One Parquet row is one classified decision by one side of one match. Every row carries:

- the full encoded observation (state entities, options, masks),
- the action label `y` and its legality mask `action_mask`,
- four aux targets (`aux_ko`, `aux_prize_delta`, `aux_terminal`, `aux_return`) with `aux_valid` gating them,
- metadata sufficient to reconstruct temporal order and provenance (`episode_id`, `side`, `step_id`, `decision_id`, `substep`, `new_episode`, `terminal`, `is_self`, `player_name`, `opponent_name`, `day_id`, deck hashes).

Currently the largest days sit at ~813k rows. Data volume in the SQLite catalog:

| day | rows |
|---|---|
| 2026-08-01 | 813,124 |
| 2026-07-31 | 810,573 |
| 2026-07-30 | 812,765 |
| 2026-07-29 | 811,539 |
| 2026-07-28 | 813,079 |

## Column schema

90 columns total. Types (pyarrow):

**Scalar state**

| column | type | notes |
|---|---|---|
| `select_type` | int32 | wire select-type ordinal |
| `select_context` | int32 | wire select-context ordinal |
| `day_index_norm` | float | normalized competition-day scalar |
| `opponent_agent_bucket` | int32 | 0–10 meta bucket (Elo decile + UNKNOWN) |
| `opponent_deck_bucket` | int32 | 0–10 meta bucket |

**State entity vectors** (`fixed_size_list<float>[K]` or `<int32>[K]`)

| column | length | purpose |
|---|---|---|
| `cls_scalars` | 19 | 13 board/turn + 5 select-dynamics + 1 our-turn offensive buff |
| `effect_id` / `effect_mask` | 2 / 2 | active effect tokens |
| `self_deck_id`/`mask`/`flag` | 60 / 60 / 60 | our decklist tokens (drawable flag) |
| `opp_deck_id`/`mask`/`flag` | 60 / 60 / 60 | opponent's decklist |
| `self_prize_id`/`mask` | 6 / 6 | our prize row |
| `opp_prize_id`/`mask` | 6 / 6 | opponent prize row |
| `self_hand_id`/`mask` | 30 / 30 | our hand (public+hidden) |
| `opp_hand_id`/`mask`/`flag` | 30 / 30 / 30 | opponent hand (only visible cards concrete) |
| `self_discard_id`/`mask` | 60 / 60 | public discard |
| `opp_discard_id`/`mask` | 60 / 60 | public discard |
| `stadium_id`/`mask` | 2 / 2 | active stadium slot per side |
| `self_unit_top_id`/`preevo_id`/`tool_id`/`energy_id`/`attr`/`mask` | 9 / 18 / 36 / 36 / 216 / 9 | 1 active + 8 bench units (attr has 24 floats/unit) |
| `opp_unit_*` | (same shapes) | opponent field |

**Option stream** (up to `MAX_OPTIONS = 192` legal actions per decision)

| column | length | notes |
|---|---|---|
| `opt_attr` | 6912 | 192 × 36 float feats per option (would_ko trio, structural, already-picked flag) |
| `opt_verb` | 192 | option-type ordinal (0..16) |
| `opt_src_pos` / `opt_tgt_pos` | 192 / 192 | global state-token positions the option references |
| `opt_src_card` / `opt_tgt_card` | 192 / 192 | resolved card IDs |
| `opt_attack_id` | 192 | attack identity for attack options; 0 = no-attack padding |
| `action_mask` | 193 | legal-action mask, index 192 = `SUBMIT_ACTION` |
| `opt_group` | 193 | equivalence-class ids used by the `--dedup` collapse |

**Meta buckets** (Elo decile + UNKNOWN, 0..10; applied per token slot)

`self_deck_meta_bucket`, `self_hand_meta_bucket`, `self_discard_meta_bucket`, `self_unit_top_meta_bucket`, `self_unit_preevo_meta_bucket`, `self_unit_tool_meta_bucket`, `self_unit_energy_meta_bucket`, and their `opp_*` twins — each `fixed_size_list<int32>[K]` matching the corresponding `_id` column.

**Label + supervision**

| column | type | notes |
|---|---|---|
| `y` | int32 | true action index (0..192) |
| `is_attack` | int8 | val-metric flag |
| `outcome` | int8 | game outcome sign in {-1,0,1} |
| `reward` | float | normalized episode reward |
| `is_self` | int8 | row belongs to our-side agent (per bc_self_aliases) |

**Aux targets**

| column | type | purpose |
|---|---|---|
| `aux_ko` | int8 | did the decision produce an immediate KO? BCE target |
| `aux_prize_delta` | float | prize delta at the decision boundary. MSE target |
| `aux_terminal` | int8 | is this the terminal row? BCE target |
| `aux_return` | float | normalized return-to-go. MSE target |
| `aux_valid` | int8 | 0/1 mask: whether targets exist and should contribute to loss |

**Sequential metadata** (needed by TBPTT and val split)

| column | type | notes |
|---|---|---|
| `episode_id` | int64 | one match |
| `side` | int8 | 0 or 1 |
| `step_id` | int32 | raw engine step |
| `decision_id` | int32 | groups substeps of the same multi-select |
| `substep` | int8 | order inside a multi-select decision |
| `new_episode` | int8 | reset boundary flag for scratch memory |
| `terminal` | int8 | final row flag |
| `day_id` | int32 | FK into `sqlite:days` |
| `player_name` | string | agent name from Kaggle replay |
| `opponent_name` | string | opponent name |
| `player_deck_hash` | string | deck fingerprint |
| `opponent_deck_hash` | string | deck fingerprint |

## Row groups

Row groups are the physical read unit inside a Parquet file. They matter operationally because the trainer's cache retention, SSD spill, and streaming projection are all row-group-scoped, not row-scoped.

For a representative day (2026-08-01):

```text
n_row_groups   23
n_rows_total   813,124
rows / row_group   ~35,000 (34,613,862 bytes for row_group 0)
```

Every day currently sits around ~800k rows and ~23 row groups. The row-group boundary is chosen by the writer at build time and is invariant afterwards — the loader cannot re-chunk without rewriting the file.

Because the KV cache retains **row groups** (not rows), the effective working-set size is (`bytes_per_row_group × resident_row_groups`). With `_HOT_ZONE_FRACTION=0.6` and 5 row groups resident (typical steady-state), the cache holds ~5 × 35k = ~175k rows warm at any time.

## Streaming reads

The trainer opens each Parquet file with `pyarrow.dataset.dataset(paths, format="parquet")` and iterates via a Scanner that:

- projects only the columns the current training step needs (drops everything else),
- reads by row group in order (or in a scan-key sequence when `--top-elo` / `--max-rows-per-day` prune episodes),
- reads ahead internally (no separate manual prefetch thread — pyarrow's Scanner overlaps I/O and decode on its own).

Because reads are column-projected, dropping unused metadata columns from a run is essentially free. The exception is the TBPTT path, which projects everything the model needs plus `episode_id`, `side`, `step_id` for lane construction.

Row-group-level operations (retention, promotion, eviction, SSD spill) live in the KV cache — see [[pokemon_tcg_kv_cache_hierarchical]].

## Aux target contract

The trainer requires `aux_valid`, `aux_ko`, `aux_prize_delta`, `aux_terminal`, `aux_return` to be present. It refuses to start if any is missing. When `aux_valid[i]==0` for a row, the four losses on that row are ignored (masked); the row still contributes to the primary cross-entropy on `y`.

A dataset's `aux_targets` column in `sqlite:datasets` records `1` when the builder emitted aux columns, `0` otherwise. Days built without aux cannot be used by the current trainer.

## Manifest sidecar

`data/bc_data/<date>.manifest.json` records the provenance of the corresponding Parquet:

- source zip SHA256 (the Kaggle replay archive)
- encoder version and card-catalog SHA256
- aux-target contract version
- writer parameters (`bc_would_ko`, `bc_wk_nvar`, `bc_both_sides`, `bc_self_aliases`, `bc_flush`, `bc_ep_timeout`)
- row count
- build timestamp

The trainer does not read the manifest at run time; it is used by the dataset builder for idempotency (already-built days skip) and by `tcg-data` for the catalog view.

## SQLite catalog

Every Parquet file appears exactly once in the SQLite table `datasets`. The trainer resolves `--days 2026-07-28,2026-08-01` (or `--last-n-days N`, or `--all-days`) by selecting rows from `datasets` joined against `days`, then opening the resolved `path` values. If a Parquet exists on disk but is missing from `datasets`, the trainer will not see it.

Population and full schema in [[pokemon_tcg_sqlite_schema_current]].

## Historical: NPY shards

Before 2026-08 the pipeline used `.npy` shards per episode, described in [[pokemon_tcg_data_pipeline]]. That page is preserved for context; do not build against it. The current builder writes Parquet directly.

## Provenance / anchor commits

- `277a5b5` (2026-07-25) — feat(D): option compaction, episode metadata, episode-level val split — landed the sequential metadata contract (originally as `episode_meta.npy`).
- `a942373` (2026-08-03) — sidecar removed, aux heads + meta features + parquet pipeline — the pivot: dataset writer moved from `.npy` shards to day-partitioned Parquet; aux target columns and meta buckets integrated as first-class row columns.
- `504118d` (2026-08-03) — smoke pipeline validated: strict semantics, competition_day, streaming TBPTT — first end-to-end validation of the Parquet path.
- `e772fe0` (2026-08-06) — train loader: KV-style parquet cache — row groups become the physical retention unit downstream, cementing the row-group boundary as a load-time contract.

Full timeline in [[pokemon_tcg_repository_timeline]].

## Cross-references

- [[pokemon_tcg_glossary]] — terminology
- [[pokemon_tcg_kv_cache_hierarchical]] — how row groups are retained across microbatches
- [[pokemon_tcg_training_pipeline]] — what the trainer does with these rows
- [[pokemon_tcg_sqlite_schema_current]] — the `datasets` and `days` tables the trainer consults
- [[pokemon_tcg_data_pipeline]] — historical NPY shard pipeline
- [[pokemon_tcg_agent_architecture]] — the token layout the columns encode into
- [[pokemon_tcg_top_elo_curriculum_filter]] — how the SQLite catalog is queried at load time to filter episodes