---
type: reference
title: "Pokemon TCG AI Battle - Data Pipeline (historical NPY era)"
description: "Historical replay-to-dataset pipeline contract for BC NPY arrays, would-KO metadata, prospective sidecars, and resumable daily orchestration. Superseded for live-state description by pokemon_tcg_parquet_dataset."
tags: [pokemon-tcg, dataset, pipeline, replay, would-ko, prospective-v2, idempotence, historical]
timestamp: "2026-08-07T11:00:00-03:00"
---

# Pokemon TCG AI Battle - Data Pipeline (historical)

This page is a historical NPY/Prospective V2 record. The sidecar was removed
at `a942373`, the live corpus moved to Parquet, and current packaging validates
strict FP32. The commands and sidecar fields below remain useful for lineage
and reproducibility of the old branch; they are not the current runtime
contract.

## Status

The `.npy` shard format described here has been superseded by day-partitioned Parquet files ([[pokemon_tcg_parquet_dataset]]) since 2026-08. The would-KO contract, aux target definitions, and idempotency principles are unchanged; the physical file format is not. Use [[pokemon_tcg_parquet_dataset]] for the current file layout, [[pokemon_tcg_sqlite_schema_current]] for the catalog that indexes them, and [[pokemon_tcg_training_pipeline]] for how the trainer consumes them.

Kept here for the record of the original design and for any tooling that still walks the NPY layout.

## Evidence Boundary

This page describes observed code in the current `pokemon-tcg` worktree after
commits `10c3750` and `acbbea0`, plus the smoke evidence captured in
[[pokemon_tcg_training_overhaul_2026_07_29]]. It is not evidence that the full
daily sequence or full prospective corpus has completed under the current
settings.

The operational rule is narrow: Codex agents may run smoke-sized documentary
checks, but they must not start long dataset rebuilds, full prospective builds,
full training sequences, or tournaments while another real build is active.

## Durable Sources

Raw replay ZIPs are the durable replay source. Encoded BC arrays and
prospective sidecars are reproducible derived artifacts. The daily sequence
retains replay ZIPs and checkpoints; encoded datasets may be deleted only after
the checkpoint and sequence state have been written successfully.

```text
raw replay ZIP
  -> streamed BC episode extraction
  -> compact NPY shard directory
  -> merged BC dataset directory
  -> optional historical prospective_v2 sidecar
  -> MLX training phase
  -> legacy PyTorch FP16 submission checkpoint
```

The pipeline treats missing source ZIPs as a cleanup blocker. This prevents a
large encoded dataset from being deleted when its raw replay source is not
available for deterministic regeneration.

## BC Dataset Contract

The ZIP builder emits a directory-form BC dataset rather than a single opaque
archive. The expected contract includes:

- `__labels__.npy`
- `__would_ko_meta__.npy`
- `action_mask.npy`
- `episode_meta.npy`
- `dataset_manifest.json`
- feature arrays produced by the structured state encoder
- optional `prospective_v2/` sidecar when prospective supervision is enabled

Each emitted row has episode-side provenance in `episode_meta.npy`, including
episode id, side, step id, `new_episode`, player/opponent names, outcome,
`is_self`, and deck hashes. A multi-select decision may emit multiple rows, but
it remains one engine decision for recurrence and TBPTT semantics; see
[[pokemon_tcg_temporal_learning]].

## Shard Idempotence

BC extraction is shard-first. Each shard writes its `.npy` arrays and writes
`.done` last. Resume accepts only shards with `.done`; partial shards are
removed and regenerated. A resume manifest records the build contract and
rejects reuse when the fingerprint differs.

The builder keeps memory bounded by processing one shard batch at a time and
merging one array key at a time through memory-mapped output. This is a
practical requirement for large Kaggle replay corpora, not just an optimization.

After the base BC arrays are merged and validated, the builder writes
`.dataset_base_stage.json`. That stage records the BC fingerprint, row count,
shard count, would-KO summary, and output paths. If the process is interrupted
before prospective construction finishes, the next run can validate the base
stage and continue from it instead of rebuilding already-merged BC arrays.

## would-KO Metadata

The would-KO pass is dataset generation, not a learned head by itself. When
enabled, the builder records `__would_ko_meta__.npy` and a manifest section with
eligible rows, computed rows, valid trials, failed trials, and zero semantics.

A zero would-KO target is not automatically a failure. The manifest separates
valid zero outcomes from simulator failure so training can distinguish "this
candidate does not KO" from "the simulator did not produce reliable metadata."
Feature values are validated as finite and bounded before the dataset contract
is accepted.

would-KO is consumed by BC/planner supervision and runtime features, while the
runtime still recomputes legal visible-state would-KO features when the
checkpoint declares that contract. It is not hidden-deck lookahead.

## Historical Prospective Sidecar

When `prospective_enabled=true`, the BC builder requires directory-form output
and creates or validates a `prospective_v2/` sidecar. The sidecar contains
compact arrays for nodes, branches, branch actions, groups, group offsets, and
episode-side joins, plus `prospective_manifest.json`.

Existing sidecars are reused only when their adapter version, compact storage
version, action-feature schema, replay sources, config, and BC dataset
fingerprint match. The worker count is operational and is intentionally removed
from the semantic config contract so serial and parallel builds can be byte
equivalent.

The sidecar contract also records that hidden opponent deck access and
synthetic fill are not used for prospective targets. This boundary is important
because planner supervision is allowed to use counterfactual visible-state
rollouts, but not illegal hidden-state information.

## Daily Sequence

`configs/train_sequence.json` currently defines a two-date daily sequence:

- `2026-07-27`
- `2026-07-28`

Each phase builds or validates the encoded dataset, trains for the configured
epochs, writes a rolling checkpoint, atomically copies the completed rolling
checkpoint to the canonical checkpoint, records state, and then optionally
deletes the reproducible encoded dataset. The sequence refuses a nominally
fresh start when an output checkpoint already exists without matching sequence
state.

The configured sequence is real infrastructure, but it is not an agent-safe
default. Agents should use `configs/smoke.json` and small documentary checks
unless Alefita explicitly authorizes a full run.

## Entry Points

The project exposes these relevant command entry points:

- `tcg-build-bc` - build BC NPY data from replay ZIPs.
- `tcg-build-daily` - build a single daily BC corpus.
- `tcg-build-prospective` - build a prospective sidecar from a BC dataset.
- `tcg-train` - train the MLX BC/planner stack.
- `tcg-train-sequence` - run the daily build-train-clean sequence.
- `tcg-build` - historical packaging path from the PyTorch FP16 inference checkpoint; current packaging is reconciled in [[pokemon_tcg_torch_inference]].

These commands are part of the operational contract; they are not permission to
start long work in a Codex documentation task.

## Migration to Parquet (2026-08-03)

The NPY shard format described on this page was retired on 2026-08-03 by commit `a942373` ("sidecar removed, aux heads + meta features + parquet pipeline"). Mapping from old to new:

| NPY-era construct | Parquet-era equivalent |
|---|---|
| Per-episode shard directory | One row group in the daily Parquet file (rows within a row group are still episode-ordered when possible) |
| `episode_meta.npy` sidecar | First-class Parquet columns: `episode_id`, `side`, `step_id`, `decision_id`, `substep`, `new_episode`, `terminal` |
| `aux_*.npy` sidecars | First-class Parquet columns: `aux_ko`, `aux_prize_delta`, `aux_terminal`, `aux_return`, `aux_valid` |
| `prospective_*.npy` sidecars | Not carried over — the sidecar planner was removed (see [[pokemon_tcg_prospective_v2]]) |
| Memory-mapped slab (`slab_rows` knob) | Row group as I/O unit + hierarchical KV cache retention (see [[pokemon_tcg_kv_cache_hierarchical]]) |
| Per-episode `.npy.manifest.json` | Per-day `data/bc_data/<date>.manifest.json` sidecar; SQLite `datasets` table indexes by day |
| `tcg-build-prospective` entry point | Removed with the sidecar; no replacement |

What survived unchanged: the would-KO contract, the aux-target semantics (columns just moved format), the episode-boundary val split logic, `is_self` derivation from `bc_self_aliases`, the manifest-driven idempotency principle.

For the live-state description of the current format use [[pokemon_tcg_parquet_dataset]]; for the incident/commit story use [[pokemon_tcg_repository_timeline]].

## Provenance / anchor commits

- `277a5b5` (2026-07-25) — feat(D): option compaction, episode metadata, episode-level val split — laid the sequential-metadata contract that Parquet later absorbed.
- `edd9bed` / `fbb73eb` (2026-07-26) — clean config flow for `bc_would_ko` via `configure()` function; then pass it from `build_bc_from_zips` to `build_bc_dataset` — established the would-KO contract wiring at build time.
- `1424e33` (2026-07-26) — connect all config fields, fix wiring bugs from audit — reconciled builder flags with the central config.
- `a942373` (2026-08-03) — sidecar removed, aux heads + meta features + parquet pipeline — the retirement commit for this page's format.

Full timeline in [[pokemon_tcg_repository_timeline]].

## Related Pages

- [[pokemon_tcg_parquet_dataset]] — live current format that replaced this one
- [[pokemon_tcg_kv_cache_hierarchical]] — the row-group retention that replaced the mmapped-slab abstraction
- [[pokemon_tcg_repository_timeline]] — commit-level story of the format migration
- [[pokemon_tcg_training_overhaul_2026_07_29]]
- [[pokemon_tcg_would_ko_prospective_search]]
- [[pokemon_tcg_prospective_v2]]
- [[pokemon_tcg_action_coverage]]
- [[pokemon_tcg_daily_corpus_training]]
- [[pokemon_tcg_temporal_learning]]