WikifitaGitHub live67e8de5
pesquisa · kaggle/pokemon_tcg_data_pipeline

Pokemon TCG AI Battle - Data Pipeline (historical NPY era)

Historical replay-to-dataset pipeline contract for BC NPY arrays, would-KO metadata, prospective sidecars, and resumable daily orchestration. Superseded for live-state description by pokemon_tcg_parquet_dataset.

Baixar raw

Pokemon TCG AI Battle - Data Pipeline (historical)

This page is a historical NPY/Prospective V2 record. The sidecar was removed at a942373, the live corpus moved to Parquet, and current packaging validates strict FP32. The commands and sidecar fields below remain useful for lineage and reproducibility of the old branch; they are not the current runtime contract.

Status

The .npy shard format described here has been superseded by day-partitioned Parquet files (pokemon_tcg_parquet_dataset) since 2026-08. The would-KO contract, aux target definitions, and idempotency principles are unchanged; the physical file format is not. Use pokemon_tcg_parquet_dataset for the current file layout, pokemon_tcg_sqlite_schema_current for the catalog that indexes them, and pokemon_tcg_training_pipeline for how the trainer consumes them.

Kept here for the record of the original design and for any tooling that still walks the NPY layout.

Evidence Boundary

This page describes observed code in the current pokemon-tcg worktree after commits 10c3750 and acbbea0, plus the smoke evidence captured in pokemon_tcg_training_overhaul_2026_07_29. It is not evidence that the full daily sequence or full prospective corpus has completed under the current settings.

The operational rule is narrow: Codex agents may run smoke-sized documentary checks, but they must not start long dataset rebuilds, full prospective builds, full training sequences, or tournaments while another real build is active.

Durable Sources

Raw replay ZIPs are the durable replay source. Encoded BC arrays and prospective sidecars are reproducible derived artifacts. The daily sequence retains replay ZIPs and checkpoints; encoded datasets may be deleted only after the checkpoint and sequence state have been written successfully.

raw replay ZIP
  -> streamed BC episode extraction
  -> compact NPY shard directory
  -> merged BC dataset directory
  -> optional historical prospective_v2 sidecar
  -> MLX training phase
  -> legacy PyTorch FP16 submission checkpoint

The pipeline treats missing source ZIPs as a cleanup blocker. This prevents a large encoded dataset from being deleted when its raw replay source is not available for deterministic regeneration.

BC Dataset Contract

The ZIP builder emits a directory-form BC dataset rather than a single opaque archive. The expected contract includes:

  • __labels__.npy
  • __would_ko_meta__.npy
  • action_mask.npy
  • episode_meta.npy
  • dataset_manifest.json
  • feature arrays produced by the structured state encoder
  • optional prospective_v2/ sidecar when prospective supervision is enabled

Each emitted row has episode-side provenance in episode_meta.npy, including episode id, side, step id, new_episode, player/opponent names, outcome, is_self, and deck hashes. A multi-select decision may emit multiple rows, but it remains one engine decision for recurrence and TBPTT semantics; see pokemon_tcg_temporal_learning.

Shard Idempotence

BC extraction is shard-first. Each shard writes its .npy arrays and writes .done last. Resume accepts only shards with .done; partial shards are removed and regenerated. A resume manifest records the build contract and rejects reuse when the fingerprint differs.

The builder keeps memory bounded by processing one shard batch at a time and merging one array key at a time through memory-mapped output. This is a practical requirement for large Kaggle replay corpora, not just an optimization.

After the base BC arrays are merged and validated, the builder writes .dataset_base_stage.json. That stage records the BC fingerprint, row count, shard count, would-KO summary, and output paths. If the process is interrupted before prospective construction finishes, the next run can validate the base stage and continue from it instead of rebuilding already-merged BC arrays.

would-KO Metadata

The would-KO pass is dataset generation, not a learned head by itself. When enabled, the builder records __would_ko_meta__.npy and a manifest section with eligible rows, computed rows, valid trials, failed trials, and zero semantics.

A zero would-KO target is not automatically a failure. The manifest separates valid zero outcomes from simulator failure so training can distinguish "this candidate does not KO" from "the simulator did not produce reliable metadata." Feature values are validated as finite and bounded before the dataset contract is accepted.

would-KO is consumed by BC/planner supervision and runtime features, while the runtime still recomputes legal visible-state would-KO features when the checkpoint declares that contract. It is not hidden-deck lookahead.

Historical Prospective Sidecar

When prospective_enabled=true, the BC builder requires directory-form output and creates or validates a prospective_v2/ sidecar. The sidecar contains compact arrays for nodes, branches, branch actions, groups, group offsets, and episode-side joins, plus prospective_manifest.json.

Existing sidecars are reused only when their adapter version, compact storage version, action-feature schema, replay sources, config, and BC dataset fingerprint match. The worker count is operational and is intentionally removed from the semantic config contract so serial and parallel builds can be byte equivalent.

The sidecar contract also records that hidden opponent deck access and synthetic fill are not used for prospective targets. This boundary is important because planner supervision is allowed to use counterfactual visible-state rollouts, but not illegal hidden-state information.

Daily Sequence

configs/train_sequence.json currently defines a two-date daily sequence:

  • 2026-07-27
  • 2026-07-28

Each phase builds or validates the encoded dataset, trains for the configured epochs, writes a rolling checkpoint, atomically copies the completed rolling checkpoint to the canonical checkpoint, records state, and then optionally deletes the reproducible encoded dataset. The sequence refuses a nominally fresh start when an output checkpoint already exists without matching sequence state.

The configured sequence is real infrastructure, but it is not an agent-safe default. Agents should use configs/smoke.json and small documentary checks unless Alefita explicitly authorizes a full run.

Entry Points

The project exposes these relevant command entry points:

  • tcg-build-bc - build BC NPY data from replay ZIPs.
  • tcg-build-daily - build a single daily BC corpus.
  • tcg-build-prospective - build a prospective sidecar from a BC dataset.
  • tcg-train - train the MLX BC/planner stack.
  • tcg-train-sequence - run the daily build-train-clean sequence.
  • tcg-build - historical packaging path from the PyTorch FP16 inference checkpoint; current packaging is reconciled in pokemon_tcg_torch_inference.

These commands are part of the operational contract; they are not permission to start long work in a Codex documentation task.

Migration to Parquet (2026-08-03)

The NPY shard format described on this page was retired on 2026-08-03 by commit a942373 ("sidecar removed, aux heads + meta features + parquet pipeline"). Mapping from old to new:

NPY-era constructParquet-era equivalent
Per-episode shard directoryOne row group in the daily Parquet file (rows within a row group are still episode-ordered when possible)
episode_meta.npy sidecarFirst-class Parquet columns: episode_id, side, step_id, decision_id, substep, new_episode, terminal
aux_*.npy sidecarsFirst-class Parquet columns: aux_ko, aux_prize_delta, aux_terminal, aux_return, aux_valid
prospective_*.npy sidecarsNot carried over — the sidecar planner was removed (see pokemon_tcg_prospective_v2)
Memory-mapped slab (slab_rows knob)Row group as I/O unit + hierarchical KV cache retention (see pokemon_tcg_kv_cache_hierarchical)
Per-episode .npy.manifest.jsonPer-day data/bc_data/<date>.manifest.json sidecar; SQLite datasets table indexes by day
tcg-build-prospective entry pointRemoved with the sidecar; no replacement

What survived unchanged: the would-KO contract, the aux-target semantics (columns just moved format), the episode-boundary val split logic, is_self derivation from bc_self_aliases, the manifest-driven idempotency principle.

For the live-state description of the current format use pokemon_tcg_parquet_dataset; for the incident/commit story use pokemon_tcg_repository_timeline.

Provenance / anchor commits

  • 277a5b5 (2026-07-25) — feat(D): option compaction, episode metadata, episode-level val split — laid the sequential-metadata contract that Parquet later absorbed.
  • edd9bed / fbb73eb (2026-07-26) — clean config flow for bc_would_ko via configure() function; then pass it from build_bc_from_zips to build_bc_dataset — established the would-KO contract wiring at build time.
  • 1424e33 (2026-07-26) — connect all config fields, fix wiring bugs from audit — reconciled builder flags with the central config.
  • a942373 (2026-08-03) — sidecar removed, aux heads + meta features + parquet pipeline — the retirement commit for this page's format.

Full timeline in pokemon_tcg_repository_timeline.

Related Pages