WikifitaGitHub live67e8de5
pesquisa · kaggle/pokemon_tcg_tbptt_training_contract

Pokemon TCG - TBPTT Training Contract

Exact recurrent training contract for decision chunks, row budget, gradient accumulation, scheduler steps, checkpoints, validation, progress accounting and the current FP32 runtime boundary.

Baixar raw

Pokemon TCG - TBPTT Training Contract

Evidence Boundary

This page describes the current MLX trainer code in scripts/bc/bc_train_mlx.py and the progress-bar correction work captured in source task 019faa94-08c1-7d80-af02-873a75020b03. It is a training-accounting contract, not a claim about final ladder strength. The current source snapshot and its FP32 correction are reconciled in pokemon_tcg_current_state_reconciliation.

Temporal Unit

TBPTT operates on engine decisions, not arbitrary emitted rows. A multi-select decision may produce multiple autoregressive training rows, but those rows share one decision boundary and must read the same incoming recurrent memory.

Rows are grouped by (episode_id, side) using the corresponding Parquet columns (episode_id, side, step_id; see pokemon_tcg_parquet_dataset) — the historical episode_meta.npy sidecar is gone, the metadata is now first-class inside the Parquet row. Rows are then split by contiguous step_id. A repeated non-contiguous step_id inside one episode-side lane is rejected.

Chunk And Row Budget

tbptt_chunk is a decision horizon. batch_size is the physical row budget. The trainer builds chunks that satisfy both:

decisions_per_chunk <= tbptt_chunk
rows_per_physical_microbatch <= batch_size

Decisions are indivisible. If one decision alone exceeds the physical row budget, the trainer raises an error and requires a larger batch_size.

The resulting _tbptt_plan is the real training work plan. Independent episode-side chunks can be packed into the same temporal microbatch only while the total row count remains within the physical row budget.

Memory Semantics

Each episode-side lane has its own recurrent memory. At epoch start, lane memories reset. Within an epoch, a chunk receives the previous chunk memory for the same lane; the memory passed across chunk boundaries is detached with stop_gradient.

This implements truncated backpropagation through time: recurrence carries state forward, but gradient history stops at the configured chunk boundary.

Optimizer Steps

microbatches_per_epoch is the exact length of _tbptt_plan when TBPTT is enabled. Without TBPTT it is the exact number of standard batches yielded across slab boundaries.

Optimizer steps are counted from microbatches:

optimizer_steps_per_epoch = ceil(microbatches_per_epoch / accum_steps)
run_optimizer_steps = local_epochs * optimizer_steps_per_epoch

The final partial accumulation window is applied before validation. If the number of processed microbatches or optimizer steps differs from the planned count, the trainer raises a runtime error instead of silently producing a misleading epoch.

The current trainer sets model parameters and forward activations to strict FP32 and validates model leaves before training. Gradient leaves and optimizer moments also use FP32. Historical FP16 checkpoints and cohort results remain documented in pokemon_tcg_torch_inference and pokemon_tcg_bc_curriculum_ablation; they are not the current dtype contract.

Scheduler Phase

The scheduler counts optimizer updates, not forward passes and not emitted rows. When scheduler state is reset, scheduler_total_steps=0 means "use this invocation's real optimizer-step horizon." When scheduler state is resumed, the checkpoint horizon is authoritative and cannot be changed without resetting the scheduler phase.

Warmup is clamped to the scheduler horizon. A resumed phase is rejected if the requested run would exceed the remaining scheduler steps.

Checkpoints

The trainer writes three checkpoint classes:

  • the --out path for the current best validation checkpoint;
  • a rolling latest checkpoint for interruption-safe resume;
  • numbered epoch snapshots at checkpoint_every_epochs, with the final epoch always retained.

The train config is a transient session sheet. Runtime architecture, optimizer/scheduler contracts, progress counters, provenance, would-KO and prospective settings travel inside the checkpoint payload rather than relying on a later train_config.json file.

Progress Accounting

The Rich progress task is phase-aware. It uses the real work plan for train, then resets for validation, metric aggregation, prospective context reconstruction, prospective training, checkpointing, and completion. A progress bar reaching 100% means the displayed phase is complete, not that unrelated future phases are complete.

The source progress-bar debugging is a cautionary boundary: do not explain a visual progress defect by changing TBPTT, batch size, scheduler semantics or training loops without first reading the actual progress/task state. The final contract is that display accounting follows the real iterator and asserts mismatch, while training semantics remain separate from UI rendering.

Validation

TBPTT validation uses a temporal validation plan and verifies that every validation row is covered exactly once after row-order reconstruction. This keeps validation memory-aware instead of measuring a stateless model.

Related Pages