---
type: reference
title: "Pokemon TCG - TBPTT Training Contract"
description: "Exact recurrent training contract for decision chunks, row budget, gradient accumulation, scheduler steps, checkpoints, validation, progress accounting and the current FP32 runtime boundary."
tags: [pokemon-tcg, tbptt, recurrence, training, scheduler, checkpoint, progress]
timestamp: "2026-07-29T20:26:00-03:00"
---

# Pokemon TCG - TBPTT Training Contract

## Evidence Boundary

This page describes the current MLX trainer code in `scripts/bc/bc_train_mlx.py`
and the progress-bar correction work captured in source task
`019faa94-08c1-7d80-af02-873a75020b03`. It is a training-accounting contract,
not a claim about final ladder strength. The current source snapshot and its
FP32 correction are reconciled in [[pokemon_tcg_current_state_reconciliation]].

## Temporal Unit

TBPTT operates on engine decisions, not arbitrary emitted rows. A multi-select
decision may produce multiple autoregressive training rows, but those rows share
one decision boundary and must read the same incoming recurrent memory.

Rows are grouped by `(episode_id, side)` using the corresponding Parquet columns (`episode_id`, `side`, `step_id`; see [[pokemon_tcg_parquet_dataset]]) — the historical `episode_meta.npy` sidecar is gone, the metadata is now first-class inside the Parquet row. Rows are then split by contiguous `step_id`. A repeated non-contiguous `step_id` inside one episode-side lane is rejected.

## Chunk And Row Budget

`tbptt_chunk` is a decision horizon. `batch_size` is the physical row budget.
The trainer builds chunks that satisfy both:

```text
decisions_per_chunk <= tbptt_chunk
rows_per_physical_microbatch <= batch_size
```

Decisions are indivisible. If one decision alone exceeds the physical row
budget, the trainer raises an error and requires a larger `batch_size`.

The resulting `_tbptt_plan` is the real training work plan. Independent
episode-side chunks can be packed into the same temporal microbatch only while
the total row count remains within the physical row budget.

## Memory Semantics

Each episode-side lane has its own recurrent memory. At epoch start, lane
memories reset. Within an epoch, a chunk receives the previous chunk memory for
the same lane; the memory passed across chunk boundaries is detached with
`stop_gradient`.

This implements truncated backpropagation through time: recurrence carries
state forward, but gradient history stops at the configured chunk boundary.

## Optimizer Steps

`microbatches_per_epoch` is the exact length of `_tbptt_plan` when TBPTT is
enabled. Without TBPTT it is the exact number of standard batches yielded across
slab boundaries.

Optimizer steps are counted from microbatches:

```text
optimizer_steps_per_epoch = ceil(microbatches_per_epoch / accum_steps)
run_optimizer_steps = local_epochs * optimizer_steps_per_epoch
```

The final partial accumulation window is applied before validation. If the
number of processed microbatches or optimizer steps differs from the planned
count, the trainer raises a runtime error instead of silently producing a
misleading epoch.

The current trainer sets model parameters and forward activations to strict
FP32 and validates model leaves before training. Gradient leaves and optimizer
moments also use FP32. Historical FP16 checkpoints and cohort results remain
documented in [[pokemon_tcg_torch_inference]] and
[[pokemon_tcg_bc_curriculum_ablation]]; they are not the current dtype
contract.

## Scheduler Phase

The scheduler counts optimizer updates, not forward passes and not emitted rows.
When scheduler state is reset, `scheduler_total_steps=0` means "use this
invocation's real optimizer-step horizon." When scheduler state is resumed, the
checkpoint horizon is authoritative and cannot be changed without resetting the
scheduler phase.

Warmup is clamped to the scheduler horizon. A resumed phase is rejected if the
requested run would exceed the remaining scheduler steps.

## Checkpoints

The trainer writes three checkpoint classes:

- the `--out` path for the current best validation checkpoint;
- a rolling latest checkpoint for interruption-safe resume;
- numbered epoch snapshots at `checkpoint_every_epochs`, with the final epoch
  always retained.

The train config is a transient session sheet. Runtime architecture,
optimizer/scheduler contracts, progress counters, provenance, would-KO and
prospective settings travel inside the checkpoint payload rather than relying
on a later `train_config.json` file.

## Progress Accounting

The Rich progress task is phase-aware. It uses the real work plan for train,
then resets for validation, metric aggregation, prospective context
reconstruction, prospective training, checkpointing, and completion. A progress
bar reaching `100%` means the displayed phase is complete, not that unrelated
future phases are complete.

The source progress-bar debugging is a cautionary boundary: do not explain a
visual progress defect by changing TBPTT, batch size, scheduler semantics or
training loops without first reading the actual progress/task state. The final
contract is that display accounting follows the real iterator and asserts
mismatch, while training semantics remain separate from UI rendering.

## Validation

TBPTT validation uses a temporal validation plan and verifies that every
validation row is covered exactly once after row-order reconstruction. This
keeps validation memory-aware instead of measuring a stateless model.

## Related Pages

- [[pokemon_tcg_training_pipeline]] — the live end-to-end pipeline this contract is embedded in
- [[pokemon_tcg_temporal_learning]] — sequence preservation and minimal recurrence design
- [[pokemon_tcg_training_overhaul_2026_07_29]] — recurrent MLX trainer landing record
- [[pokemon_tcg_kv_cache_hierarchical]] — the row-group cache the TBPTT temporal batcher touches
- [[pokemon_tcg_train_config_reference]] — the flags this contract exposes (`--tbptt-chunk`, `--batch`, `--accum-steps`, ...)
- [[pokemon_tcg_bc_curriculum_ablation]] — evidence of this contract running clean across 10 configs
- [[pokemon_tcg_data_pipeline]]
- [[pokemon_tcg_prospective_v2]]
- [[pokemon_tcg_mlx_migration]]