Pokémon TCG AI Battle — Current Agent Architecture

August 14 boundary update

This page still describes the live entity/action Transformer. The August 14 handoff adds a documented future architecture, not a silent replacement of this one. A direct source-tree check found no tracked rl/ropend, rl/moe, rl/policy_moe_torch.py, rl/policy_moe_mlx.py or rl/deck/vehicle_draft.py implementation. The corresponding RoPE-ND, four-expert MoE, vehicle-draft and Apex documents remain preserved as planned or experimental work in pokemon_tcg_aug14_architecture_and_handoff_audit.

The current source-backed model still uses the existing scratch and auxiliary-head path. The current FP32 inference contract and the August 14 database, ablation and tournament evidence are cataloged in pokemon_tcg_aug14_data_etl_database_audit and pokemon_tcg_aug14_ablations_tournaments.

Architecture identity

Entity/action Transformer trained by behavioral cloning over a structured, externally reconstructed belief state. Not a causal language decoder. It exposes a recurrent memory interface through scratch registers and TBPTT, but it has no dedicated recurrent cell. The exact register count belongs to the run configuration: the default is 16 and the session configuration used 32.

Current build (config default → session baseline in configs/train_config.json):

d_model              128
attention heads      4
Transformer layers   3        (session runs 4)
FFN width            512      (4 × d_model)
scratch registers    16       (session runs 32)
static card features enabled
split policy/value   enabled  (dedicated VALUE_TOK and SUBMIT_TOK heads)
structured head      disabled for current BC baseline
options              up to MAX_OPTIONS=192 plus SUBMIT_ACTION=192
total trainable      ~1.30M params  (~845k Muon-routed + ~456k AdamW-routed)

Positional information does not use RoPE, sinusoidal or RoPE-ND. Positions are carried by zone-typed token embeddings (type_emb, sel_type_emb, sel_ctx_emb) plus explicit gather from opt_src_pos / opt_tgt_pos — each option token points at the exact state token of the card/unit it references, so the option's src/tgt embedding IS that state token, not a rebuilt copy. RoPE-ND is deferred backlog, not shipped (pokemon_tcg_ladder_and_research).

The token sequence has approximately 337 positions before options:

[CLS] [SELECT_TYPE] [SELECT_CONTEXT]
[SELF_DECK ×60] [OPP_DECK ×60]
[SELF_PRIZE ×6] [OPP_PRIZE ×6]
[SELF_HAND ×30] [OPP_HAND ×30]
[SELF_DISCARD ×60] [OPP_DISCARD ×60]
[STADIUM ×2] [EFFECT ×2]
[SELF_UNITS ×(1+8)] [OPP_UNITS ×(1+8)]
[SCRATCH ×N_SCRATCH]
[OPTIONS ×(up to 192)]

Padding capacity is part of the storage format; not every position holds a real entity. padding_idx=0 semantics apply — card and attack ID zero embed to a zero vector, not a learned "absence" embedding. The option surface holds up to 192 tokens plus one reserved SUBMIT_ACTION index at position 192.

What the model actually receives

The encoder combines several kinds of information:

Input	Role
Card and attack IDs	Categorical identity embeddings
Static card features	Deterministic domain attributes projected into model space
Unit attributes	Dynamic HP, energy, effects, status and other board state
Zone/type embeddings	Ownership and semantic role of deck, hand, discard, active, bench, stadium and effects
Selection context	Current action phase and selection type
Source/target positions	Explicit references from an option to state tokens
Action mask	Exact legality boundary supplied by the engine/encoder
Tracker flags	Information about visibility, drawability and certainty

The option token is a structured candidate, not a fixed semantic class tied to an array index. It combines source and target representations, structural attributes, verb identity and attack identity. The output head scores those candidates and applies the legal-action mask.

Mathematically, the current policy is close to:

[ \pi_\theta(a_t\mid\hat b_t), \qquad \hat b_t=\mathcal T(o_t,\text{logs}t,H{t-1}), ]

where GameTracker and AbilityTracker construct the working belief state before the Transformer is called.

External state is a strength and a boundary

Keeping the exact rules transition outside the neural network is appropriate. The engine should remain the sound authority for legal actions and state transitions:

[ s_{t+1}=P(s_t,a_t), \qquad m(s_t,a_t)\in{0,1}. ]

The current limitation is that the external system also supplies most of the agent's temporal belief compression. If two histories produce the same tracker output, the current policy cannot distinguish them:

[ \mathcal T(H_1)=\mathcal T(H_2) \Rightarrow \pi(\cdot\mid H_1)=\pi(\cdot\mid H_2). ]

This is not a claim that the tracker is wrong. It is a statement about what information is available to the neural policy. The current model is therefore best described as an autoregressive action scorer over each externally reconstructed belief-state substep, with bounded learned memory rather than a full internal game-state model.

Scratch/register tokens

Scratch registers are learned workspace vectors inserted between state and option tokens. In the current implementation they are initialized from learned parameters at match start, then the final substep's output becomes the next decision's memory_in:

[ J_0^{\mathrm{in}}=J_{\mathrm{init}}, \qquad J_{t+1}^{\mathrm{in}}=J_t^{\mathrm{out}}. ]

They support intrapass integration and inter-decision memory. agent/main.py stores the memory per tracked side, and the inference model carries it through each autoregressive selection. This memory interface does not prove that the learned state is strategically sufficient.

Action semantics

The dataset expands multi-select actions into sequential substeps:

picked=[]       -> option_1
picked=[1]      -> option_2
picked=[1,2]    -> SUBMIT

The earlier live path used one forward pass and topk(count). The current agent/main.py path recomputes the encoded observation and logits after each selected option, masks already-picked options, and terminates on legal SUBMIT_ACTION. The old topk(count) behavior remains a historical defect recorded in the migration pages, not the current action contract.

The corrected factorization is:

[ \pi(a_1\mid s), \quad \pi(a_2\mid s,a_1), \quad \ldots, \quad \pi(\mathrm{SUBMIT}\mid s,a_{<k}). ]

Each substep must update picked, the option mask and the option tokens, then run the next forward pass.

Evidence and current ceiling

The repository's local evaluation artifacts show strong performance against the included public opponents and a smaller set of difficult matchups. The recorded tournament results are useful regression evidence, but they are not a causal attribution of which architectural component produced them. In particular, local win rate is not interchangeable with the competition's proprietary ladder rating.

The current performance can be explained without assuming a hidden breakthrough:

1. the rules engine removes illegal actions;
2. the encoder provides typed entities and action references;
3. static card attributes reduce the burden of learning domain facts;
4. replay labels provide a strong behavioral prior;
5. interchangeable actions can be grouped or deduplicated;
6. the model only needs to rank a reduced candidate surface.

The architecture's likely ceiling is temporal rather than purely computational. It preserves a bounded learned memory, but the rules engine and tracker still perform most belief reconstruction, and the train/inference feature contract must remain aligned. The current code passes the observation logs into the tracker:

[ \hat b_{\mathrm{train}}=\mathcal T(o_t,\mathrm{logs}t,H{t-1}), \qquad \hat b_{\mathrm{live}}=\mathcal T(o_t,\mathrm{logs}t,H{t-1}). ]

The immediate repair is to pass the complete observation through the same tracker and encoder path used during dataset construction.

Embeddings by zone

The token vocabulary is grouped into families, each with its own learned embedding table. All embed to d_model unless otherwise noted:

family	table	notes
card identity	card_emb: Embedding(vocab+1, d_model)	padding_idx=0
card static features	static_proj: Linear(24, d_model)	fed by EN_Card_Data.csv buffer
zone type	type_emb: Embedding(N_TTYPES, d_model)	which zone this token belongs to (SELF_DECK, OPP_HAND, ...)
select type	sel_type_emb: Embedding(N_SELECT_TYPES=16, d_model)	wire select.type ordinal
select ctx	sel_ctx_emb: Embedding(N_SELECT_CTX=64, d_model)	wire select.context ordinal
unit attributes	unit_attr_proj: Linear(24, d_model)	dynamic HP, energy, status, next-turn dmg-reduce buff
visibility flags	drawable_emb / opp_drawable_emb / hand_certain_emb	scalar mx.array, added when the row's visibility flag is set
option verb	opt_verb_emb: Embedding(17, d_model)	option-type ordinal (0..16 incl SPECIAL_CONDITION)
option src/tgt	opt_src_proj / opt_tgt_proj: Linear(d_model, d_model)	gathered from resolved state token, not rebuilt
option structural	opt_attr_proj: Linear(36, d_model)	36 float feats/option (would_ko trio, structural, already-picked flag)
attack identity	attack_emb: Embedding(2048, d_model)	attack-id (padding_idx=0)
CLS scalars	scalar_proj: Linear(19, d_model)	13 board/turn + 5 select-dynamics + 1 our-turn offensive buff
meta buckets	meta_bucket_emb / agent_bucket_emb / deck_bucket_emb: Embedding(11, d_model)	0..9 Elo decile + UNKNOWN
day scalar	day_proj: Linear(1, d_model)	normalized competition-day integer
meta context base	meta_ctx_base: mx.array(d_model)	learned bias mixed into the meta context
scratch	scratch: mx.array(N_SCRATCH, d_model) + learned_init: mx.array(N_SCRATCH, d_model)	see below

N_META_BUCKETS = 11 — Elo deciles 0–9 (0 = strongest) plus slot 10 (UNKNOWN_BUCKET) for domain-legitimate "not observed on this day yet" cases (newly released cards, first-game opponents, etc.). Cards / agents / decks all share this schema.

Scratch registers

A small block of learned workspace tokens inserted between the state stream and the option stream:

J_0_in     = learned_init             (seed at match/decision start)
J_{t+1}    = scratch_slice(model_output_t)
memory_in  is the previous chunk's scratch output
memory_out is the current chunk's scratch output

--scratch-registers N controls the count. Range 0–64. Config default: 16. Current session config: 32.

When TBPTT is on (see pokemon_tcg_tbptt_training_contract), memory is carried per (episode_id, side) lane across chunks, with stop_gradient only at the chunk boundary. Reset on new_episode, isolated per side. Not shared across matches or processes.

Inspired by "ViTs Need Registers": the scratch tokens are non-semantic workspace positions the model can use as scratchpad without stealing capacity from state or option tokens.

Split heads (dedicated VALUE and SUBMIT)

split_heads=true reserves two token slots at the head of the sequence with dedicated output heads:

• VALUE_TOK — feeds the value head (scalar or categorical). Short residual keeps prize-count signal clean.
• SUBMIT_TOK — feeds the submit_head (single logit that is spliced into the option scoring at index SUBMIT_ACTION).

Off (split_heads=false): a single CLS token feeds both value and action heads (2 × d_model input to the value head instead of d_model).

Auxiliary heads

Four Linear(d_model, 1) heads applied to the CLS output (or VALUE_TOK when split_heads=true):

head	target column	loss	default weight
ko_head_aux	aux_ko (int8)	BCE	0.5
prize_head_aux	aux_prize_delta (float)	MSE	0.5
terminal_head_aux	aux_terminal (int8)	BCE	0.5
return_head_aux	aux_return (float)	MSE	1.0

All four masked by aux_valid per row. Fixed loss weights via --aux-*-weight flags — automatic uncertainty weighting (Kendall & Gal) is deferred backlog.

Attention mechanics

• Additive mask — the padded-key mask is added to attention scores. The shared finite negative sentinel is -65504.0; 0 marks valid keys. The current trainer/inference model is strict FP32 even though this sentinel is also finite in FP16.
• MHA bias — attention projections carry a bias term (bias=True), matching the PyTorch reference. The MLX side instantiates this explicitly to avoid inheriting a different default.
• FP32 current path — the trainer sets model parameters and forward activations to FP32, and the PyTorch converter validates FP32 floating tensors. The historical FP16 path remains documented in pokemon_tcg_torch_inference.

Static card features

EN_Card_Data.csv is loaded once as an np.ndarray and stored as a buffer (_card_feat_np), not a trainable parameter. A learned Linear(24, d_model) projects it into model space when --static true. The buffer is checkpointed by SHA256 in the payload so a resume with a changed CSV raises.

Structured verb head (disabled in baseline)

--structured true adds per-OptionType query and bias embeddings (type_query: Embedding(17, d_model), type_bias: Embedding(17, 1)) that gate the shared opt_head scoring. A structured_weight_decay (default 0.1, higher than the shared AdamW decay of 0.01) collapses rare verbs toward the shared fallback. Off by default in the current BC baseline.

Categorical value head (opt-in)

When instantiated with value_categorical=True, the value head is Linear(d_model, value_atoms) producing atom logits over support [-value_vmax, +value_vmax]. The runtime output is the expected scalar E[value] = softmax(atom_logits) · atom_support — atom logits are never returned as-is. Off in the current baseline (scalar value).

What the model actually receives

The encoder combines several kinds of information:

Input	Role
Card and attack IDs	Categorical identity embeddings
Static card features	Deterministic domain attributes projected into model space
Unit attributes	Dynamic HP, energy, effects, status and other board state
Zone/type embeddings	Ownership and semantic role of deck, hand, discard, active, bench, stadium and effects
Selection context	Current action phase and selection type
Source/target positions	Explicit references from an option to state tokens (gathered, not rebuilt)
Action mask	Exact legality boundary supplied by the engine/encoder
Tracker flags	Visibility, drawability and certainty per token
Meta buckets	Elo-decile bucket per card / agent / deck
Day scalar	Normalized competition-day integer
Scratch registers	Persistent inter-decision memory when TBPTT is on

The option token is a structured candidate, not a fixed semantic class tied to an array index. It combines source and target representations, structural attributes, verb identity and attack identity. The output head scores those candidates and applies the legal-action mask.

Mathematically, the current policy is close to:

[ \pi_\theta(a_t\mid\hat b_t), \qquad \hat b_t=\mathcal T(o_t,\text{logs}t,H{t-1}), ]

where GameTracker and AbilityTracker construct the working belief state before the Transformer is called. H_{t-1} is the previous decision's scratch memory (or learned_init at match start).

External state is a strength and a boundary

Keeping the exact rules transition outside the neural network is appropriate. The engine should remain the sound authority for legal actions and state transitions:

[ s_{t+1}=P(s_t,a_t), ]

Architecture boundaries

Current and authorized

• Entity/action Transformer at 128/4/3-4/512.
• Exact additive action masks and structured option tokens.
• External rules engine and trackers (GameTracker + AbilityTracker).
• Behavioral cloning with four aux heads.
• 16–32 scratch tokens made persistent across decisions when TBPTT is on.
• Sequential metadata and truncated backpropagation through time.
• MLX training on M3 Pro, strict-FP32 PyTorch arena inference (see pokemon_tcg_current_state_reconciliation and pokemon_tcg_torch_inference).

Deferred

Mamba/Mamba-2, Hope/Nested Learning, multidimensional RoPE / RoPE-ND, Energy-Based Transformer, TRM, J-Lens, strategic MoE, PPO/GRPO/GSPO, PSRO, world models in submitted inference, symbolic regression, custom Metal and distributed Mac training remain research backlog items in pokemon_tcg_ladder_and_research. They must not be smuggled into the current port under the label of optimization.

Related pages

• pokemon_tcg_ai_battle — project hub and evidence boundary
• pokemon_tcg_training_pipeline — how this architecture is trained today
• pokemon_tcg_torch_inference — historical FP16 delivery path and current FP32 correction
• pokemon_tcg_train_config_reference — every flag that steers architecture choices
• pokemon_tcg_mlx_migration — migration record (historical)
• pokemon_tcg_temporal_learning — sequence preservation and minimal recurrence
• pokemon_tcg_tbptt_training_contract — TBPTT accounting on top of these scratch registers
• pokemon_tcg_bc_curriculum_ablation — the ablation that validated this architecture at scale
• pokemon_tcg_ladder_and_research — deferred research (RoPE-ND, Mamba, etc.)