Arena Elo Rating System

Overview

The Red Team Arena implements a standard chess-style Elo rating system to track the relative capability of both attacker and defender agents. Unlike traditional Elo (which rates players in a symmetric game), the arena maintains two independent rating pools -- one for attackers, one for defenders -- updated simultaneously after each round.

The system lives in arena/elo.py and is used by the ArenaController to update ratings after every attack/defense/judge exchange.

The Elo Formula

The implementation uses the standard Elo expected-score formula:

E_A = 1 / (1 + 10^((R_B - R_A) / 400))

Where:

• E_A is the expected score for player A (probability of winning)
• R_A and R_B are current ratings
• The divisor 400 controls the spread of expected scores

After each game, ratings update as:

R_A' = R_A + K * (S_A - E_A)
R_B' = R_B + K * (S_B - E_B)

Where:

• K is the K-factor (default: 32)
• S_A is the actual score (1.0 for win, 0.0 for loss)
• S_B = 1.0 - S_A (zero-sum)

Key Parameters

Parameter	Default	Env Var	Description
K-factor	32	ELO_K_FACTOR	Higher = faster rating changes. 32 is standard for club-level chess.
Initial rating	1200	ELO_INITIAL_RATING	Standard chess starting point. All agents begin equal.

The K-factor of 32 means the maximum rating change per game is 32 points (when a 1000-rated agent beats a 2800-rated agent). Typical changes are 10-20 points.

Data Structure: EloRecord

Each agent is tracked with an EloRecord:

Field	Type	Description
entity_id	str	Agent identifier (e.g., template_dan)
entity_type	str	"attacker" or "defender"
display_name	str	Human-readable name for charts
rating	float	Current Elo rating
wins	int	Total wins
losses	int	Total losses
draws	int	Total draws (unused in current implementation)
history	list[float]	Rating snapshot before each update

Derived properties:

• games = wins + losses + draws
• win_rate = wins / games (0.0 if no games)

How Ratings Update

In each arena round, the attacker and defender are opponents in a zero-sum game:

def update(self, attacker_id, defender_id, attacker_won):
    atk = self._attackers[attacker_id]
    dfn = self._defenders[defender_id]

    new_atk, new_dfn = _new_ratings(atk.rating, dfn.rating, attacker_won, self.k_factor)

    # Record history before update
    atk.history.append(atk.rating)
    dfn.history.append(dfn.rating)

    atk.rating = new_atk
    dfn.rating = new_dfn

    if attacker_won:
        atk.wins += 1
        dfn.losses += 1
    else:
        dfn.wins += 1
        atk.losses += 1

The attacker_won flag comes from the judge: attacker_won = bool(judge_verdict.attack_success).

Example Rating Changes

Consider an initial matchup where both agents start at 1200:

Scenario	Attacker Rating Change	Defender Rating Change
Attacker wins (both at 1200)	+16	-16
Defender wins (both at 1200)	-16	+16
Attacker wins (atk=1400, def=1200)	+8.5	-8.5
Attacker wins (atk=1000, def=1400)	+24.8	-24.8

The asymmetry is the core property of Elo: beating a stronger opponent earns more points, while losing to a weaker opponent costs more.

Two Independent Pools

A critical design decision: attackers and defenders are rated independently. This means:

• An attacker's rating reflects how well it bypasses all defenders it has faced
• A defender's rating reflects how well it blocks all attackers it has faced
• The two pools never directly compete -- they are not on the same scale

This is different from a single-pool system where all agents share one leaderboard. The two-pool approach enables meaningful rankings within each role:

Attacker leaderboard: "Which attack strategies are most effective overall?" Defender leaderboard: "Which defense mechanisms catch the most attacks?"

Rating History and Trajectories

The history list in each EloRecord stores the rating before each update. Combined with the current rating, this produces a time series:

def rating_trajectories(self):
    out = {}
    for eid, rec in self._attackers.items():
        out[eid] = rec.history + [rec.rating]
    for eid, rec in self._defenders.items():
        out[eid] = rec.history + [rec.rating]
    return out

The ArenaController takes periodic snapshots (every 10 rounds by default) and writes them to the elo_snapshots SQLite table, enabling the dashboard to render Elo trajectory charts.

Tournament Integration

In ArenaController.run_tournament():

1. All participants are registered with the Elo system at initialization
2. For each round, the controller records elo_before for both agents
3. After judge scoring, elo.update() is called with the outcome
4. Every snapshot_every rounds (default 10), all ratings are written to SQLite
5. A final snapshot is taken after the tournament completes

For multi-turn experiments (adaptive vs static), Elo is updated once per episode, not once per turn. The update_elo=False parameter on run_round() prevents intermediate turns from inflating defender ratings.

Serialization

The EloSystem supports JSON serialization for persistence:

# Serialize
data = elo.to_dict()  # {k_factor, initial_rating, attackers: {...}, defenders: {...}}

# Deserialize
elo = EloSystem.from_dict(data)

However, the primary persistence mechanism is the SQLite elo_snapshots table, which stores periodic snapshots with timestamps for trajectory reconstruction.

Actual Results

From the tournament run 6ed3ad0a (40 rounds, 4 attackers vs 1 defender):

Attacker Rankings

Agent	Rating	Wins	Losses	Win Rate
Template/Hypothetical	1319	8	2	80%
Template/Random	1176	4	6	40%
Template/Roleplay	1154	1	9	10%
Template/Dan	1096	0	10	0%

Key insight: The hypothetical/academic framing template is dramatically more effective than the DAN template. DAN (the most well-known jailbreak) is also the most easily detected by even a simple keyword defender. This aligns with the broader finding that semantic reframing (hypothetical, academic) is harder to detect than explicit jailbreak tokens.

Defender Rankings

Agent	Rating	Wins	Losses	Win Rate
Keyword/Baseline	1255	27	13	67.5%

The keyword defender achieves a 67.5% block rate against template attackers, establishing the baseline that OpenAI Moderation and PromptGuard must exceed.

Cross-References with Other Elo Implementations

Three projects in the knowledge base implement Elo systems with different semantics:

Project	Domain	Elo Semantics	K-Factor	Rating Scale
red-team-arena	AI Safety	Attacker vs Defender (asymmetric)	32	1200 initial
unit-distance-elo-ranking	Math Research	Hypothesis quality ranking	Custom	Elo mapped to research tiers
co-scientist	Multi-Agent	Agent capability ranking	Custom	Elo for hypothesis generation

Key Differences

Red Team Arena: Two independent pools (attackers/defenders). Ratings reflect offensive/defensive capability against the full roster of opponents. The zero-sum property holds within each game but not across pools.

Unit Distance Elo: Single pool ranking mathematical hypotheses. Elo maps to research quality tiers (1800 = proof of concept, 2200 = matches Sawin, 2700+ = human SOTA). Uses a custom Elo variant where the "opponent" is the problem difficulty.

Co-Scientist Elo: Rates agent teams in multi-agent scientific reasoning. Similar to the arena in that it rates capabilities, but the "game" is hypothesis generation quality rather than attack/defense.

The common thread: Elo is a universal language for ranking relative capability in any competitive or evaluative context. The formula is the same; only the semantics of "win" and "loss" change.

Cross-References

• red-team-arena -- Full system overview
• red-team-arena-agents -- The agents whose ratings are tracked
• red-team-arena-experiments -- Experiments that produce Elo data
• unit-distance-elo-ranking -- Elo in mathematical research context
• co-scientist -- Multi-agent Elo in scientific reasoning