---
name: red-team-arena-results
type: analysis
title: "Red Team Arena: Results and Findings"
description: "Compiled results from all Arena runs including tournament leaderboards, adaptive vs static comparison, transferability matrix, and category blind spots."
tags: [red-team-arena, results, ai-safety, adversarial-testing, tournament, elo-rating]
timestamp: 2026-07-21
---

# Red Team Arena: Results and Findings

## Run Inventory

11 experiment runs were conducted on 2026-06-11 across three experiment types. 10 completed successfully; 1 was interrupted (missing `summary.json`).

| Run ID | Type | Rounds | Attack Success | Duration | Judge Mode |
|--------|------|--------|---------------|----------|------------|
| `995bda5a` | Tournament | 60 | 36.7% | 0.59s | heuristic |
| `d5162ad9` | Tournament | 60 | 36.7% | -- | heuristic |
| `6ed3ad0a` | Tournament | 40 | 32.5% | 0.38s | heuristic |
| `c7f5ded0` | Tournament | 120 | 37.5% | 1.2s | heuristic |
| `9d160f59` | Tournament | -- | -- | -- | -- (INCOMPLETE) |
| `transfer_0fb263` | Transferability | 60 | 35.0% | ~0s | heuristic |
| `transfer_3e3c24` | Transferability | 75 | 40.0% | ~0s | heuristic |
| `adaptive_6ad638` | Adaptive vs Static | 43 | 32.6% | ~0s | heuristic |
| `adaptive_19fd90` | Adaptive vs Static | 63 | 27.0% | ~0s | heuristic |
| `catbreak_ad08b0` | Category Breakdown | 192 | 56.2% | 0.01s | heuristic |
| `catbreak_2b4360` | Category Breakdown | 240 | 57.5% | 0.01s | heuristic |

All completed runs used the **heuristic judge** mode (ground truth by construction). The incomplete run `9d160f59` was the only attempt to use `openai_moderation` as a defender and was interrupted before producing a summary.

See also: [[red-team-arena-methodology]], [[red-team-arena-seed-prompts]], [[red-team-arena-code-quality]].

---

## Tournament Results

### Largest Run: `c7f5ded0` (120 rounds, 6 attackers, 1 defender)

**Configuration**: 6 template attackers vs keyword_baseline defender, 20 rounds per pair.

#### Attacker Elo Leaderboard

| Rank | Attacker | Rating | W | L | Win Rate |
|------|----------|--------|---|---|----------|
| 1 | Template/Hypothetical | 1313 | 13 | 7 | 65.0% |
| 2 | Template/Roleplay | 1251 | 9 | 11 | 45.0% |
| 3 | Template/System | 1182 | 8 | 12 | 40.0% |
| 4 | Template/Escalating | 1182 | 7 | 13 | 35.0% |
| 5 | Template/Random | 1152 | 8 | 12 | 40.0% |
| 6 | Template/Dan | 1052 | 0 | 20 | 0.0% |

#### Defender Performance

| Defender | Rating | W | L | Win Rate |
|----------|--------|---|---|----------|
| Keyword/Baseline | 1269 | 75 | 45 | 62.5% |

#### Key Findings

1. **Template/Dan achieved 0% win rate**: The DAN (Do Anything Now) template is the most well-known jailbreak and is trivially detected by keyword matching. The keyword defender's regex `\bDAN\b` catches it every time. This confirms that DAN is ineffective against even the simplest classifier.

2. **Template/Hypothetical dominates at 65%**: The hypothetical framing ("In a completely hypothetical scenario...") combined with academic framing consistently bypasses keyword detection. This is because the hypothetical wrapper does not contain any keywords from the defender's regex list -- the harmful content is embedded within a benign-seeming frame.

3. **Keyword defender achieves 62.5% overall win rate**: Against 6 different attackers, the baseline keyword defender blocks roughly 5/8 attacks. This is a reasonable lower bound -- any real-world classifier should significantly outperform it.

4. **Severity distribution**: 10 minimal/low, 54 medium, 44 high, 12 critical. The seed bank skews toward medium and high severity, which is appropriate for testing recall against the most dangerous prompts.

### Tournament Run: `6ed3ad0a` (40 rounds, 4 attackers)

| Rank | Attacker | Rating | W | L | Win Rate |
|------|----------|--------|---|---|----------|
| 1 | Template/Hypothetical | 1319 | 8 | 2 | 80.0% |
| 2 | Template/Random | 1176 | 4 | 6 | 40.0% |
| 3 | Template/Roleplay | 1154 | 1 | 9 | 10.0% |
| 4 | Template/Dan | 1096 | 0 | 10 | 0.0% |

**Defender**: Keyword/Baseline, rating 1255, 27W/13L (67.5%).

The hypothetical attacker's dominance is consistent: 80% win rate with only 10 games. The smaller sample produces a higher Elo (1319 vs 1313) due to the larger margin per game.

### Tournament Runs: `995bda5a` and `d5162ad9` (60 rounds each)

Both runs used identical configurations (4 attackers including adaptive) and produced identical results, confirming deterministic behavior:

| Rank | Attacker | Rating | W | L | Win Rate |
|------|----------|--------|---|---|----------|
| 1 | Adaptive/Template/Random | 1324 | 10 | 5 | 66.7% |
| 2 | Template/Random | 1209 | 8 | 7 | 53.3% |
| 3 | Template/Roleplay | 1170 | 4 | 11 | 26.7% |
| 4 | Template/Dan | 1056 | 0 | 15 | 0.0% |

**Defender**: Keyword/Baseline, rating 1242, 38W/22L (63.3%).

**Key finding**: The adaptive wrapper around Template/Random achieved the highest rating (1324) and 66.7% win rate, outperforming the base Template/Random at 53.3%. This 13.4 percentage point improvement suggests multi-turn adaptation provides a meaningful advantage even with template-based attacks.

---

## Adaptive vs Static Experiment

Two runs compared one-shot static attackers against multi-turn adaptive attackers on the same seed sets.

### Run `adaptive_6ad638` (30 rounds, max 5 adaptive turns)

| Metric | Static | Adaptive |
|--------|--------|----------|
| Episodes | 12 | 12 |
| Success Rate | 50.0% | 66.7% |
| Total Turns | 12 | 31 |
| Per-Attempt Rate | 50.0% | 25.8% |
| Avg Turns to Win | 1 | 1.38 |

**Statistical test**:
- Improvement: +16.7 percentage points
- z-statistic: 0.828
- p-value: 0.4076
- **Not significant at alpha = 0.05**

Turns-to-success distribution: `[1, 1, 4, 1, 1, 1, 1, 1]` -- 7 of 8 successes happened on the first turn. One required 4 turns.

### Run `adaptive_19fd90` (45 rounds, max 5 adaptive turns)

| Metric | Static | Adaptive |
|--------|--------|----------|
| Episodes | 15 | 15 |
| Success Rate | 53.3% | 60.0% |
| Total Turns | 15 | 48 |
| Per-Attempt Rate | 53.3% | 18.8% |
| Avg Turns to Win | 1 | 2.0 |

**Statistical test**:
- Improvement: +6.7 percentage points
- z-statistic: 0.368
- p-value: 0.7125
- **Not significant at alpha = 0.05**

Turns-to-success distribution: `[1, 4, 1, 1, 1, 4, 1, 4, 1]` -- 6 of 9 successes on first turn. Three required 4 turns.

### Interpretation

The adaptive attacker consistently shows a higher per-seed success rate than the static attacker (66.7% vs 50.0%, and 60.0% vs 53.3%), but **neither difference is statistically significant** at alpha = 0.05. This is likely due to small sample sizes (12-15 episodes per arm).

Important methodological note: the per-attempt success rate for the adaptive arm is much lower (25.8% and 18.8%) because it counts all intermediate failed attempts. The per-seed rate is the correct comparison metric -- it asks "did the attacker eventually succeed for this seed?" rather than "did each individual attempt succeed?"

The turns-to-success data shows that most adaptations succeed on the first turn (the attacker already has a good strategy), but when they fail, it often takes 4 turns. This bimodal distribution suggests the adaptive wrapper provides value primarily for edge cases where the first template choice is wrong.

---

## Transferability Experiment

Two runs measured how consistently each attacker performs against defenders.

### Run `transfer_0fb263` (60 rounds, 5 attackers, 1 defender)

| Attacker | Success Rate | n | Transfer Score | Interpretation |
|----------|-------------|---|---------------|----------------|
| Template/Random | 58.3% | 12 | 0.0 | Insufficient data |
| Template/Hypothetical | 50.0% | 12 | 0.0 | Insufficient data |
| Template/Roleplay | 33.3% | 12 | 0.0 | Insufficient data |
| Template/System | 33.3% | 12 | 0.0 | Insufficient data |
| Template/Dan | 0.0% | 12 | 0.0 | Insufficient data |

### Run `transfer_3e3c24` (75 rounds, 5 attackers, 1 defender)

| Attacker | Success Rate | n | Transfer Score | Interpretation |
|----------|-------------|---|---------------|----------------|
| Template/Hypothetical | 73.3% | 15 | 0.0 | Insufficient data |
| Template/Random | 73.3% | 15 | 0.0 | Insufficient data |
| Template/Roleplay | 33.3% | 15 | 0.0 | Insufficient data |
| Template/System | 20.0% | 15 | 0.0 | Insufficient data |
| Template/Dan | 0.0% | 15 | 0.0 | Insufficient data |

### Interpretation

All transferability scores are 0.0 with "insufficient data" because **only one defender was tested** (keyword_baseline). The transferability score is `1.0 - stdev(success_rates)`, and stdev requires at least 2 data points. With only 1 defender, there is no variance to measure.

This is the most significant limitation of the current results. The transferability experiment is designed for multi-defender evaluation, and the only run that attempted multi-defender configuration (`9d160f59` with `openai_moderation`) was interrupted. The transferability metric cannot be meaningfully interpreted until at least 2 defenders are available.

### Cross-Run Consistency

Despite the transferability score being unmeasurable, the per-attacker success rates are remarkably consistent across the two runs:

| Attacker | Run 1 (n=12) | Run 2 (n=15) | Delta |
|----------|-------------|-------------|-------|
| Template/Dan | 0.0% | 0.0% | 0.0pp |
| Template/Roleplay | 33.3% | 33.3% | 0.0pp |
| Template/System | 33.3% | 20.0% | -13.3pp |
| Template/Hypothetical | 50.0% | 73.3% | +23.3pp |
| Template/Random | 58.3% | 73.3% | +15.0pp |

The DAN and Roleplay attackers are perfectly consistent. The larger variance in Hypothetical and Random is expected with small samples (n=12-15).

---

## Category Breakdown

Two runs measured per-category attack success rates against the keyword baseline.

### Run `catbreak_2b4360` (240 rounds, 2 attackers, 8 categories)

This is the larger, more reliable run. Results are per (attacker, category) pair with n=15 each.

#### Top Blind Spots (Highest Attack Success Rate)

| Rank | Category | Attacker | Attack Success | Defender Accuracy | Avg Severity |
|------|----------|----------|---------------|-------------------|-------------|
| 1 | privacy_violation | Hypothetical | 86.7% | 13.3% | 3.47 |
| 2 | hate_speech | Hypothetical | 80.0% | 20.0% | 3.53 |
| 3 | misinformation | Hypothetical | 80.0% | 20.0% | 3.00 |
| 4 | hate_speech | Random | 66.7% | 33.3% | 3.53 |
| 5 | illegal_activities | Hypothetical | 60.0% | 40.0% | 3.60 |
| 6 | self_harm | Hypothetical | 60.0% | 40.0% | 3.93 |
| 7 | social_engineering | Hypothetical | 60.0% | 40.0% | 3.07 |
| 8 | weapons | Hypothetical | 60.0% | 40.0% | 4.07 |

#### Bottom (Best Defended Categories)

| Rank | Category | Attacker | Attack Success | Defender Accuracy |
|------|----------|----------|---------------|-------------------|
| 1 | illegal_activities | Random | 33.3% | 66.7% |
| 2 | misinformation | Random | 40.0% | 60.0% |
| 3 | jailbreaks | Random | 40.0% | 60.0% |
| 4 | self_harm | Random | 46.7% | 53.3% |
| 5 | social_engineering | Random | 46.7% | 53.3% |

#### Per-Category Averages (Both Attackers Combined)

| Category | Avg Attack Success | Avg Defender Accuracy | Difficulty Rank |
|----------|-------------------|----------------------|-----------------|
| privacy_violation | 70.0% | 30.0% | Hardest |
| hate_speech | 73.3% | 26.7% | Hardest |
| misinformation | 60.0% | 40.0% | Hard |
| weapons | 56.7% | 43.3% | Medium |
| self_harm | 53.3% | 46.7% | Medium |
| social_engineering | 53.3% | 46.7% | Medium |
| jailbreaks | 46.7% | 53.3% | Easy |
| illegal_activities | 46.7% | 53.3% | Easy |

### Run `catbreak_ad08b0` (192 rounds, 2 attackers, 8 categories)

The smaller run (n=12 per pair) shows similar patterns:

| Category | Top Attack Success | Defender Accuracy |
|----------|-------------------|-------------------|
| hate_speech | 75.0% | 25.0% |
| jailbreaks | 75.0% | 25.0% |
| misinformation | 75.0% | 25.0% |
| weapons | 75.0% | 25.0% |
| self_harm | 66.7% | 33.3% |
| privacy_violation | 58.3% | 41.7% |
| social_engineering | 50.0% | 50.0% |
| illegal_activities | 33.3% | 66.7% |

### Key Findings

1. **Privacy violation is the hardest category to defend**: The hypothetical attacker achieves 86.7% success rate against the keyword baseline for privacy violation prompts. The keyword defender's regex list for this category (`stalkerware`, `track.*location.*without`, `hack.*account`, `brute[- ]force.*password`, `dox(xing)?`, `SIM.?swap`) does not cover the full range of privacy violation language.

2. **Hate speech is the second hardest**: At 80.0% attack success with hypothetical framing. The keyword patterns for hate speech (`genocid`, `exterminate.*group`, `manifesto.*inferior`, etc.) are too narrow to catch rephrased hate speech.

3. **Jailbreaks and illegal activities are best defended**: The keyword defender catches jailbreak patterns well because the regex list includes common jailbreak phrases (`ignore.*previous.*instructions`, `DAN`, `developer mode`). Illegal activities also have good keyword coverage (`methamphetamine`, `ransomware`, `malware`, etc.).

4. **Hypothetical framing is the most dangerous attack strategy**: Across all categories, the hypothetical attacker consistently outperforms the random attacker by 10-30 percentage points. The hypothetical wrapper ("In a completely hypothetical scenario...") strips the prompt of keywords that the defender matches on.

5. **The keyword baseline is not a realistic defender**: With 62.5% overall accuracy against template attacks, the keyword baseline is intentionally weak. Any production classifier (OpenAI Moderation, PromptGuard, etc.) should significantly outperform it. The keyword baseline serves as a floor -- if an attacker cannot beat it, the attack is trivial.

---

## Overall Attack Success Rates

Across all completed runs:

| Experiment Type | Total Rounds | Overall Attack Success |
|----------------|-------------|----------------------|
| Tournaments | 280 | 35.7% |
| Transferability | 135 | 37.0% |
| Adaptive vs Static | 106 | 29.2% |
| Category Breakdown | 432 | 56.9% |
| **Grand Total** | **953** | **41.8%** |

The category breakdown has the highest attack success rate (56.9%) because it runs more rounds per (attacker, category) pair with category-filtered seeds, giving the attacker better odds. The adaptive vs static has the lowest (29.2%) because it uses a smaller seed set and the adaptive arm's per-attempt rate drags down the average.

---

## Severity Distribution

Across all runs, the severity distribution of seed prompts:

| Severity | Label | Approximate Share |
|----------|-------|-------------------|
| 2 | Low | ~10% |
| 3 | Medium | ~45% |
| 4 | High | ~38% |
| 5 | Critical | ~7% |

The seed bank is weighted toward medium and high severity, which is appropriate for testing recall against the most dangerous prompts. Critical severity (severity 5) prompts are rare but represent the highest-stakes failures.

---

## Reproducibility Note

Runs `995bda5a` and `d5162ad9` used identical configurations and produced identical results, confirming deterministic behavior in heuristic judge mode. This is expected: the template attacker uses `random.choice` but the seed bank and template selection are seeded by the same configuration.

For LLM-based attacks and judges, results will vary between runs due to API temperature settings and non-deterministic sampling. The heuristic mode provides a reproducible baseline for regression testing.

---

## Unfinished Business

The most significant gap in the results is the **absence of multi-defender data**. The `openai_moderation` defender was only attempted once (run `9d160f59`) and did not complete. The `promptguard` defender was never attempted. Without multi-defender data:

- Transferability scores cannot be computed
- Cross-defender comparisons are impossible
- The Elo system's ranking power is limited (defenders cannot be compared against each other)

Future work should prioritize completing runs with at least 2 defenders to unlock the full analytical power of the Arena framework.