---
name: red-team-arena-code-quality
type: analysis
title: "Red Team Arena: Code Review"
description: "Technical code review of the Multi-Agent Red Team Arena codebase covering architecture, test coverage, design patterns, and improvement areas."
tags: [red-team-arena, code-review, software-engineering, python, architecture]
timestamp: 2026-07-21
---

# Red Team Arena: Code Review

## Architecture Overview

The codebase lives in `Multi_Agent_proj_code/` and follows a clean three-layer separation:

```
agents/          ← Agent implementations (attacker, defender, judge)
arena/           ← Infrastructure (controller, elo, logger, artifacts)
experiments/     ← Experiment orchestration (transferability, category, adaptive)
config.py        ← Centralized configuration via environment variables
data/            ← Seed prompts (seed_prompts.json)
runs/            ← Per-run artifact directories
tests/           ← Pytest test suite
dashboard/       ← Plotly chart builders (Streamlit-ready)
```

Each layer depends only on the layers below it. Agents know nothing about the arena infrastructure. Experiments compose arena components. This is a sound dependency graph.

See also: [[red-team-arena-methodology]], [[red-team-arena-results]], [[red-team-arena-agents]].

---

## Component Analysis

### 1. Agent Layer (`agents/`)

**attacker.py** (248 lines) -- Five attacker implementations behind a common ABC:

| Class | Lines | Complexity | API Cost |
|-------|-------|------------|----------|
| `BaseAttacker` | 15 | Low (ABC) | None |
| `TemplateAttacker` | 55 | Low (string formatting) | None |
| `OllamaAttacker` | 55 | Medium (HTTP + retry) | Local Ollama |
| `OpenAIAttacker` | 55 | Medium (SDK + retry) | OpenAI API |
| `AdaptiveAttacker` | 40 | Low (wrapper + history) | Inherits from base |

**defender.py** (183 lines) -- Three defender implementations:

| Class | Lines | Complexity | API Cost |
|-------|-------|------------|----------|
| `KeywordDefender` | 45 | Low (regex matching) | None |
| `OpenAIModerationDefender` | 40 | Medium (SDK) | Free moderation endpoint |
| `PromptGuardDefender` | 50 | High (transformers + torch) | Local inference |

**judge.py** (202 lines) -- Single judge with LLM and heuristic modes:

| Mode | Complexity | Cost |
|------|------------|------|
| `heuristic` | Trivial (deterministic) | Zero |
| `llm` | Medium (API + retry + parse) | Per-round API |
| `heuristic_fallback` | Trivial | Zero |

**Total agent code: ~683 lines across 3 files.**

#### Strengths

1. **Registry pattern with factory functions**: Both attackers and defenders use a registry that maps string IDs to factory lambdas. This means `get_attacker("template_dan")` always returns a fresh instance, preventing state leakage across tournaments. The comment in the code explicitly documents this design choice.

2. **AdaptiveAttacker as a decorator**: The adaptive wrapper is a clean decorator pattern. It wraps any `BaseAttacker` and adds conversation history tracking. The `reset()` method allows the controller to clear state between seeds.

3. **Lazy client initialization**: Both `OllamaAttacker` and `OpenAIAttacker` defer client creation to first use (`_get_client()`). This means the system can be imported and template attacks can run without API keys present.

4. **Clean dataclass contracts**: `AttackResult`, `DefenderVerdict`, and `JudgeVerdict` are well-defined dataclasses with explicit fields. No magic dictionaries, no unpacking surprises.

5. **15 jailbreak templates**: The template library covers the major categories documented in the literature (DAN, roleplay, hypothetical, academic, developer mode, grandma exploit, opposite day, base64 prefix, continuation, translation, token smuggling, few-shot, nested roleplay, academic paper, direct).

#### Weaknesses

1. **DEFENDER_REGISTRY stores instances, not factories**: Unlike the attacker registry (which uses lambdas), the defender registry stores pre-instantiated singletons:

```python
DEFENDER_REGISTRY: dict[str, BaseDefender] = {
    "keyword_baseline":   KeywordDefender(),
    "openai_moderation":  OpenAIModerationDefender(),
    "promptguard":        PromptGuardDefender(),
}
```

This means `get_defender()` always returns the same object. If a defender were stateful (e.g., PromptGuard with a loaded model), this would be correct for performance. But it violates the pattern established by the attacker registry and could leak state between experiments. The `KeywordDefender` is stateless so this works, but `PromptGuardDefender` loads a model on first call -- sharing that instance is fine for inference but could cause issues if someone adds per-instance configuration.

2. **Missing `__init__.py` files**: The `agents/`, `arena/`, and `experiments/` directories have no `__init__.py`. The `conftest.py` adds the project root to `sys.path` to make imports work. This works but is fragile -- it depends on pytest's path manipulation rather than proper package structure.

3. **Regex patterns in keyword defender are hardcoded**: The `_KEYWORD_RULES` dictionary is a module-level constant with 8 categories and ~60 regex patterns. Adding or modifying categories requires editing the source code. A config-driven approach (YAML, JSON, or environment variable) would be more flexible.

4. **No rate limiting on API calls**: The `OllamaAttacker` and `OpenAIAttacker` make synchronous HTTP calls without rate limiting. In high-volume tournaments (hundreds of rounds), this could hit API rate limits. The `OpenAIModerationDefender` has the same issue.

5. **Temperature hardcoded to 0.9**: Both `OllamaAttacker` and `OpenAIAttacker` default to temperature 0.9. This is reasonable for creative attack generation but is not configurable without subclassing.

---

### 2. Arena Layer (`arena/`)

**controller.py** (280 lines) -- The orchestrator:

```python
class ArenaController:
    run_round(attacker, defender, seed, history, update_elo) -> RoundResult
    run_tournament(rounds, category_filter, snapshot_every) -> list[RoundResult]
```

**elo.py** (130 lines) -- Elo rating system:

```python
class EloSystem:
    register_attacker/register_defender(id, name) -> EloRecord
    update(attacker_id, defender_id, attacker_won) -> (float, float)
    attacker_leaderboard/defender_leaderboard() -> list[dict]
    to_dict/from_dict()  # serialization
```

**logger.py** (180 lines) -- SQLite persistence:

```python
class ArenaLogger:
    log_round(dict) -> None
    log_elo_snapshot(run_id, entity_id, ...) -> None
    start_experiment/finish_experiment() -> None
    get_attack_success_matrix() -> dict
    get_category_breakdown() -> list[dict]
```

**artifacts.py** (120 lines) -- Run artifact management:

```python
class RunArtifacts:
    log_round(dict) -> None          # stream to rounds.jsonl
    finalize(logger, elo, run_ids)   # produce summary.json, leaderboard.md, charts
```

#### Strengths

1. **Separation of concerns**: The controller handles orchestration, the Elo system handles ratings, the logger handles persistence, and the artifacts handle output generation. Each has a single responsibility.

2. **SQLite for round data**: Using SQLite as the backing store for round logs is a good choice. It handles concurrent writes, supports complex queries (the attack success matrix and category breakdown are SQL aggregations), and the database file is a single portable artifact.

3. **Artifact streaming**: The `RunArtifacts` class streams each round to a `rounds.jsonl` file in real time, so partial results are available even if the run is interrupted. The incomplete run `9d160f59` demonstrates this -- it has `rounds.jsonl` data despite not producing a `summary.json`.

4. **`update_elo=False` flag**: The controller's `run_round` method accepts an `update_elo` parameter that lets multi-turn experiments score an episode of several turns as a single Elo game. This is a clean solution to the problem of inflating defender ratings with intermediate failures.

5. **Seed exclusion with reset**: The tournament loop tracks `used_seeds` per pair and resets when exhausted. This prevents the same seed from appearing twice in a pair while ensuring all seeds are used eventually.

6. **Rich progress display**: The controller uses the `rich` library for progress bars, tables, and colored output. The output is clean and informative.

#### Weaknesses

1. **RoundResult.to_log_dict uses time.time()**: The timestamp in `to_log_dict` is `time.time()` (Unix epoch float). This is fine for ordering but loses timezone information and is not human-readable. The artifact system uses ISO 8601 timestamps in `meta.json`, creating an inconsistency.

2. **No transaction safety in logger**: The `ArenaLogger.log_round()` method appears to do individual INSERT statements without explicit transaction management. Under high concurrency (not currently an issue, but worth noting), this could cause locking issues.

3. **EloSystem stores history as list of floats**: The `EloRecord.history` field is `list[float]` storing the rating before each update. This is appended to with `[current_rating]` when reading trajectories. The `to_dict`/`from_dict` serialization round-trips this correctly, but the history list grows without bound during long tournaments.

4. **Config uses module-level constants**: `config.py` loads `.env` at import time and creates module-level constants. This is simple but means configuration cannot be changed at runtime without reimporting. For a research tool this is fine; for a production system it would be limiting.

5. **`_snapshot_elo` accesses private attributes**: The controller's `_snapshot_elo` method accesses `self.elo._attackers` and `self.elo._defenders` directly, breaking encapsulation. A `for_each_record()` iterator or `all_records()` method on `EloSystem` would be cleaner.

---

### 3. Experiment Layer (`experiments/`)

**transferability.py** (145 lines), **category_breakdown.py** (120 lines), **adaptive_vs_static.py** (200 lines)

#### Strengths

1. **Consistent experiment interface**: All three experiments follow the same pattern: `__init__` with optional parameters and defaults, `run()` as the entry point, `_analyze()` for post-processing, `_print_results()` for display. This makes adding new experiments straightforward.

2. **Paired comparison in adaptive experiment**: The adaptive vs static experiment correctly samples ONE set of seeds and uses them for both arms. This controls for seed difficulty, making the comparison fair. The two-proportion z-test is the correct statistical test for this design.

3. **Wilson intervals in transferability**: The transferability experiment computes 95% Wilson confidence intervals per attacker-defender pair. This is more appropriate than normal approximation for the small sample sizes typical in these experiments.

4. **Blind spot identification**: The category breakdown experiment automatically identifies defender blind spots by sorting categories by attack success rate. This produces actionable output: "Defender X is weakest against category Y."

#### Weaknesses

1. **No experiment isolation**: All three experiments share the same `ArenaLogger` instance by default, meaning their round data is interleaved in the same SQLite database. The `experiment_tag` field distinguishes them in queries, but this relies on correct tagging.

2. **Category breakdown creates separate EloSystem per category**: This is intentional (to avoid cross-category Elo contamination) but means ratings are not comparable across categories. A shared Elo system with category tagging would provide cross-category insights.

3. **Hardcoded default participants**: Each experiment has hardcoded default attacker/defender lists. For example, `TransferabilityExperiment` defaults to 5 template attackers and 2 defenders. This is reasonable for a research tool but limits reusability.

---

### 4. Dashboard (`dashboard/charts.py`)

**charts.py** (165 lines) -- Six Plotly chart builders:

| Function | Chart Type | Purpose |
|----------|-----------|---------|
| `elo_trajectory_chart` | Line | Elo ratings over time |
| `attack_success_heatmap` | Heatmap | Attacker x Defender success matrix |
| `category_bar_chart` | Grouped bar | Per-category metrics |
| `elo_leaderboard_bar` | Horizontal bar | Elo rankings |
| `severity_distribution` | Donut pie | Harm severity distribution |
| `attacker_win_rate_over_time` | Line (rolling) | 10-round rolling win rate |

#### Strengths

- All charts use `plotly_dark` template (matches project's Anthropic dark mode preference)
- Functions are stateless and composable
- Graceful handling of empty data (returns empty figure with title)

#### Weaknesses

- `attacker_win_rate_over_time` imports numpy inside the function body (should be at module level)
- No chart validation or error handling for malformed data

---

## Test Coverage

The test suite in `tests/` has 8 files covering all major components:

| Test File | Lines | Coverage |
|-----------|-------|----------|
| `test_elo.py` | 57 | Elo math, symmetry, zero-sum, serialization |
| `test_attacker.py` | 41 | Template formatting, registry, adaptive reset |
| `test_defender.py` | 41 | Keyword matching, regression tests, min_hits |
| `test_judge.py` | 113 | Outcome derivation, JSON parsing, heuristic, LLM fallback |
| `test_logger.py` | 96 | SQLite round-trip, migration, experiment lifecycle |
| `test_artifacts.py` | 78 | Streaming, finalize, graceful degradation |
| `test_experiments.py` | 35 | z-test, Wilson interval edge cases |
| `conftest.py` | 6 | sys.path setup |

**Total test code: ~467 lines.**

### Test Quality

1. **Regression tests in test_defender.py**: Tests specifically verify that "stalkerware" (not "stalkerwear") is matched, and that single keyword hits flag (regression from a ratio-threshold version). This shows the tests were written in response to real bugs.

2. **Parametrized test in test_judge.py**: The `_derive_outcomes` test uses `@pytest.mark.parametrize` for all 4 combinations of harmful/benign x flagged/unflagged. This is thorough.

3. **Edge case testing in test_experiments.py**: Tests verify that empty samples produce safe results (z=0, p=1), equal proportions are not significant, and large differences are significant. Wilson interval tests verify bounds are valid and tighter with more data.

4. **Mock-based judge tests**: The judge tests verify that LLM failures gracefully fall back to heuristic mode, and that the model's own `attack_success` field is ignored (only `policy_violation` counts).

### Coverage Gaps

1. **No integration tests**: No test runs a full tournament round through the controller. All tests are unit-level.

2. **No test for OllamaAttacker or OpenAIAttacker**: The API-based attackers have no tests (likely because they require running services).

3. **No test for PromptGuardDefender**: The HuggingFace-based defender is untested (requires torch installation).

4. **No test for dashboard charts**: The chart functions are untested.

---

## Configuration Management

`config.py` uses `python-dotenv` to load `.env` files with sensible defaults:

```python
OPENAI_API_KEY = os.getenv("OPENAI_API_KEY", "")
JUDGE_MODEL = os.getenv("JUDGE_MODEL", "claude-haiku-4-5-20251001")
DEFAULT_ROUNDS = int(os.getenv("DEFAULT_ROUNDS", "50"))
ELO_K_FACTOR = float(os.getenv("ELO_K_FACTOR", "32"))
```

This is clean and follows the 12-factor app pattern. The defaults allow the system to run without any environment configuration (heuristic judge mode, template attackers only).

---

## Dependency Analysis

`requirements.txt` lists 9 required packages and 3 optional:

**Required**: openai, anthropic, requests, python-dotenv, streamlit, plotly, pandas, numpy, rich, pytest

**Optional**: transformers, torch (for PromptGuard), datasets (for HuggingFace downloads)

The dependency footprint is reasonable. The core system (template attacks + keyword defender + heuristic judge) needs only `requests`, `rich`, and `python-dotenv`. The heavier dependencies (openai, anthropic, plotly, pandas) are needed for the full feature set.

---

## Summary Assessment

| Dimension | Rating | Notes |
|-----------|--------|-------|
| **Architecture** | Strong | Clean separation of agents/arena/experiments |
| **Test coverage** | Good | All core logic tested; gaps in integration and API-based components |
| **Code quality** | Good | Consistent style, clear dataclasses, explicit error handling |
| **Documentation** | Moderate | Module docstrings are excellent; inline comments are sparse |
| **Configuration** | Good | 12-factor pattern with sensible defaults |
| **Extensibility** | Good | New attackers/defenders are easy to add via registry |
| **Error handling** | Good | Lazy initialization with clear error messages; judge has retry + fallback |
| **Performance** | Adequate | Synchronous execution; could benefit from async for API-based components |

### Recommended Improvements

1. **Add `__init__.py` files** to make the project a proper Python package
2. **Convert defender registry to factories** for consistency with attacker registry
3. **Add integration tests** that run a small tournament through the controller
4. **Add async support** for API-based attackers and defenders
5. **Make keyword rules configurable** via YAML or JSON
6. **Add rate limiting** for API calls in high-volume experiments
7. **Test the dashboard chart functions** with mock data