Red Team Arena: Code Review

Architecture Overview

The codebase lives in Multi_Agent_proj_code/ and follows a clean three-layer separation:

agents/          ← Agent implementations (attacker, defender, judge)
arena/           ← Infrastructure (controller, elo, logger, artifacts)
experiments/     ← Experiment orchestration (transferability, category, adaptive)
config.py        ← Centralized configuration via environment variables
data/            ← Seed prompts (seed_prompts.json)
runs/            ← Per-run artifact directories
tests/           ← Pytest test suite
dashboard/       ← Plotly chart builders (Streamlit-ready)

Each layer depends only on the layers below it. Agents know nothing about the arena infrastructure. Experiments compose arena components. This is a sound dependency graph.

See also: red-team-arena-methodology, red-team-arena-results, red-team-arena-agents.

---

Component Analysis

1. Agent Layer (agents/)

attacker.py (248 lines) -- Five attacker implementations behind a common ABC:

Class	Lines	Complexity	API Cost
BaseAttacker	15	Low (ABC)	None
TemplateAttacker	55	Low (string formatting)	None
OllamaAttacker	55	Medium (HTTP + retry)	Local Ollama
OpenAIAttacker	55	Medium (SDK + retry)	OpenAI API
AdaptiveAttacker	40	Low (wrapper + history)	Inherits from base

defender.py (183 lines) -- Three defender implementations:

Class	Lines	Complexity	API Cost
KeywordDefender	45	Low (regex matching)	None
OpenAIModerationDefender	40	Medium (SDK)	Free moderation endpoint
PromptGuardDefender	50	High (transformers + torch)	Local inference

judge.py (202 lines) -- Single judge with LLM and heuristic modes:

Mode	Complexity	Cost
heuristic	Trivial (deterministic)	Zero
llm	Medium (API + retry + parse)	Per-round API
heuristic_fallback	Trivial	Zero

Total agent code: ~683 lines across 3 files.

Strengths

1. Registry pattern with factory functions: Both attackers and defenders use a registry that maps string IDs to factory lambdas. This means get_attacker("template_dan") always returns a fresh instance, preventing state leakage across tournaments. The comment in the code explicitly documents this design choice.

2. AdaptiveAttacker as a decorator: The adaptive wrapper is a clean decorator pattern. It wraps any BaseAttacker and adds conversation history tracking. The reset() method allows the controller to clear state between seeds.

3. Lazy client initialization: Both OllamaAttacker and OpenAIAttacker defer client creation to first use (_get_client()). This means the system can be imported and template attacks can run without API keys present.

4. Clean dataclass contracts: AttackResult, DefenderVerdict, and JudgeVerdict are well-defined dataclasses with explicit fields. No magic dictionaries, no unpacking surprises.

5. 15 jailbreak templates: The template library covers the major categories documented in the literature (DAN, roleplay, hypothetical, academic, developer mode, grandma exploit, opposite day, base64 prefix, continuation, translation, token smuggling, few-shot, nested roleplay, academic paper, direct).

Weaknesses

1. DEFENDER_REGISTRY stores instances, not factories: Unlike the attacker registry (which uses lambdas), the defender registry stores pre-instantiated singletons:

DEFENDER_REGISTRY: dict[str, BaseDefender] = {
    "keyword_baseline":   KeywordDefender(),
    "openai_moderation":  OpenAIModerationDefender(),
    "promptguard":        PromptGuardDefender(),
}

This means get_defender() always returns the same object. If a defender were stateful (e.g., PromptGuard with a loaded model), this would be correct for performance. But it violates the pattern established by the attacker registry and could leak state between experiments. The KeywordDefender is stateless so this works, but PromptGuardDefender loads a model on first call -- sharing that instance is fine for inference but could cause issues if someone adds per-instance configuration.

2. Missing __init__.py files: The agents/, arena/, and experiments/ directories have no __init__.py. The conftest.py adds the project root to sys.path to make imports work. This works but is fragile -- it depends on pytest's path manipulation rather than proper package structure.

3. Regex patterns in keyword defender are hardcoded: The _KEYWORD_RULES dictionary is a module-level constant with 8 categories and ~60 regex patterns. Adding or modifying categories requires editing the source code. A config-driven approach (YAML, JSON, or environment variable) would be more flexible.

4. No rate limiting on API calls: The OllamaAttacker and OpenAIAttacker make synchronous HTTP calls without rate limiting. In high-volume tournaments (hundreds of rounds), this could hit API rate limits. The OpenAIModerationDefender has the same issue.

5. Temperature hardcoded to 0.9: Both OllamaAttacker and OpenAIAttacker default to temperature 0.9. This is reasonable for creative attack generation but is not configurable without subclassing.

---

2. Arena Layer (arena/)

controller.py (280 lines) -- The orchestrator:

class ArenaController:
    run_round(attacker, defender, seed, history, update_elo) -> RoundResult
    run_tournament(rounds, category_filter, snapshot_every) -> list[RoundResult]

elo.py (130 lines) -- Elo rating system:

class EloSystem:
    register_attacker/register_defender(id, name) -> EloRecord
    update(attacker_id, defender_id, attacker_won) -> (float, float)
    attacker_leaderboard/defender_leaderboard() -> list[dict]
    to_dict/from_dict()  # serialization

logger.py (180 lines) -- SQLite persistence:

class ArenaLogger:
    log_round(dict) -> None
    log_elo_snapshot(run_id, entity_id, ...) -> None
    start_experiment/finish_experiment() -> None
    get_attack_success_matrix() -> dict
    get_category_breakdown() -> list[dict]

artifacts.py (120 lines) -- Run artifact management:

class RunArtifacts:
    log_round(dict) -> None          # stream to rounds.jsonl
    finalize(logger, elo, run_ids)   # produce summary.json, leaderboard.md, charts

Strengths

1. Separation of concerns: The controller handles orchestration, the Elo system handles ratings, the logger handles persistence, and the artifacts handle output generation. Each has a single responsibility.

2. SQLite for round data: Using SQLite as the backing store for round logs is a good choice. It handles concurrent writes, supports complex queries (the attack success matrix and category breakdown are SQL aggregations), and the database file is a single portable artifact.

3. Artifact streaming: The RunArtifacts class streams each round to a rounds.jsonl file in real time, so partial results are available even if the run is interrupted. The incomplete run 9d160f59 demonstrates this -- it has rounds.jsonl data despite not producing a summary.json.

4. update_elo=False flag: The controller's run_round method accepts an update_elo parameter that lets multi-turn experiments score an episode of several turns as a single Elo game. This is a clean solution to the problem of inflating defender ratings with intermediate failures.

5. Seed exclusion with reset: The tournament loop tracks used_seeds per pair and resets when exhausted. This prevents the same seed from appearing twice in a pair while ensuring all seeds are used eventually.

6. Rich progress display: The controller uses the rich library for progress bars, tables, and colored output. The output is clean and informative.

Weaknesses

1. RoundResult.to_log_dict uses time.time(): The timestamp in to_log_dict is time.time() (Unix epoch float). This is fine for ordering but loses timezone information and is not human-readable. The artifact system uses ISO 8601 timestamps in meta.json, creating an inconsistency.

2. No transaction safety in logger: The ArenaLogger.log_round() method appears to do individual INSERT statements without explicit transaction management. Under high concurrency (not currently an issue, but worth noting), this could cause locking issues.

3. EloSystem stores history as list of floats: The EloRecord.history field is list[float] storing the rating before each update. This is appended to with [current_rating] when reading trajectories. The to_dict/from_dict serialization round-trips this correctly, but the history list grows without bound during long tournaments.

4. Config uses module-level constants: config.py loads .env at import time and creates module-level constants. This is simple but means configuration cannot be changed at runtime without reimporting. For a research tool this is fine; for a production system it would be limiting.

5. _snapshot_elo accesses private attributes: The controller's _snapshot_elo method accesses self.elo._attackers and self.elo._defenders directly, breaking encapsulation. A for_each_record() iterator or all_records() method on EloSystem would be cleaner.

---

3. Experiment Layer (experiments/)

transferability.py (145 lines), category_breakdown.py (120 lines), adaptive_vs_static.py (200 lines)

Strengths

1. Consistent experiment interface: All three experiments follow the same pattern: __init__ with optional parameters and defaults, run() as the entry point, _analyze() for post-processing, _print_results() for display. This makes adding new experiments straightforward.

2. Paired comparison in adaptive experiment: The adaptive vs static experiment correctly samples ONE set of seeds and uses them for both arms. This controls for seed difficulty, making the comparison fair. The two-proportion z-test is the correct statistical test for this design.

3. Wilson intervals in transferability: The transferability experiment computes 95% Wilson confidence intervals per attacker-defender pair. This is more appropriate than normal approximation for the small sample sizes typical in these experiments.

4. Blind spot identification: The category breakdown experiment automatically identifies defender blind spots by sorting categories by attack success rate. This produces actionable output: "Defender X is weakest against category Y."

Weaknesses

1. No experiment isolation: All three experiments share the same ArenaLogger instance by default, meaning their round data is interleaved in the same SQLite database. The experiment_tag field distinguishes them in queries, but this relies on correct tagging.

2. Category breakdown creates separate EloSystem per category: This is intentional (to avoid cross-category Elo contamination) but means ratings are not comparable across categories. A shared Elo system with category tagging would provide cross-category insights.

3. Hardcoded default participants: Each experiment has hardcoded default attacker/defender lists. For example, TransferabilityExperiment defaults to 5 template attackers and 2 defenders. This is reasonable for a research tool but limits reusability.

---

4. Dashboard (dashboard/charts.py)

charts.py (165 lines) -- Six Plotly chart builders:

Function	Chart Type	Purpose
elo_trajectory_chart	Line	Elo ratings over time
attack_success_heatmap	Heatmap	Attacker x Defender success matrix
category_bar_chart	Grouped bar	Per-category metrics
elo_leaderboard_bar	Horizontal bar	Elo rankings
severity_distribution	Donut pie	Harm severity distribution
attacker_win_rate_over_time	Line (rolling)	10-round rolling win rate

Strengths

• All charts use plotly_dark template (matches project's Anthropic dark mode preference)
• Functions are stateless and composable
• Graceful handling of empty data (returns empty figure with title)

Weaknesses

• attacker_win_rate_over_time imports numpy inside the function body (should be at module level)
• No chart validation or error handling for malformed data

---

Test Coverage

The test suite in tests/ has 8 files covering all major components:

Test File	Lines	Coverage
test_elo.py	57	Elo math, symmetry, zero-sum, serialization
test_attacker.py	41	Template formatting, registry, adaptive reset
test_defender.py	41	Keyword matching, regression tests, min_hits
test_judge.py	113	Outcome derivation, JSON parsing, heuristic, LLM fallback
test_logger.py	96	SQLite round-trip, migration, experiment lifecycle
test_artifacts.py	78	Streaming, finalize, graceful degradation
test_experiments.py	35	z-test, Wilson interval edge cases
conftest.py	6	sys.path setup

Total test code: ~467 lines.

Test Quality

1. Regression tests in test_defender.py: Tests specifically verify that "stalkerware" (not "stalkerwear") is matched, and that single keyword hits flag (regression from a ratio-threshold version). This shows the tests were written in response to real bugs.

2. Parametrized test in test_judge.py: The _derive_outcomes test uses @pytest.mark.parametrize for all 4 combinations of harmful/benign x flagged/unflagged. This is thorough.

3. Edge case testing in test_experiments.py: Tests verify that empty samples produce safe results (z=0, p=1), equal proportions are not significant, and large differences are significant. Wilson interval tests verify bounds are valid and tighter with more data.

4. Mock-based judge tests: The judge tests verify that LLM failures gracefully fall back to heuristic mode, and that the model's own attack_success field is ignored (only policy_violation counts).

Coverage Gaps

1. No integration tests: No test runs a full tournament round through the controller. All tests are unit-level.

2. No test for OllamaAttacker or OpenAIAttacker: The API-based attackers have no tests (likely because they require running services).

3. No test for PromptGuardDefender: The HuggingFace-based defender is untested (requires torch installation).

4. No test for dashboard charts: The chart functions are untested.

---

Configuration Management

config.py uses python-dotenv to load .env files with sensible defaults:

OPENAI_API_KEY = os.getenv("OPENAI_API_KEY", "")
JUDGE_MODEL = os.getenv("JUDGE_MODEL", "claude-haiku-4-5-20251001")
DEFAULT_ROUNDS = int(os.getenv("DEFAULT_ROUNDS", "50"))
ELO_K_FACTOR = float(os.getenv("ELO_K_FACTOR", "32"))

This is clean and follows the 12-factor app pattern. The defaults allow the system to run without any environment configuration (heuristic judge mode, template attackers only).

---

Dependency Analysis

requirements.txt lists 9 required packages and 3 optional:

Required: openai, anthropic, requests, python-dotenv, streamlit, plotly, pandas, numpy, rich, pytest

Optional: transformers, torch (for PromptGuard), datasets (for HuggingFace downloads)

The dependency footprint is reasonable. The core system (template attacks + keyword defender + heuristic judge) needs only requests, rich, and python-dotenv. The heavier dependencies (openai, anthropic, plotly, pandas) are needed for the full feature set.

---

Summary Assessment

Dimension	Rating	Notes
Architecture	Strong	Clean separation of agents/arena/experiments
Test coverage	Good	All core logic tested; gaps in integration and API-based components
Code quality	Good	Consistent style, clear dataclasses, explicit error handling
Documentation	Moderate	Module docstrings are excellent; inline comments are sparse
Configuration	Good	12-factor pattern with sensible defaults
Extensibility	Good	New attackers/defenders are easy to add via registry
Error handling	Good	Lazy initialization with clear error messages; judge has retry + fallback
Performance	Adequate	Synchronous execution; could benefit from async for API-based components

Recommended Improvements

1. Add __init__.py files to make the project a proper Python package
2. Convert defender registry to factories for consistency with attacker registry
3. Add integration tests that run a small tournament through the controller
4. Add async support for API-based attackers and defenders
5. Make keyword rules configurable via YAML or JSON
6. Add rate limiting for API calls in high-volume experiments
7. Test the dashboard chart functions with mock data