WikifitaGitHub live67e8de5
outro · red-team-arena/red-team-arena-dashboard

Arena Dashboard and Visualization System

Streamlit dashboard with 5 tabs, Plotly charts, and the RunArtifacts streaming system for live tournament monitoring

Baixar raw

Arena Dashboard and Visualization System

Overview

The Red Team Arena includes a Streamlit dashboard (dashboard/app.py) and a Plotly chart library (dashboard/charts.py) for visualizing tournament and experiment results. Additionally, the arena/artifacts.py module generates standalone HTML artifacts for every run, enabling offline analysis without running the dashboard.

The visualization system operates at two levels:

  1. Live artifacts: Each tournament/experiment writes a timestamped folder with streaming rounds.jsonl and final summary.json, leaderboard.md, and interactive Plotly HTML charts
  2. Streamlit dashboard: A persistent web app that reads from the SQLite database and renders 5 interactive tabs

Streamlit Dashboard

Launch: streamlit run dashboard/app.py (opens http://localhost:8501)

Page Configuration

  • Layout: Wide mode with expanded sidebar
  • Theme: Dark mode (Catppuccin-inspired CSS with #1e1e2e background, #cba6f7 accent)
  • Caching: @st.cache_resource on the ArenaLogger to avoid re-initializing SQLite on every rerun

Sidebar

The sidebar provides:

  1. Run selector: Dropdown populated from experiments table in SQLite. Options include individual runs and an "All runs (aggregated)" option
  2. Links: Quick links to GitHub, HarmBench, JailbreakBench

Metric Cards

Five top-level metrics are displayed as styled cards:

MetricSourceFormat
Total RoundsCOUNT(*) from rounds tableComma-separated integer
Attack Success RateAVG(judge_attack_success)Percentage
Defender AccuracyAVG(judge_defender_correct)Percentage
Avg Harm SeverityAVG(judge_harm_severity)X.XX/5
Avg Judge ConfidenceAVG(judge_confidence)X.XX

The 5 Tabs

Tab 1: Leaderboard

Visualizations:

  • Attacker Elo bar chart: Horizontal bar chart, red/orange color, sorted by rating ascending
  • Defender Elo bar chart: Horizontal bar chart, green color, sorted by rating ascending
  • Elo trajectory line chart: Multi-line time series showing all agents' ratings over time, with entity type encoded as line dash (solid for attackers, dashed for defenders)

Data source: Elo ratings are derived from the latest round data in the database (attacker_elo_after, defender_elo_after columns).

Tab 2: Heatmap

Visualization: Attack success rate heatmap -- rows are attackers, columns are defenders, cell values are the fraction of rounds where the attacker bypassed the defender.

Color scale: RdYlGn_r (reversed Red-Yellow-Green) -- red cells mean high attack success (bad for defender), green cells mean low attack success (good for defender).

Data source: get_attack_success_matrix() query that groups by attacker_id and defender_id, computing AVG(judge_attack_success).

Tab 3: Categories

Visualizations:

  • Grouped bar chart: Attack success rate, defender accuracy, or average severity broken down by harm category. Selector dropdown lets the user choose the metric.
  • Raw data table: Expandable pandas DataFrame showing all category breakdown data with formatted percentages.

Data source: get_category_breakdown() query that groups by seed_category, attacker_id, defender_id.

Tab 4: Round Log

The most detailed tab. Provides:

  1. Filter controls: Multi-select for attacker, defender, and category. Checkbox for "successful attacks only".
  2. Severity distribution pie chart: Donut chart showing the distribution of harm severity levels (1-5) in the filtered rounds.
  3. Rolling win rate chart: 10-round rolling average of attack success for the first selected attacker.
  4. Round table: DataFrame with columns: Round, Attacker, Defender, Category, Atk Win (checkmark/cross), Severity, Def Flagged (flag/dash), Reasoning.
  5. Round inspector: Dropdown to select a specific round and view the full attack prompt, judge reasoning, policy violation, harm severity, and defender confidence.

Tab 5: Experiments

Data source: get_all_run_ids() query on the experiments table.

Displays a table of all recorded experiments with: Run ID, Type, Started timestamp, Total Rounds. This is a metadata view; the actual experiment results are in the artifact folders.

Plotly Chart Library

dashboard/charts.py provides six reusable chart builder functions, all returning plotly.graph_objects.Figure objects with the plotly_dark template:

elo_trajectory_chart(trajectories)

  • Type: Multi-line time series
  • X: Timestamp (converted from epoch seconds to datetime)
  • Y: Elo rating
  • Color: Entity ID (agent name)
  • Dash: Entity type (attacker=solid, defender=dashed)
  • Features: Unified hover mode, legend with agent names

attack_success_heatmap(matrix)

  • Type: Heatmap (go.Heatmap)
  • X: Defender IDs
  • Y: Attacker IDs
  • Values: Success rates (0.0-1.0)
  • Color scale: RdYlGn_r (reversed)
  • Features: Text overlay showing percentage values, hover with exact rates

category_bar_chart(breakdown, metric)

  • Type: Grouped bar chart (px.bar)
  • X: Harm category
  • Y: Selected metric (attack_success_rate, defender_accuracy, or avg_severity)
  • Color: Defender ID
  • Features: -35 degree X-axis labels for readability

elo_leaderboard_bar(leaderboard, entity_type)

  • Type: Horizontal bar chart (go.Bar)
  • X: Rating
  • Y: Display name
  • Color: Red-orange for attackers, medium sea green for defenders
  • Features: Rating values as text outside bars

severity_distribution(rounds)

  • Type: Donut pie chart (go.Pie with hole=0.4)
  • Labels: Minimal, Low, Medium, High, Critical
  • Colors: Green(1) -> Yellow(2) -> Orange(3) -> Red(4) -> Purple(5)
  • Data source: Counter of judge_harm_severity values

attacker_win_rate_over_time(rounds, attacker_id)

  • Type: Line chart with markers
  • X: Round number
  • Y: 10-round rolling average of attack success
  • Features: min_periods=1 for the rolling window so the chart starts from round 1

RunArtifacts System

arena/artifacts.py implements a per-run artifact writer that creates a self-contained folder for every tournament or experiment.

Folder Structure

runs/20260611_192241_tournament_6ed3ad0a/
  meta.json          -- Run config + agents (written at start)
  rounds.jsonl       -- One JSON line per round (streamed live)
  summary.json       -- Aggregate stats (written at finalize)
  leaderboard.md     -- Human-readable Elo table
  report.html        -- Index linking the charts
  chart_*.html       -- Interactive Plotly visualizations

Live Streaming: rounds.jsonl

The log_round() method appends one JSON line per round and calls flush() immediately. This means:

  • rounds.jsonl is always consistent, even if the process crashes mid-run
  • You can tail -f rounds.jsonl while a tournament is running to watch results in real-time
  • Each line contains the full round data: attacker/defender IDs, attack prompt, defender verdict, judge verdict, Elo before/after, template used, turn number

Finalization

When finalize() is called (at the end of a tournament or experiment):

  1. summary.json: Writes aggregate stats (total rounds, attack success rate, severity distribution, judge modes, full Elo leaderboard)
  2. leaderboard.md: Generates a markdown table with attacker and defender Elo ratings
  3. Charts: Calls dashboard/charts.py functions to generate HTML files:
    • chart_elo_trajectory.html
    • chart_attack_heatmap.html
    • chart_category_breakdown.html
    • chart_severity.html
    • chart_attacker_elo.html
    • chart_defender_elo.html
  4. report.html: Generates an index HTML page linking all charts with a dark-themed design

Graceful Degradation

If plotly or pandas are not installed, chart generation is skipped and a CHARTS_SKIPPED.txt file is written explaining the situation. The JSON and markdown artifacts are always written regardless.

Artifact Bundle for Experiments

Experiments use a replay pattern: they stream to SQLite during execution, then run_experiments.py replays the logged rounds through a RunArtifacts writer to populate the standard folder structure. The results.json file is added alongside the standard artifacts.

SQLite Database Schema

The arena.db SQLite database uses WAL mode and contains three tables:

rounds

One row per arena round with 28 columns including:

  • Run metadata: run_id, experiment_tag, timestamp, round_number
  • Agent IDs: attacker_id, defender_id
  • Seed info: seed_prompt_id, seed_category, seed_severity
  • Attack: attack_prompt, template_used
  • Defense: defender_flagged, defender_confidence, defender_latency_ms
  • Judge: policy_violation, harm_severity, attack_success, defender_correct, harm_category, reasoning, confidence, judge_mode, judge_latency_ms
  • Elo: attacker_elo_before, attacker_elo_after, defender_elo_before, defender_elo_after
  • Metadata: turn_number, metadata (JSON)

Indexes: run_id, attacker_id, defender_id, seed_category, judge_attack_success

elo_snapshots

Periodic snapshots for trajectory analysis:

  • run_id, timestamp, entity_id, entity_type, rating, games

experiments

High-level experiment metadata:

  • run_id (unique), experiment_type, config_json, started_at, finished_at, total_rounds, notes

Migration support: The logger checks for the judge_mode column on initialization and adds it via ALTER TABLE if missing, supporting databases created before the judge mode feature was added.

Standalone HTML Artifacts

The report.html generated by RunArtifacts is a self-contained dark-themed page:

<style>
  body { font-family: system-ui, sans-serif; background:#111; color:#eee; }
  a { color:#ff7a59; }
  code { background:#222; padding:2px 6px; border-radius:4px; }
</style>

Each chart is an interactive Plotly HTML file loaded via CDN (plotly.js from jsdelivr). The charts support zoom, pan, hover tooltips, and can be opened in any browser without running the Streamlit app.

Cross-References