--- name: red-team-arena-dashboard type: reference title: "Arena Dashboard and Visualization System" description: "Streamlit dashboard with 5 tabs, Plotly charts, and the RunArtifacts streaming system for live tournament monitoring" tags: [ai-safety, dashboard, streamlit, plotly, visualization, artifacts, sqlite, reference] timestamp: 2026-07-21 --- # Arena Dashboard and Visualization System ## Overview The Red Team Arena includes a Streamlit dashboard (`dashboard/app.py`) and a Plotly chart library (`dashboard/charts.py`) for visualizing tournament and experiment results. Additionally, the `arena/artifacts.py` module generates standalone HTML artifacts for every run, enabling offline analysis without running the dashboard. The visualization system operates at two levels: 1. **Live artifacts**: Each tournament/experiment writes a timestamped folder with streaming `rounds.jsonl` and final `summary.json`, `leaderboard.md`, and interactive Plotly HTML charts 2. **Streamlit dashboard**: A persistent web app that reads from the SQLite database and renders 5 interactive tabs --- ## Streamlit Dashboard **Launch**: `streamlit run dashboard/app.py` (opens `http://localhost:8501`) ### Page Configuration - **Layout**: Wide mode with expanded sidebar - **Theme**: Dark mode (Catppuccin-inspired CSS with `#1e1e2e` background, `#cba6f7` accent) - **Caching**: `@st.cache_resource` on the ArenaLogger to avoid re-initializing SQLite on every rerun ### Sidebar The sidebar provides: 1. **Run selector**: Dropdown populated from `experiments` table in SQLite. Options include individual runs and an "All runs (aggregated)" option 2. **Links**: Quick links to GitHub, HarmBench, JailbreakBench ### Metric Cards Five top-level metrics are displayed as styled cards: | Metric | Source | Format | |--------|--------|--------| | Total Rounds | `COUNT(*)` from rounds table | Comma-separated integer | | Attack Success Rate | `AVG(judge_attack_success)` | Percentage | | Defender Accuracy | `AVG(judge_defender_correct)` | Percentage | | Avg Harm Severity | `AVG(judge_harm_severity)` | X.XX/5 | | Avg Judge Confidence | `AVG(judge_confidence)` | X.XX | ### The 5 Tabs #### Tab 1: Leaderboard **Visualizations**: - **Attacker Elo bar chart**: Horizontal bar chart, red/orange color, sorted by rating ascending - **Defender Elo bar chart**: Horizontal bar chart, green color, sorted by rating ascending - **Elo trajectory line chart**: Multi-line time series showing all agents' ratings over time, with entity type encoded as line dash (solid for attackers, dashed for defenders) **Data source**: Elo ratings are derived from the latest round data in the database (attacker_elo_after, defender_elo_after columns). #### Tab 2: Heatmap **Visualization**: Attack success rate heatmap -- rows are attackers, columns are defenders, cell values are the fraction of rounds where the attacker bypassed the defender. **Color scale**: `RdYlGn_r` (reversed Red-Yellow-Green) -- red cells mean high attack success (bad for defender), green cells mean low attack success (good for defender). **Data source**: `get_attack_success_matrix()` query that groups by attacker_id and defender_id, computing `AVG(judge_attack_success)`. #### Tab 3: Categories **Visualizations**: - **Grouped bar chart**: Attack success rate, defender accuracy, or average severity broken down by harm category. Selector dropdown lets the user choose the metric. - **Raw data table**: Expandable pandas DataFrame showing all category breakdown data with formatted percentages. **Data source**: `get_category_breakdown()` query that groups by `seed_category`, `attacker_id`, `defender_id`. #### Tab 4: Round Log **The most detailed tab**. Provides: 1. **Filter controls**: Multi-select for attacker, defender, and category. Checkbox for "successful attacks only". 2. **Severity distribution pie chart**: Donut chart showing the distribution of harm severity levels (1-5) in the filtered rounds. 3. **Rolling win rate chart**: 10-round rolling average of attack success for the first selected attacker. 4. **Round table**: DataFrame with columns: Round, Attacker, Defender, Category, Atk Win (checkmark/cross), Severity, Def Flagged (flag/dash), Reasoning. 5. **Round inspector**: Dropdown to select a specific round and view the full attack prompt, judge reasoning, policy violation, harm severity, and defender confidence. #### Tab 5: Experiments **Data source**: `get_all_run_ids()` query on the `experiments` table. Displays a table of all recorded experiments with: Run ID, Type, Started timestamp, Total Rounds. This is a metadata view; the actual experiment results are in the artifact folders. --- ## Plotly Chart Library `dashboard/charts.py` provides six reusable chart builder functions, all returning `plotly.graph_objects.Figure` objects with the `plotly_dark` template: ### `elo_trajectory_chart(trajectories)` - **Type**: Multi-line time series - **X**: Timestamp (converted from epoch seconds to datetime) - **Y**: Elo rating - **Color**: Entity ID (agent name) - **Dash**: Entity type (attacker=solid, defender=dashed) - **Features**: Unified hover mode, legend with agent names ### `attack_success_heatmap(matrix)` - **Type**: Heatmap (`go.Heatmap`) - **X**: Defender IDs - **Y**: Attacker IDs - **Values**: Success rates (0.0-1.0) - **Color scale**: `RdYlGn_r` (reversed) - **Features**: Text overlay showing percentage values, hover with exact rates ### `category_bar_chart(breakdown, metric)` - **Type**: Grouped bar chart (`px.bar`) - **X**: Harm category - **Y**: Selected metric (attack_success_rate, defender_accuracy, or avg_severity) - **Color**: Defender ID - **Features**: -35 degree X-axis labels for readability ### `elo_leaderboard_bar(leaderboard, entity_type)` - **Type**: Horizontal bar chart (`go.Bar`) - **X**: Rating - **Y**: Display name - **Color**: Red-orange for attackers, medium sea green for defenders - **Features**: Rating values as text outside bars ### `severity_distribution(rounds)` - **Type**: Donut pie chart (`go.Pie` with `hole=0.4`) - **Labels**: Minimal, Low, Medium, High, Critical - **Colors**: Green(1) -> Yellow(2) -> Orange(3) -> Red(4) -> Purple(5) - **Data source**: Counter of `judge_harm_severity` values ### `attacker_win_rate_over_time(rounds, attacker_id)` - **Type**: Line chart with markers - **X**: Round number - **Y**: 10-round rolling average of attack success - **Features**: `min_periods=1` for the rolling window so the chart starts from round 1 --- ## RunArtifacts System `arena/artifacts.py` implements a per-run artifact writer that creates a self-contained folder for every tournament or experiment. ### Folder Structure ``` runs/20260611_192241_tournament_6ed3ad0a/ meta.json -- Run config + agents (written at start) rounds.jsonl -- One JSON line per round (streamed live) summary.json -- Aggregate stats (written at finalize) leaderboard.md -- Human-readable Elo table report.html -- Index linking the charts chart_*.html -- Interactive Plotly visualizations ``` ### Live Streaming: rounds.jsonl The `log_round()` method appends one JSON line per round and calls `flush()` immediately. This means: - `rounds.jsonl` is always consistent, even if the process crashes mid-run - You can `tail -f rounds.jsonl` while a tournament is running to watch results in real-time - Each line contains the full round data: attacker/defender IDs, attack prompt, defender verdict, judge verdict, Elo before/after, template used, turn number ### Finalization When `finalize()` is called (at the end of a tournament or experiment): 1. **summary.json**: Writes aggregate stats (total rounds, attack success rate, severity distribution, judge modes, full Elo leaderboard) 2. **leaderboard.md**: Generates a markdown table with attacker and defender Elo ratings 3. **Charts**: Calls `dashboard/charts.py` functions to generate HTML files: - `chart_elo_trajectory.html` - `chart_attack_heatmap.html` - `chart_category_breakdown.html` - `chart_severity.html` - `chart_attacker_elo.html` - `chart_defender_elo.html` 4. **report.html**: Generates an index HTML page linking all charts with a dark-themed design ### Graceful Degradation If `plotly` or `pandas` are not installed, chart generation is skipped and a `CHARTS_SKIPPED.txt` file is written explaining the situation. The JSON and markdown artifacts are always written regardless. ### Artifact Bundle for Experiments Experiments use a replay pattern: they stream to SQLite during execution, then `run_experiments.py` replays the logged rounds through a `RunArtifacts` writer to populate the standard folder structure. The `results.json` file is added alongside the standard artifacts. --- ## SQLite Database Schema The `arena.db` SQLite database uses WAL mode and contains three tables: ### `rounds` One row per arena round with 28 columns including: - Run metadata: `run_id`, `experiment_tag`, `timestamp`, `round_number` - Agent IDs: `attacker_id`, `defender_id` - Seed info: `seed_prompt_id`, `seed_category`, `seed_severity` - Attack: `attack_prompt`, `template_used` - Defense: `defender_flagged`, `defender_confidence`, `defender_latency_ms` - Judge: `policy_violation`, `harm_severity`, `attack_success`, `defender_correct`, `harm_category`, `reasoning`, `confidence`, `judge_mode`, `judge_latency_ms` - Elo: `attacker_elo_before`, `attacker_elo_after`, `defender_elo_before`, `defender_elo_after` - Metadata: `turn_number`, `metadata` (JSON) **Indexes**: `run_id`, `attacker_id`, `defender_id`, `seed_category`, `judge_attack_success` ### `elo_snapshots` Periodic snapshots for trajectory analysis: - `run_id`, `timestamp`, `entity_id`, `entity_type`, `rating`, `games` ### `experiments` High-level experiment metadata: - `run_id` (unique), `experiment_type`, `config_json`, `started_at`, `finished_at`, `total_rounds`, `notes` **Migration support**: The logger checks for the `judge_mode` column on initialization and adds it via `ALTER TABLE` if missing, supporting databases created before the judge mode feature was added. --- ## Standalone HTML Artifacts The `report.html` generated by `RunArtifacts` is a self-contained dark-themed page: ```html ``` Each chart is an interactive Plotly HTML file loaded via CDN (`plotly.js` from jsdelivr). The charts support zoom, pan, hover tooltips, and can be opened in any browser without running the Streamlit app. ## Cross-References - [[red-team-arena]] -- Full system overview - [[red-team-arena-agents]] -- Agents whose data feeds the dashboard - [[red-team-arena-elo]] -- Elo system that produces leaderboard data - [[red-team-arena-experiments]] -- Experiments that produce structured results