Rendering Pipeline Deep-Dive

The Wikifita Atlas renders wiki pages from raw markdown to sanitized HTML using a 14-plugin unified pipeline. This pipeline runs server-side on every request (the site is fully force-dynamic) and produces the final HTML that browsers and agents consume.

The rendering function renderMarkdown lives in lib/markdown.ts (lines 290-309). It operates on page.body -- the markdown content after frontmatter removal -- not on the raw file.

Pipeline Architecture

flowchart TD
    A["page.body (markdown string)"] --> B["1. remarkParse"]
    B --> C["2. remarkGfm"]
    C --> D["3. remarkMath"]
    D --> E["4. remarkMermaidGuard"]
    E --> F["5. remarkWikiLinks"]
    F --> G["6. remarkCanonicalLinks"]
    G --> H["7. remarkRehype (MDAST -> HAST)"]
    H --> I["8. rehypeRaw"]
    I --> J["9. rehypeSanitize"]
    J --> K["10. rehypeWikiLinkClasses"]
    K --> L["11. rehypeMediaEmbeds"]
    L --> M["12. rehypeKatex"]
    M --> N["13. rehypeHighlight"]
    N --> O["14. rehypeStringify"]
    O --> P["HTML string"]

    style B fill:#2a2a3a
    style C fill:#2a2a3a
    style D fill:#2a2a3a
    style E fill:#2a2a3a
    style F fill:#2a2a3a
    style G fill:#2a2a3a
    style H fill:#2a2a3a
    style I fill:#2a2a3a
    style J fill:#2a2a3a
    style K fill:#2a2a3a
    style L fill:#2a2a3a
    style M fill:#2a2a3a
    style N fill:#2a2a3a
    style O fill:#2a2a3a

The pipeline operates in two phases:

Remark phase (plugins 1-6): Transforms the Markdown AST (MDAST)
Rehype phase (plugins 7-14): Converts to HTML AST (HAST), sanitizes, and serializes

Plugin 1: remarkParse

Package: remark-parse

Converts the raw markdown string into a Markdown Abstract Syntax Tree (MDAST). This is the foundation -- every subsequent plugin operates on this tree.

What it produces

{
  "type": "root",
  "children": [
    { "type": "heading", "depth": 1, "children": [{ "type": "text", "value": "Title" }] },
    { "type": "paragraph", "children": [{ "type": "text", "value": "Body text." }] },
    { "type": "code", "lang": "python", "value": "print('hello')" },
    { "type": "link", "url": "target.md", "children": [{ "type": "text", "value": "Link" }] }
  ]
}

Important properties

Code fences become code nodes with lang and value fields. Their content is not parsed as markdown.
HTML blocks become html nodes with raw string values. They are preserved for later processing by rehypeRaw.
The parser handles GFM extensions only after the next plugin adds them.

Plugin 2: remarkGfm

Package: remark-gfm

Enables GitHub Flavored Markdown extensions on top of the base CommonMark parser:

Extension	Syntax	AST Node
Tables	`\| col1 \| col2 \|`	`table` with `tableRow` and `tableCell`
Strikethrough	`~~deleted~~`	`delete` node
Task lists	`- [ ] item`	`listItem` with `checked` property
Autolinks	`https://example.com`	`link` node (without explicit syntax)
Footnotes	`[^1]` and `[^1]: text`	`footnoteReference` and `footnoteDefinition`

Impact on downstream plugins

Tables, strikethrough, and task lists are common in the wikifita corpus. The autolink extension means bare URLs in text are automatically converted to link nodes, which are then processed by remarkCanonicalLinks (plugin 6).

Plugin 3: remarkMath

Package: remark-math

Parses mathematical expressions into dedicated AST nodes:

Syntax	AST Node	Example
$inline$	`inlineMath`	$E = mc^2$
`$$block$$`	`math` (block)	`$$\int_0^1 f(x) dx$$`

The math nodes are preserved through the MDAST-to-HAST conversion and rendered to HTML by rehypeKatex (plugin 12). The parser handles nested delimiters and escaped dollar signs.

Plugin 4: remarkMermaidGuard

Custom plugin (lib/markdown.ts, lines 99-110)

Validates mermaid code blocks against a whitelist of recognized diagram types. Invalid blocks are demoted to plain text to prevent client-side rendering errors.

Recognized Diagram Types

graph, flowchart, sequenceDiagram, classDiagram, stateDiagram,
erDiagram, journey, gantt, pie, mindmap, timeline, gitGraph,
quadrantChart, xychart-beta, block-beta, packet-beta, sankey-beta

Validation Logic

function remarkMermaidGuard() {
  return (tree) => {
    visit(tree, 'code', (node) => {
      if (node.lang !== 'mermaid') return;

      const firstLine = node.value.trim().split('\n')[0].toLowerCase();
      const isValid = DIAGRAM_KEYWORDS.some(kw => firstLine.startsWith(kw));

      if (!isValid) {
        node.lang = 'text';
        node.meta = 'mermaid-invalid';
      }
    });
  };
}

Design Rationale

The mermaid client-side renderer (mermaid.render()) crashes on invalid syntax. By validating on the server, invalid blocks are gracefully degraded to <pre><code> elements rather than causing client-side JavaScript errors. The mermaid-invalid meta tag allows CSS styling to indicate the block was intended as a diagram.

Why Server-Side Validation, Server-Side Rendering Would Fail

Mermaid requires a DOM environment for rendering (it uses SVG generation internally). While jsdom could theoretically provide this, the server-side overhead is unnecessary. The validation happens server-side, but the actual SVG rendering happens client-side in the useMermaid hook.

Plugin 5: remarkWikiLinks

Custom plugin (lib/markdown.ts, lines 42-79)

Transforms WIKI_LINK_TARGET_LABEL text patterns into proper MDAST link nodes. This is the core wiki-link processor.

Processing Logic

For each text node in the AST, the plugin:

Scans for triple-dot patterns using regex
Looks up the target in page.links (pre-resolved during corpus building)
Replaces the text node with a sequence of: text-before, link, text-after

Link Construction

const link = {
  type: 'link',
  url: resolved ? `/wiki/${routePath}` : `#${encodedTarget}`,
  children: [{ type: 'text', value: label }],
  data: {
    hProperties: {
      className: resolved ? 'wiki-link' : 'wiki-link unresolved',
      'data-link-state': state  // "resolved", "missing", or "ambiguous"
    }
  }
};

CSS Classes

State	Class	Visual
Resolved	`wiki-link`	Standard link styling (accent color)
Missing	`wiki-link unresolved`	Dashed underline, muted color
Ambiguous	`wiki-link unresolved ambiguous`	Dashed underline, warning color

Plugin 6: remarkCanonicalLinks

Custom plugin (lib/markdown.ts, lines 81-97)

Rewrites standard markdown links and images to use canonical wiki routes.

For internal links (resolved)

Before: [text](target.md)
After:  [text](/wiki/target#fragment)

For missing/ambiguous links

Before: [text](missing-page)
After:  [text](#missing-page)
  + className: "unresolved"
  + data-link-state: "missing"

For external links

External links are left unchanged. They preserve their original URL (e.g., https://example.com).

Plugin 7: remarkRehype

Package: remark-rehype

Converts the MDAST (Markdown AST) to HAST (HTML AST). This is the critical boundary between the markdown processing domain and the HTML processing domain.

Configuration

remarkRehype({ allowDangerousHtml: true })

The allowDangerousHtml: true setting preserves raw HTML blocks from the markdown source as raw HAST nodes. Without this, HTML blocks in the wiki content would be escaped.

What changes

MDAST	HAST
`paragraph`	`p`
`heading`	`h1`, `h2`, ...
`link`	`a`
`emphasis`	`em`
`strong`	`strong`
`code`	`pre > code`
`inlineMath`	Preserved as custom node
`math`	Preserved as custom node
`html` (raw)	`raw` (string)

Plugin 8: rehypeRaw

Package: rehype-raw

Parses raw HTML strings embedded in the HAST into proper HAST elements. This converts <iframe>, <video>, and other HTML elements written directly in markdown into structured AST nodes.

Why this matters

The wikifita corpus uses raw HTML for media embeds:

<iframe src="https://www.youtube-nocookie.com/embed/ID" allowfullscreen></iframe>

Without rehypeRaw, this would remain as a raw string in the HAST. With it, the iframe becomes a proper element node with tag name, properties, and children -- enabling rehypeMediaEmbeds to process it.

Security note

This plugin operates BEFORE rehypeSanitize (plugin 9). All dangerous HTML is neutralized by the sanitizer, not by rehypeRaw itself.

Plugin 9: rehypeSanitize

Package: rehype-sanitize

The security boundary of the pipeline. All HTML output passes through a whitelist-based sanitizer that removes disallowed elements, attributes, and protocols.

Custom Schema

The Wikifita sanitizer extends rehype-sanitize's defaultSchema with wiki-specific allowances:

Allowed Attributes

{
  a: ['className', 'data-link-state', 'target', 'rel', 'href', 'title'],
  code: ['className'],
  input: ['type', 'checked', 'disabled'],
  pre: ['data-mermaid-source'],
  span: ['style'],
  img: ['src', 'alt', 'title', 'width', 'height', 'loading'],
  iframe: ['src', 'allow', 'allowfullscreen', 'loading', 'referrerpolicy'],
  video: ['src', 'controls', 'preload', 'poster', 'width', 'height'],
  audio: ['src', 'controls', 'preload'],
  source: ['src', 'type']
}

MathML Elements

The sanitizer adds all MathML tag names to the allowlist:

math, mi, mn, mo, ms, mtext, mspace, mrow, mfrac, msqrt,
msup, msub, msubsup, semantics, annotation

These are required for KaTeX's HTML output, which uses MathML elements for accessible math rendering.

Additional Elements

Element	Purpose
`figure`	Media embed wrapper
`figcaption`	Media embed caption
`input`	Task list checkboxes

Protocols

Only http:, https:, and mailto: protocols are allowed in href and src attributes. javascript: and other dangerous protocols are stripped.

What Gets Removed

<script> tags (not in allowlist)
onclick and other event handler attributes (not in allowlist)
style attributes on block elements (only allowed on span)
javascript: URLs in href/src
Any elements or attributes not explicitly whitelisted

Plugin 10: rehypeWikiLinkClasses

Custom plugin (lib/markdown.ts)

Adds the wiki-link CSS class to all <a> elements whose href starts with /wiki/. This enables consistent styling of internal wiki links regardless of how they were originally written (markdown link or wiki link).

function rehypeWikiLinkClasses() {
  return (tree) => {
    visit(tree, 'element', (node) => {
      if (node.tagName === 'a' && node.properties?.href?.startsWith('/wiki/')) {
        addClass(node, 'wiki-link');
      }
    });
  };
}

Plugin 11: rehypeMediaEmbeds

Custom plugin (lib/markdown.ts, lines 179-205)

Transforms <a> tags pointing to media URLs into semantic <figure> elements with appropriate media tags.

Transformation Rules

URL Classification	Output HTML
Image (`.png`, `.jpg`, etc.)	`<figure class="media-embed media-embed-image"><img loading="lazy" src="..." alt="..."/><figcaption>...</figcaption></figure>`
Video (`.mp4`, `.webm`, etc.)	`<figure class="media-embed media-embed-video"><video controls preload="metadata" src="...">...</video><figcaption>...</figcaption></figure>`
Audio (`.mp3`, `.ogg`, etc.)	`<figure class="media-embed media-embed-audio"><audio controls preload="metadata" src="...">...</audio><figcaption>...</figcaption></figure>`
YouTube/Vimeo embed	`<figure class="media-embed media-embed-frame"><iframe src="..." allowfullscreen loading="lazy" referrerpolicy="strict-origin-when-cross-origin">...</iframe><figcaption>...</figcaption></figure>`
Blocked (non-HTTPS)	`<a class="media-link media-link-blocked" data-media-state="remote_media_requires_https">...</a>`

Lazy Loading

All <img> and <iframe> elements receive loading="lazy" by default. This defers loading of off-screen media until the user scrolls near it, improving initial page load performance.

YouTube Privacy

YouTube URLs are rewritten to the privacy-enhanced youtube-nocookie.com domain:

https://youtube.com/watch?v=ID      -> https://www.youtube-nocookie.com/embed/ID
https://youtu.be/ID                 -> https://www.youtube-nocookie.com/embed/ID
https://youtube.com/shorts/ID       -> https://www.youtube-nocookie.com/embed/ID

Plugin 12: rehypeKatex

Package: rehype-katex

Renders math nodes (from remarkMath) to KaTeX HTML. This is a server-side rendering step -- the math is fully rendered as HTML with CSS, not as JavaScript.

Output

Inline math $E = mc^2$ becomes:

<span class="katex">
  <span class="katex-mathml"><math>...</math></span>
  <span class="katex-html"><span class="base">...</span></span>
</span>

Block math $$...$$ uses a <span class="katex-display"> wrapper for centered, display-style rendering.

CSS

The KaTeX CSS is loaded globally in layout.tsx. It provides the font metrics and layout rules for math elements.

Plugin 13: rehypeHighlight

Package: rehype-highlight

Applies syntax highlighting to fenced code blocks using highlight.js. The library auto-detects the language from the className attribute (set by the markdown parser from the code fence info string).

Theme

The site uses the github-dark theme, loaded globally in layout.tsx.

Output

A code block like:

```python
def hello():
    print("world")
```

Becomes:

<pre><code class="hljs language-python"><span class="hljs-keyword">def</span> <span class="hljs-title function_">hello</span>():
    <span class="hljs-built_in">print</span>(<span class="hljs-string">"world"</span>)</code></pre>

Auto-Detection

If no language is specified in the code fence, highlight.js attempts auto-detection. This is generally reliable for common languages but may produce incorrect results for ambiguous snippets.

Plugin 14: rehypeStringify

Package: rehype-stringify

Serializes the HAST (HTML AST) into a final HTML string. This is the output of the renderMarkdown function.

Configuration

rehypeStringify({ allowDangerousHtml: true })

The allowDangerousHtml setting ensures that any remaining raw HTML nodes in the HAST are serialized as-is rather than escaped.

Client-Side Rendering

Two rendering steps happen on the client, not in the server pipeline:

Mermaid Diagrams

Mermaid diagrams are NOT rendered on the server. The AtlasClient.tsx component's useMermaid hook (lines 696-725) handles client-side rendering:

// Dynamic import (client-side only)
const mermaid = await import('mermaid');

// Initialize with security settings
mermaid.initialize({
  startOnLoad: false,
  theme: 'dark',
  securityLevel: 'strict'
});

// Render each mermaid code block
for (const block of document.querySelectorAll('code.language-mermaid')) {
  const { svg } = await mermaid.render(uniqueId, block.textContent);
  block.parentElement.innerHTML = svg;
}

Security Level: strict

The securityLevel: "strict" setting prevents mermaid from:

Running JavaScript in diagrams
Loading external resources
Executing event handlers in SVG output

This is critical because mermaid diagrams are rendered from user-authored markdown content.

Server Validation + Client Rendering

The server-side remarkMermaidGuard (plugin 4) validates that mermaid blocks contain recognized diagram syntax. Invalid blocks are demoted to plain text. The client-side renderer only processes blocks with the language-mermaid class, so demoted blocks are silently skipped.

KaTeX (Server-Side)

Unlike mermaid, KaTeX is rendered server-side by rehypeKatex (plugin 12). The client only needs the KaTeX CSS for display. No JavaScript rendering is needed.

Sanitization Security Model

The rendering pipeline implements defense-in-depth against XSS attacks:

flowchart LR
    A[User markdown] --> B[remarkParse]
    B --> C[remarkRehype + rehypeRaw]
    C --> D[rehypeSanitize]
    D --> E[Safe HTML]

    F["<script>alert(1)</script>"] --> B
    B --> C
    C --> D
    D -->|"stripped"| G["(removed)"]

    H["<img src=x onerror=alert(1)>"] --> B
    B --> C
    C --> D
    D -->|"onerror removed"| I['<img src="x">']

Layers

remarkParse: Does not execute HTML. Parses it as raw strings.
rehypeRaw: Parses raw HTML strings into HAST elements. No execution.
rehypeSanitize: Whitelist-based. Only allowed elements, attributes, and protocols survive.
Client-side: Mermaid runs with securityLevel: "strict". No other dynamic content.

What Survives Sanitization

Safe HTML elements (p, div, span, table, img, video, audio, iframe with safe src)
KaTeX MathML elements
Wiki link attributes (data-link-state, className)
Checkbox inputs (for task lists)
style attribute only on <span> elements (for KaTeX and highlight.js)

What Does Not Survive

<script> tags
Event handler attributes (onclick, onerror, onload)
javascript: URLs
<form>, <input> (except checkbox), <select>, <textarea>
CSS expressions in style attributes
Any element or attribute not in the whitelist

Raw Content Preservation

A fundamental design principle: the rendering pipeline never modifies the source markdown. The page.raw field contains the original, byte-for-byte file content. The page.body field is the content after frontmatter removal (a substring, not a re-serialization). The HTML output is derived from body but does not replace it.

This means:

The raw markdown is the source of truth
HTML is always a derived view
Frontmatter is preserved verbatim
Whitespace, newlines, and formatting in the source are unchanged
The /raw/{path}.md endpoint serves page.raw directly

Performance Characteristics

Metric	Value	Notes
Pipeline setup	~1ms	Plugin chain creation (amortized)
Parse + transform	1-5ms	Depends on page length
Sanitize	0.5-2ms	Whitelist-based, O(n) in AST size
Stringify	0.5-1ms	HAST to HTML string
Total server render	3-10ms	Per page, per request

Since the site is force-dynamic, this rendering happens on every request. For the current corpus size (50-200 pages), this is well within Cloudflare Workers' time limits.

wikifita-site-pipeline -- The broader wiki processing pipeline
wikifita-site-graph-construction -- Link extraction that feeds wiki link resolution
wikifita-site-corpus-pipeline -- Corpus building where page.body is produced
wikifita-site-deployment -- Cloudflare Workers execution environment
wikifita-site-architecture -- Full technical architecture

Rendering Pipeline Deep-Dive

Pipeline Architecture

Plugin 1: remarkParse

What it produces

Important properties

Plugin 2: remarkGfm

Impact on downstream plugins

Plugin 3: remarkMath

Plugin 4: remarkMermaidGuard

Recognized Diagram Types

Validation Logic

Design Rationale

Why Server-Side Validation, Server-Side Rendering Would Fail

Plugin 5: remarkWikiLinks

Processing Logic

Link Construction

CSS Classes

Plugin 6: remarkCanonicalLinks

For internal links (resolved)

For missing/ambiguous links

For external links

Plugin 7: remarkRehype

Configuration

What changes

Plugin 8: rehypeRaw

Why this matters

Security note

Plugin 9: rehypeSanitize

Custom Schema

Allowed Attributes

MathML Elements

Additional Elements

Protocols

What Gets Removed

Plugin 10: rehypeWikiLinkClasses

Plugin 11: rehypeMediaEmbeds

Transformation Rules

Lazy Loading

YouTube Privacy

Plugin 12: rehypeKatex

Output

CSS

Plugin 13: rehypeHighlight

Theme

Output

Auto-Detection

Plugin 14: rehypeStringify

Configuration

Client-Side Rendering

Mermaid Diagrams

Security Level: strict

Server Validation + Client Rendering

KaTeX (Server-Side)

Sanitization Security Model

Layers

What Survives Sanitization

What Does Not Survive

Raw Content Preservation

Performance Characteristics

Related Pages