---
name: wikifita-site-pipeline
type: reference
title: "Wikifita Site Wiki Pipeline"
description: "How markdown files flow from the GitHub repository through frontmatter parsing, link resolution, PageRank computation, and markdown rendering into the final HTML served to browsers."
tags: [wikifita, pipeline, markdown, pagerank, corpus, wiki-processing]
timestamp: 2026-07-21
---

# Wikifita Site Wiki Pipeline

The wiki pipeline is the core data processing layer of the Wikifita Atlas. It transforms raw markdown files from a GitHub repository into a fully indexed, link-resolved, ranked corpus that powers search, graph visualization, and page rendering.

All pipeline logic lives in `lib/wiki.ts` (corpus building, search, path resolution) and `lib/markdown.ts` (HTML rendering).

---

## Overview

```mermaid
flowchart TD
    A[GitHub API / Snapshot] -->|RawWikiFile[]| B[normalizeRawFile]
    B --> C[isContentDocument filter]
    C -->|only .md / .mdx| D[parseFrontmatter]
    D -->|WikiFrontmatter + body| E[extractMediaReferences]
    E --> F[build base WikiPage]
    F --> G[resolvePageLinks]
    G --> H[computePagerank]
    H --> I[WikiCorpus]
    I --> J[renderMarkdown]
    I --> K[searchWiki]
    I --> L[toClientData]
    J -->|HTML string| M[SSR to Browser]
    K -->|WikiSearchResult[]| N[Search API]
    L -->|WikiClientData| O[D3 Graph + AtlasClient]
```

---

## Phase 1: Data Acquisition

### Live Source (GitHub)

When `WIKIFITA_SOURCE=github`, the pipeline loads the corpus from the GitHub REST API:

1. **Commit resolution**: `GET /repos/{owner}/{repo}/commits/{branch}` returns the current commit SHA and date.
2. **SHA deduplication**: If the SHA matches the in-memory `liveCache.key`, the cached corpus is returned immediately with no API calls.
3. **Tree traversal**: `GET /repos/{owner}/{repo}/git/trees/{sha}?recursive=1` returns all file entries. Entries are filtered to `.md` and `.mdx` files, excluding `AGENTS.md`.
4. **Blob loading**: Each blob is fetched via `GET /repos/{owner}/{repo}/git/blobs/{sha}`, decoded from base64, and cached in `blobCache` (a `Map<string, string>` keyed by blob SHA). Concurrency is limited to 8 simultaneous fetches via `mapLimit`.

### Snapshot Fallback

When the source is `snapshot` or GitHub fails with `WIKIFITA_FALLBACK=snapshot`, the pipeline reads from `data/wiki-snapshot.json`. This file is pre-built by `scripts/build-wiki-snapshot.mjs` and contains the same `RawWikiFile` shape (path, raw content, blobSha, byteLength).

### Caching Strategy

| Cache | Scope | Key | Invalidation |
|-------|-------|-----|-------------|
| `liveCache` | Module-level | `{repo}:{branch}:{sha}` | New commit SHA |
| `snapshotCache` | Module-level | `{commit}:{state}:{error}` | New snapshot |
| `blobCache` | Module-level | blob SHA | Never (within process lifetime) |
| `searchCache` | WeakMap on corpus | — | New corpus instance |

---

## Phase 2: Corpus Building (`buildCorpus`)

`buildCorpus(rawFiles, status)` is the central function. It takes an array of `RawWikiFile` objects and produces a `WikiCorpus` with fully resolved links, computed PageRank, and indexed pages.

### Step 2a: Normalization

Each `RawWikiFile` is normalized:
- Path is POSIX-normalized (backslashes to forward slashes, no leading `./` or `../`)
- `blobSha` defaults to SHA-256 of `raw` if not provided
- `byteLength` is computed from UTF-8 encoding of `raw`

Files that are not `.md` or `.mdx`, or whose path ends with `AGENTS.md`, are filtered out.

### Step 2b: Frontmatter Parsing (`parseFrontmatter`)

The parser extracts YAML frontmatter delimited by `---`:

```yaml
---
type: reference
title: "Page Title"
description: "One-line description"
tags: [tag1, tag2]
timestamp: 2026-07-21
custom_field: value
---
```

**Parsing rules:**
- The `---` delimiter must be the first line of the file
- YAML is parsed via the `yaml` library (`YAML.parseDocument`) with error collection
- The `okf` fields (`type`, `title`, `description`, `tags`, `timestamp`) are extracted into a typed structure
- All other fields go into `metadata` (a generic `Record<string, unknown>`)
- Parse errors are captured in `frontmatter.diagnostics` rather than crashing
- If no frontmatter exists, the entire file is treated as `body`
- The `body` is the raw content after the closing `---`, preserving all whitespace and formatting

**Important**: The `raw` field on the resulting `WikiPage` is the *original, unmodified* file content. The pipeline never re-serializes or re-encodes the source markdown.

### Step 2c: Page Construction

Each normalized file becomes a `WikiPage` with:

| Field | Source | Description |
|-------|--------|-------------|
| `id` | path without extension | Stable identifier |
| `routePath` | path without extension | Used in `/wiki/` URLs |
| `title` | frontmatter `title` or first `# heading` or filename | Human-readable name |
| `description` | frontmatter `description` | One-line summary |
| `type` | frontmatter `type` or `"page"` | OKF type |
| `category` | Computed from `type` + path prefix | One of: person, project, research, directive, memory, other |
| `tags` | frontmatter `tags` | String array |
| `excerpt` | First 240 chars of description or stripped body | For search results and suggestions |
| `hash` | SHA-256 of `raw` | Content integrity check |
| `media` | Extracted from markdown AST + HTML | `WikiMediaReference[]` |

**Category classification** (`categoryFor`):

| Category | Trigger (type or path prefix) |
|----------|-------------------------------|
| `person` | `type` contains `person`/`pessoa`/`contact`, or path starts with `pessoas/` |
| `directive` | `type` contains `directive`/`diretiva`, or path starts with `diretivas/` |
| `research` | `type` contains `research`/`pesquisa`, or path starts with `pesquisas/`, `kaggle/`, `fitdnuvo/`, `clickfix/` |
| `project` | `type` contains `project`/`projeto`, or path starts with `infraestrutura/`, `claude_desktop/`, `ai_modelos/`, `mcp/`, `hacks/` |
| `memory` | `type` contains `memory`/`memoria`/`memória`, or path starts with `memorias/` |
| `other` | Default fallback |

---

## Phase 3: Link Resolution

### Link Extraction

Links are extracted from the parsed AST using `unist-util-visit`:

1. **Markdown links and images**: standard markdown link and image syntax nodes from the remark AST
2. **Wiki links**: double-bracket patterns (target or target with label) found in raw text nodes
3. **HTML media references**: `<iframe>`, `<video>`, `<audio>`, `<source>`, `<img>` tags via regex

### Link Resolution (`resolveLinkTarget`)

Each extracted link target is resolved against the set of all page paths:

| Step | Action | Result |
|------|--------|--------|
| 1 | Check if external URL (`https://...`) | `state: "external"` |
| 2 | Fragment-only (`#section`) | `state: "resolved"`, destination = source page |
| 3 | Exact match in `allPaths` | `state: "resolved"` |
| 4 | With `.md`/`.mdx` extension appended | `state: "resolved"` |
| 5 | Relative path resolution (from source directory) | `state: "resolved"` |
| 6 | Slug match (case-insensitive, last path segment) | `state: "resolved"` if unique, `"ambiguous"` if multiple |
| 7 | No match found | `state: "missing"` |

### Backlink Computation

After all pages have their outlinks resolved, backlinks are computed as the reverse index: if page A links to page B, then B's `backlinks` includes A. This enables the "context rail" in the UI showing which pages reference the current document.

---

## Phase 4: PageRank

The corpus computes a standard PageRank with damping factor 0.85:

| Parameter | Value |
|-----------|-------|
| Damping factor | 0.85 |
| Tolerance | 1e-10 |
| Max iterations | 200 |

**Dangling node handling**: Pages with no outgoing links distribute their rank equally to all pages (standard PageRank treatment).

**Normalization**: Final ranks are normalized so they sum to 1.0 across the corpus.

PageRank is used for:
- Search result ranking (boost factor)
- Home page suggestions (`topSuggestedPages`)
- Node radius in the D3 graph visualization
- Tie-breaking in search results

---

## Phase 5: Media Reference Extraction

The `extractMediaReferences` function walks the markdown AST and HTML to find all media references:

| Media Kind | Detection | Allowed | Embed URL |
|-----------|-----------|---------|-----------|
| `image` | `.avif`, `.gif`, `.jpg`, `.jpeg`, `.png`, `.svg`, `.webp` extensions | Yes | — |
| `video` | `.m4v`, `.mov`, `.mp4`, `.webm` extensions | Yes | — |
| `audio` | `.flac`, `.m4a`, `.mp3`, `.ogg`, `.wav` extensions | Yes | — |
| `embed` | YouTube or Vimeo URLs | Yes | Transformed to `youtube-nocookie.com/embed/` or `player.vimeo.com/video/` |
| `link` | Non-HTTPS URLs | No | — |

**Security**: Only HTTPS URLs are allowed. HTTP URLs are marked as `allowed: false` with reason `remote_media_requires_https`. There is no local asset storage; all media is served from remote URLs.

YouTube URL transformation:
- `https://youtube.com/watch?v=ID` -> `https://www.youtube-nocookie.com/embed/ID`
- `https://youtu.be/ID` -> `https://www.youtube-nocookie.com/embed/ID`
- `https://youtube.com/shorts/ID` -> `https://www.youtube-nocookie.com/embed/ID`

---

## Phase 6: HTML Rendering (`renderMarkdown`)

The rendering pipeline transforms `page.body` (markdown without frontmatter) into sanitized HTML using a unified/remark/rehype pipeline:

```mermaid
flowchart LR
    A[page.body] --> B[remarkParse]
    B --> C[remarkGfm]
    C --> D[remarkMath]
    D --> E[remarkMermaidGuard]
    E --> F[remarkWikiLinks]
    F --> G[remarkCanonicalLinks]
    G --> H[remarkRehype]
    H --> I[rehypeRaw]
    I --> J[rehypeSanitize]
    J --> K[rehypeWikiLinkClasses]
    K --> L[rehypeMediaEmbeds]
    L --> M[rehypeKatex]
    M --> N[rehypeHighlight]
    N --> O[rehypeStringify]
    O --> P[HTML string]
```

### Plugin Details

| Plugin | Purpose |
|--------|---------|
| `remarkParse` | Parse markdown to MDAST |
| `remarkGfm` | GitHub Flavored Markdown (tables, strikethrough, autolinks) |
| `remarkMath` | Parse `$...$` and `$$...$$` math syntax |
| `remarkMermaidGuard` | Validate mermaid code blocks; demote invalid ones to plain text |
| `remarkWikiLinks` | Transform wiki-link syntax to `<a>` nodes with resolved hrefs |
| `remarkCanonicalLinks` | Rewrite markdown links to canonical `/wiki/...` paths |
| `remarkRehype` | Convert MDAST to HAST (allowing raw HTML) |
| `rehypeRaw` | Parse raw HTML embedded in markdown |
| `rehypeSanitize` | Whitelist-based HTML sanitization (extended schema for wiki-specific attributes) |
| `rehypeWikiLinkClasses` | Add `class="wiki-link"` to internal `/wiki/` links |
| `rehypeMediaEmbeds` | Convert media links to `<figure>` elements with `<img>`, `<video>`, `<audio>`, or `<iframe>` |
| `rehypeKatex` | Render math expressions to HTML with KaTeX |
| `rehypeHighlight` | Syntax highlighting for code blocks |
| `rehypeStringify` | Serialize HAST to HTML string |

### Wiki Link Rendering

Wiki links with display text are transformed to standard `<a>` tags:

- Resolved links get `class="wiki-link"` and href `/wiki/{routePath}`
- Missing links get `class="wiki-link unresolved"` and href `#target`
- Ambiguous links get `class="wiki-link unresolved ambiguous"` and href `#target`
- The `data-link-state` attribute carries the resolution state

### Media Embed Rendering

Media links are converted to semantic `<figure>` elements:

```html
<!-- Image -->
<figure class="media-embed media-embed-image">
  <img src="https://..." alt="label" loading="lazy" />
  <figcaption>label</figcaption>
</figure>

<!-- YouTube -->
<figure class="media-embed media-embed-frame">
  <iframe src="https://www.youtube-nocookie.com/embed/..." allowfullscreen loading="lazy" />
  <figcaption>label</figcaption>
</figure>

<!-- Blocked (HTTP) -->
<a class="media-link media-link-blocked" data-media-state="remote_media_requires_https">
```

### Sanitization Schema

The sanitization schema extends the default rehype-sanitize schema with:
- Custom `className` values for link states (`wiki-link`, `unresolved`, `missing`, `ambiguous`, `media-link-blocked`)
- `data-link-state`, `data-mermaid-source` attributes
- Math elements (`math`, `mi`, `mn`, `mo`, `ms`, `mtext`, `mfrac`, `msqrt`, `msup`, `msub`)
- Figure/figcaption elements
- Input elements with checkbox support

---

## Phase 7: Search Index

The search system uses a custom BM25 implementation with PageRank and graph-proximity boosting.

### Index Construction

The search index is built lazily and cached via `WeakMap` on the corpus:

1. Each page's searchable text is assembled: `title + description + tags + type + category + path + body`
2. Text is normalized: NFD decomposition, diacritic removal, lowercase (Portuguese-aware)
3. Tokens are extracted via regex: `[\p{L}\p{N}_-]+`
4. Document frequency and average length are computed for BM25 IDF

### BM25 Scoring

| Parameter | Value |
|-----------|-------|
| k1 | 1.35 |
| b | 0.72 |

### Score Composition

| Factor | Weight | Description |
|--------|--------|-------------|
| BM25 | 1.8x | Text relevance |
| Title boost | +4.0 (exact) / +1.5 (substring) | Query matches page title |
| PageRank boost | up to +1.25 | Proportional to normalized PageRank |
| Proximity boost | up to +1.25 | Graph distance from focus path (1.25 - distance * 0.35) |

### Proximity Boost

When a `focusPath` is provided (e.g., the currently viewed page), search results that are close to the focused page in the link graph receive a boost. Distance is computed via BFS on the undirected graph (outlinks + backlinks), capped at depth 3.

### Suppression

Pages with stems `claude`, `agents`, `index`, or filenames containing `prefill` or `template` are suppressed from search results and suggestions.

---

## Data Types Summary

### WikiCorpus

```typescript
type WikiCorpus = {
  status: WikiSourceStatus;      // Repository, branch, commit, source, state
  pages: WikiPage[];             // All processed pages
  pageByPath: Map<string, WikiPage>;  // Fast lookup by path
  links: WikiGraphLink[];        // All resolved links for D3 graph
};
```

### WikiPage

```typescript
type WikiPage = {
  id: string;                    // path without extension
  path: string;                  // Original file path
  routePath: string;             // Used in /wiki/ URLs
  rawUrl: string;                // /raw/{path}
  canonicalUrl: string;          // Full URL with site base
  raw: string;                   // Unmodified source markdown
  body: string;                  // Markdown without frontmatter
  frontmatter: WikiFrontmatter;  // Parsed OKF + metadata + diagnostics
  title: string;
  description?: string;
  type: string;
  category: WikiCategory;
  tags: string[];
  timestamp?: string;
  excerpt: string;               // First 240 chars for previews
  byteLength: number;
  blobSha: string;
  hash: string;                  // SHA-256 of raw
  links: WikiLink[];             // All extracted + resolved links
  media: WikiMediaReference[];   // All media references
  outlinks: string[];            // Resolved outgoing link paths
  backlinks: string[];           // Pages that link to this page
  pagerank: number;              // Normalized PageRank score
};
```

### WikiClientPage

The client-safe subset sent to the browser (excludes `raw`, `body`, and full `links`):

```typescript
type WikiClientPage = Omit<WikiPage, "raw" | "body" | "links"> & {
  links: Array<Pick<WikiLink, "target" | "label" | "fragment" | "destination" | "state">>;
};
```

---

## Path Resolution (`resolveWikiPath`)

When a user requests `/wiki/some/path`, the resolver attempts:

1. Exact match: `some/path` -> check `pageByPath`
2. With extension: `some/path.md`, `some/path.mdx`
3. Slug match: extract last segment, case-insensitive comparison
4. Unique alias: if exactly one page matches the slug
5. Ambiguous: multiple pages share the same slug -> `state: "ambiguous"`
6. Missing: no match found -> `state: "missing"` (renders 404)

---

## Related Pages

- [[wikifita-site-architecture]] -- Full technical architecture
- [[wikifita-site-llm-integration]] -- LLM endpoint design
- [[wikifita-site]] -- Project overview stub