Wikifita Site Wiki Pipeline

The wiki pipeline is the core data processing layer of the Wikifita Atlas. It transforms raw markdown files from a GitHub repository into a fully indexed, link-resolved, ranked corpus that powers search, graph visualization, and page rendering.

All pipeline logic lives in lib/wiki.ts (corpus building, search, path resolution) and lib/markdown.ts (HTML rendering).

Overview

flowchart TD
    A[GitHub API / Snapshot] -->|RawWikiFile[]| B[normalizeRawFile]
    B --> C[isContentDocument filter]
    C -->|only .md / .mdx| D[parseFrontmatter]
    D -->|WikiFrontmatter + body| E[extractMediaReferences]
    E --> F[build base WikiPage]
    F --> G[resolvePageLinks]
    G --> H[computePagerank]
    H --> I[WikiCorpus]
    I --> J[renderMarkdown]
    I --> K[searchWiki]
    I --> L[toClientData]
    J -->|HTML string| M[SSR to Browser]
    K -->|WikiSearchResult[]| N[Search API]
    L -->|WikiClientData| O[D3 Graph + AtlasClient]

Phase 1: Data Acquisition

Live Source (GitHub)

When WIKIFITA_SOURCE=github, the pipeline loads the corpus from the GitHub REST API:

Commit resolution: GET /repos/{owner}/{repo}/commits/{branch} returns the current commit SHA and date.
SHA deduplication: If the SHA matches the in-memory liveCache.key, the cached corpus is returned immediately with no API calls.
Tree traversal: GET /repos/{owner}/{repo}/git/trees/{sha}?recursive=1 returns all file entries. Entries are filtered to .md and .mdx files, excluding AGENTS.md.
Blob loading: Each blob is fetched via GET /repos/{owner}/{repo}/git/blobs/{sha}, decoded from base64, and cached in blobCache (a Map<string, string> keyed by blob SHA). Concurrency is limited to 8 simultaneous fetches via mapLimit.

Snapshot Fallback

When the source is snapshot or GitHub fails with WIKIFITA_FALLBACK=snapshot, the pipeline reads from data/wiki-snapshot.json. This file is pre-built by scripts/build-wiki-snapshot.mjs and contains the same RawWikiFile shape (path, raw content, blobSha, byteLength).

Caching Strategy

Cache	Scope	Key	Invalidation
`liveCache`	Module-level	`{repo}:{branch}:{sha}`	New commit SHA
`snapshotCache`	Module-level	`{commit}:{state}:{error}`	New snapshot
`blobCache`	Module-level	blob SHA	Never (within process lifetime)
`searchCache`	WeakMap on corpus	—	New corpus instance

Phase 2: Corpus Building (`buildCorpus`)

buildCorpus(rawFiles, status) is the central function. It takes an array of RawWikiFile objects and produces a WikiCorpus with fully resolved links, computed PageRank, and indexed pages.

Step 2a: Normalization

Each RawWikiFile is normalized:

Path is POSIX-normalized (backslashes to forward slashes, no leading ./ or ../)
blobSha defaults to SHA-256 of raw if not provided
byteLength is computed from UTF-8 encoding of raw

Files that are not .md or .mdx, or whose path ends with AGENTS.md, are filtered out.

Step 2b: Frontmatter Parsing (`parseFrontmatter`)

The parser extracts YAML frontmatter delimited by ---:

---
type: reference
title: "Page Title"
description: "One-line description"
tags: [tag1, tag2]
timestamp: 2026-07-21
custom_field: value
---

Parsing rules:

The --- delimiter must be the first line of the file
YAML is parsed via the yaml library (YAML.parseDocument) with error collection
The okf fields (type, title, description, tags, timestamp) are extracted into a typed structure
All other fields go into metadata (a generic Record<string, unknown>)
Parse errors are captured in frontmatter.diagnostics rather than crashing
If no frontmatter exists, the entire file is treated as body
The body is the raw content after the closing ---, preserving all whitespace and formatting

Important: The raw field on the resulting WikiPage is the original, unmodified file content. The pipeline never re-serializes or re-encodes the source markdown.

Step 2c: Page Construction

Each normalized file becomes a WikiPage with:

Field	Source	Description
`id`	path without extension	Stable identifier
`routePath`	path without extension	Used in `/wiki/` URLs
`title`	frontmatter `title` or first `# heading` or filename	Human-readable name
`description`	frontmatter `description`	One-line summary
`type`	frontmatter `type` or `"page"`	OKF type
`category`	Computed from `type` + path prefix	One of: person, project, research, directive, memory, other
`tags`	frontmatter `tags`	String array
`excerpt`	First 240 chars of description or stripped body	For search results and suggestions
`hash`	SHA-256 of `raw`	Content integrity check
`media`	Extracted from markdown AST + HTML	`WikiMediaReference[]`

Category classification (categoryFor):

Category	Trigger (type or path prefix)
`person`	`type` contains `person`/`pessoa`/`contact`, or path starts with `pessoas/`
`directive`	`type` contains `directive`/`diretiva`, or path starts with `diretivas/`
`research`	`type` contains `research`/`pesquisa`, or path starts with `pesquisas/`, `kaggle/`, `fitdnuvo/`, `clickfix/`
`project`	`type` contains `project`/`projeto`, or path starts with `infraestrutura/`, `claude_desktop/`, `ai_modelos/`, `mcp/`, `hacks/`
`memory`	`type` contains `memory`/`memoria`/`memória`, or path starts with `memorias/`
`other`	Default fallback

Phase 3: Link Resolution

Link Extraction

Links are extracted from the parsed AST using unist-util-visit:

Markdown links and images: standard markdown link and image syntax nodes from the remark AST
Wiki links: double-bracket patterns (target or target with label) found in raw text nodes
HTML media references: <iframe>, <video>, <audio>, <source>, <img> tags via regex

Link Resolution (`resolveLinkTarget`)

Each extracted link target is resolved against the set of all page paths:

Step	Action	Result
1	Check if external URL (`https://...`)	`state: "external"`
2	Fragment-only (`#section`)	`state: "resolved"`, destination = source page
3	Exact match in `allPaths`	`state: "resolved"`
4	With `.md`/`.mdx` extension appended	`state: "resolved"`
5	Relative path resolution (from source directory)	`state: "resolved"`
6	Slug match (case-insensitive, last path segment)	`state: "resolved"` if unique, `"ambiguous"` if multiple
7	No match found	`state: "missing"`

Backlink Computation

After all pages have their outlinks resolved, backlinks are computed as the reverse index: if page A links to page B, then B's backlinks includes A. This enables the "context rail" in the UI showing which pages reference the current document.

Phase 4: PageRank

The corpus computes a standard PageRank with damping factor 0.85:

Parameter	Value
Damping factor	0.85
Tolerance	1e-10
Max iterations	200

Dangling node handling: Pages with no outgoing links distribute their rank equally to all pages (standard PageRank treatment).

Normalization: Final ranks are normalized so they sum to 1.0 across the corpus.

PageRank is used for:

Search result ranking (boost factor)
Home page suggestions (topSuggestedPages)
Node radius in the D3 graph visualization
Tie-breaking in search results

Phase 5: Media Reference Extraction

The extractMediaReferences function walks the markdown AST and HTML to find all media references:

Media Kind	Detection	Allowed	Embed URL
`image`	`.avif`, `.gif`, `.jpg`, `.jpeg`, `.png`, `.svg`, `.webp` extensions	Yes	—
`video`	`.m4v`, `.mov`, `.mp4`, `.webm` extensions	Yes	—
`audio`	`.flac`, `.m4a`, `.mp3`, `.ogg`, `.wav` extensions	Yes	—
`embed`	YouTube or Vimeo URLs	Yes	Transformed to `youtube-nocookie.com/embed/` or `player.vimeo.com/video/`
`link`	Non-HTTPS URLs	No	—

Security: Only HTTPS URLs are allowed. HTTP URLs are marked as allowed: false with reason remote_media_requires_https. There is no local asset storage; all media is served from remote URLs.

YouTube URL transformation:

https://youtube.com/watch?v=ID -> https://www.youtube-nocookie.com/embed/ID
https://youtu.be/ID -> https://www.youtube-nocookie.com/embed/ID
https://youtube.com/shorts/ID -> https://www.youtube-nocookie.com/embed/ID

Phase 6: HTML Rendering (`renderMarkdown`)

The rendering pipeline transforms page.body (markdown without frontmatter) into sanitized HTML using a unified/remark/rehype pipeline:

flowchart LR
    A[page.body] --> B[remarkParse]
    B --> C[remarkGfm]
    C --> D[remarkMath]
    D --> E[remarkMermaidGuard]
    E --> F[remarkWikiLinks]
    F --> G[remarkCanonicalLinks]
    G --> H[remarkRehype]
    H --> I[rehypeRaw]
    I --> J[rehypeSanitize]
    J --> K[rehypeWikiLinkClasses]
    K --> L[rehypeMediaEmbeds]
    L --> M[rehypeKatex]
    M --> N[rehypeHighlight]
    N --> O[rehypeStringify]
    O --> P[HTML string]

Plugin Details

Plugin	Purpose
`remarkParse`	Parse markdown to MDAST
`remarkGfm`	GitHub Flavored Markdown (tables, strikethrough, autolinks)
`remarkMath`	Parse $...$ and `$$...$$` math syntax
`remarkMermaidGuard`	Validate mermaid code blocks; demote invalid ones to plain text
`remarkWikiLinks`	Transform wiki-link syntax to `<a>` nodes with resolved hrefs
`remarkCanonicalLinks`	Rewrite markdown links to canonical `/wiki/...` paths
`remarkRehype`	Convert MDAST to HAST (allowing raw HTML)
`rehypeRaw`	Parse raw HTML embedded in markdown
`rehypeSanitize`	Whitelist-based HTML sanitization (extended schema for wiki-specific attributes)
`rehypeWikiLinkClasses`	Add `class="wiki-link"` to internal `/wiki/` links
`rehypeMediaEmbeds`	Convert media links to `<figure>` elements with `<img>`, `<video>`, `<audio>`, or `<iframe>`
`rehypeKatex`	Render math expressions to HTML with KaTeX
`rehypeHighlight`	Syntax highlighting for code blocks
`rehypeStringify`	Serialize HAST to HTML string

Wiki Link Rendering

Wiki links with display text are transformed to standard <a> tags:

Resolved links get class="wiki-link" and href /wiki/{routePath}
Missing links get class="wiki-link unresolved" and href #target
Ambiguous links get class="wiki-link unresolved ambiguous" and href #target
The data-link-state attribute carries the resolution state

Media Embed Rendering

Media links are converted to semantic <figure> elements:

<!-- Image -->
<figure class="media-embed media-embed-image">
  <img src="https://..." alt="label" loading="lazy" />
  <figcaption>label</figcaption>
</figure>

<!-- YouTube -->
<figure class="media-embed media-embed-frame">
  <iframe src="https://www.youtube-nocookie.com/embed/..." allowfullscreen loading="lazy" />
  <figcaption>label</figcaption>
</figure>

<!-- Blocked (HTTP) -->
<a class="media-link media-link-blocked" data-media-state="remote_media_requires_https">

Sanitization Schema

The sanitization schema extends the default rehype-sanitize schema with:

Custom className values for link states (wiki-link, unresolved, missing, ambiguous, media-link-blocked)
data-link-state, data-mermaid-source attributes
Math elements (math, mi, mn, mo, ms, mtext, mfrac, msqrt, msup, msub)
Figure/figcaption elements
Input elements with checkbox support

Phase 7: Search Index

The search system uses a custom BM25 implementation with PageRank and graph-proximity boosting.

Index Construction

The search index is built lazily and cached via WeakMap on the corpus:

Each page's searchable text is assembled: title + description + tags + type + category + path + body
Text is normalized: NFD decomposition, diacritic removal, lowercase (Portuguese-aware)
Tokens are extracted via regex: [\p{L}\p{N}_-]+
Document frequency and average length are computed for BM25 IDF

BM25 Scoring

Parameter	Value
k1	1.35
b	0.72

Score Composition

Factor	Weight	Description
BM25	1.8x	Text relevance
Title boost	+4.0 (exact) / +1.5 (substring)	Query matches page title
PageRank boost	up to +1.25	Proportional to normalized PageRank
Proximity boost	up to +1.25	Graph distance from focus path (1.25 - distance * 0.35)

Proximity Boost

When a focusPath is provided (e.g., the currently viewed page), search results that are close to the focused page in the link graph receive a boost. Distance is computed via BFS on the undirected graph (outlinks + backlinks), capped at depth 3.

Suppression

Pages with stems claude, agents, index, or filenames containing prefill or template are suppressed from search results and suggestions.

Data Types Summary

WikiCorpus

type WikiCorpus = {
  status: WikiSourceStatus;      // Repository, branch, commit, source, state
  pages: WikiPage[];             // All processed pages
  pageByPath: Map<string, WikiPage>;  // Fast lookup by path
  links: WikiGraphLink[];        // All resolved links for D3 graph
};

WikiPage

type WikiPage = {
  id: string;                    // path without extension
  path: string;                  // Original file path
  routePath: string;             // Used in /wiki/ URLs
  rawUrl: string;                // /raw/{path}
  canonicalUrl: string;          // Full URL with site base
  raw: string;                   // Unmodified source markdown
  body: string;                  // Markdown without frontmatter
  frontmatter: WikiFrontmatter;  // Parsed OKF + metadata + diagnostics
  title: string;
  description?: string;
  type: string;
  category: WikiCategory;
  tags: string[];
  timestamp?: string;
  excerpt: string;               // First 240 chars for previews
  byteLength: number;
  blobSha: string;
  hash: string;                  // SHA-256 of raw
  links: WikiLink[];             // All extracted + resolved links
  media: WikiMediaReference[];   // All media references
  outlinks: string[];            // Resolved outgoing link paths
  backlinks: string[];           // Pages that link to this page
  pagerank: number;              // Normalized PageRank score
};

WikiClientPage

The client-safe subset sent to the browser (excludes raw, body, and full links):

type WikiClientPage = Omit<WikiPage, "raw" | "body" | "links"> & {
  links: Array<Pick<WikiLink, "target" | "label" | "fragment" | "destination" | "state">>;
};

Path Resolution (`resolveWikiPath`)

When a user requests /wiki/some/path, the resolver attempts:

Exact match: some/path -> check pageByPath
With extension: some/path.md, some/path.mdx
Slug match: extract last segment, case-insensitive comparison
Unique alias: if exactly one page matches the slug
Ambiguous: multiple pages share the same slug -> state: "ambiguous"
Missing: no match found -> state: "missing" (renders 404)

wikifita-site-architecture -- Full technical architecture
wikifita-site-llm-integration -- LLM endpoint design
wikifita-site -- Project overview stub