Corpus Building Pipeline

The buildCorpus function (lib/wiki.ts, lines 457-520) is the central processing pipeline of the Wikifita Atlas. It transforms an array of RawWikiFile objects -- sourced from either the GitHub API or a pre-built snapshot -- into a fully indexed WikiCorpus with resolved links, computed PageRank, and classified pages.

This page documents all 7 phases in detail, including the data loading strategies and fallback mechanisms.

---

Pipeline Overview

flowchart TD
    A["Data Source"] -->|"RawWikiFile[]"| B["Phase 1: Normalization"]
    B --> C["Phase 2: Frontmatter Parsing"]
    C --> D["Phase 3: Title Resolution"]
    D --> E["Phase 4: Category Classification"]
    E --> F["Phase 5: Link Extraction & Resolution"]
    F --> G["Phase 6: Media Extraction"]
    G --> H["Phase 7: PageRank Computation"]
    H --> I["WikiCorpus"]

    subgraph "Data Sources"
        J["GitHub Live (API)"] --> A
        K["Snapshot (JSON)"] --> A
    end

    subgraph "Outputs"
        I --> L["pages: WikiPage[]"]
        I --> M["pageByPath: Map"]
        I --> N["links: WikiGraphLink[]"]
        I --> O["status: WikiSourceStatus"]
    end

---

Data Sources

Before the 7-phase pipeline runs, raw files must be acquired. The Wikifita Atlas supports two data sources with a fallback chain.

Source 1: GitHub Live (loadGitHubCorpus)

The primary data source fetches wiki files from the aleffita/wikifita GitHub repository via the REST API.

Fetch Sequence

sequenceDiagram
    participant WikiLib as lib/wiki.ts
    participant GH as GitHub API

    WikiLib->>GH: GET /repos/{owner}/{repo}/commits/{branch}
    GH-->>WikiLib: { sha, commit.date }

    alt SHA matches cache
        WikiLib->>WikiLib: Return cached corpus
    else New SHA
        WikiLib->>GH: GET /repos/{owner}/{repo}/git/trees/{sha}?recursive=1
        GH-->>WikiLib: tree entries

        WikiLib->>WikiLib: Filter .md/.mdx, exclude AGENTS.md

        loop For each blob (max 8 concurrent)
            WikiLib->>GH: GET /repos/{owner}/{repo}/git/blobs/{sha}
            GH-->>WikiLib: base64 content
            WikiLib->>WikiLib: Decode to UTF-8, cache by blob SHA
        end

        WikiLib->>WikiLib: buildCorpus(rawFiles, status)
    end

SHA Caching

The in-memory liveCache stores the most recent corpus keyed by {repo}:{branch}:{sha}. If the GitHub API returns the same commit SHA as the cached key, the entire corpus is returned without any additional API calls. This makes repeated requests within the same commit essentially free.

Blob Fetching

Blobs are fetched with a concurrency limit of 8 simultaneous requests (mapLimit). Each blob is:

1. Fetched from the GitHub API (returns base64-encoded content)
2. Decoded from base64 to UTF-8
3. Cached in blobCache (a Map<string, string> keyed by blob SHA)

The blobCache persists for the lifetime of the worker process. If the same blob SHA appears in a future corpus (e.g., unchanged files across commits), it is served from cache without an API call.

Tree Filtering

The recursive git tree is filtered to:

• Only .md and .mdx files
• Excluding files whose path ends with AGENTS.md
• Sorted by path (deterministic ordering)

Source 2: Snapshot (loadSnapshotCorpus)

The fallback data source reads from a pre-built JSON file: data/wiki-snapshot.json.

Snapshot Generation

The snapshot is generated by scripts/build-wiki-snapshot.mjs:

npm run wiki:snapshot

This script:

1. Fetches all .md/.mdx files from the GitHub repository
2. Encodes them as RawWikiFile objects (path, raw content, blobSha, byteLength)
3. Writes to data/wiki-snapshot.json

Snapshot Loading

const snapshot = JSON.parse(readFileSync('data/wiki-snapshot.json', 'utf-8'));
const rawFiles = snapshot.rawFiles || snapshot.pages;
return buildCorpus(rawFiles, { source: 'snapshot', ... });

The snapshot format is flexible -- it accepts either rawFiles (the preferred key) or pages (legacy).

Fallback Chain

The loadWikiCorpus function (lines 625-634) implements the fallback logic:

1. If WIKIFITA_SOURCE=snapshot:
     Always use snapshot

2. If WIKIFITA_SOURCE=github (default):
     Try loadGitHubCorpus()
     If success: return live corpus
     If failure + WIKIFITA_FALLBACK=snapshot:
       Return snapshot corpus with state="degraded"
       Include error message in status
     If failure + no fallback:
       Throw error

The degraded state is important: it signals to the UI and API responses that the data may be stale. The X-Wikifita-Source-State header exposes this state.

---

Phase 1: Raw File Normalization

Input: RawWikiFile[] from data source Output: Filtered, normalized RawWikiFile[]

Processing Steps

flowchart LR
    A[RawWikiFile[]] --> B[Filter by extension]
    B --> C[Exclude AGENTS.md]
    C --> D[Normalize paths]
    D --> E[Compute SHA/bytes]
    E --> F[Sort by path]
    F --> G[Normalized RawWikiFile[]]

Filtering

Only files with .md or .mdx extensions pass through. Files named AGENTS.md (at any path depth) are excluded -- these are symlinks to CLAUDE.md in the wikifita repository and are not wiki content.

Path Normalization (normalizePosix)

1. Convert backslashes to forward slashes:  "dir\\file" -> "dir/file"
2. Remove leading "../":  "../dir/file" -> "dir/file"
3. Remove leading "./":  "./dir/file" -> "dir/file"
4. Remove leading "/":  "/dir/file" -> "dir/file"

This ensures all paths are POSIX-normalized relative paths, regardless of the source operating system.

SHA and Byte Length

Field	Source	Fallback
blobSha	Git blob SHA (from GitHub API)	SHA-256 hash of raw content
byteLength	Provided by source	new TextEncoder().encode(raw).byteLength

The blobSha is used for content identification and caching. The byteLength is the exact UTF-8 byte count, not the character count -- this matters for Portuguese text with multi-byte characters.

---

Phase 2: Frontmatter Parsing

Input: Normalized RawWikiFile Output: WikiFrontmatter + body string

YAML Frontmatter Extraction

The parser extracts YAML frontmatter delimited by ---:

---             ← Must be the first line (byte 0)
type: reference
title: "Page Title"
description: "One-line description"
tags: [tag1, tag2]
timestamp: 2026-07-21
custom_field: value
---             ← Closing delimiter
Body content here...

Parsing Rules

1. Delimiter detection: The frontmatter must start at byte 0 with ---\n or ---\r\n
2. Parser: Uses YAML.parseDocument() (not YAML.parse()) to capture errors as diagnostics instead of throwing
3. Root type: Must be an object (not array, scalar, or null). Non-object roots produce a frontmatter_root_is_not_object diagnostic
4. Missing close delimiter: If the closing --- is missing, a diagnostic is recorded and the entire file is treated as body

OKF Field Extraction

Field	YAML Path	Type	Notes
type	metadata.type	string	OKF type (e.g., "reference", "analysis")
title	metadata.title	string	Human-readable title
description	metadata.description	string	One-line summary
tags	metadata.tags	string[]	Parsed from array, comma-separated string, or bracket-enclosed string
timestamp	metadata.timestamp	string	ISO 8601 date

Tag Parsing

The tags field accepts three formats and normalizes them all to string[]:

# Array (preferred)
tags: [tag1, tag2, tag3]

# Comma-separated string
tags: "tag1, tag2, tag3"

# Bracket-enclosed string
tags: "[tag1, tag2, tag3]"

Metadata Preservation

The entire raw YAML object is preserved in frontmatter.metadata. This allows arbitrary custom fields beyond the OKF standard to be accessed programmatically.

---
type: research
title: "Research Page"
custom_field: value
nested:
  key: value
---

frontmatter.metadata would be { type: "research", title: "Research Page", custom_field: "value", nested: { key: "value" } }.

Diagnostic Accumulation

Parse errors are accumulated (never thrown):

Diagnostic	Condition
YAML parse error	Malformed YAML syntax
Missing close delimiter	No closing --- found
Root is not object	YAML root is array or scalar
Frontmatter not found	No opening --- at byte 0

Diagnostics are available on page.frontmatter.diagnostics for debugging and quality checks.

---

Phase 3: Title Resolution

Input: Parsed frontmatter + body Output: Resolved title string

Resolution Cascade

Priority 1: frontmatter.okf.title
  └── If present and non-empty, use as title

Priority 2: First H1 heading in body
  └── Pattern: /^#\s+(.+)$/m
  └── Searches the body (after frontmatter) for the first markdown H1

Priority 3: Filename stem
  └── Extract filename without extension
  └── Replace hyphens and underscores with spaces
  └── "my-page_name.md" -> "my page name"

titleFromBody Function

function titleFromBody(body: string): string | undefined {
  const match = body.match(/^#\s+(.+)$/m);
  return match ? match[1].trim() : undefined;
}

The regex /^#\s+(.+)$/m matches:

• ^ -- start of line (with m flag)
• # -- literal hash
• \s+ -- one or more whitespace characters
• (.+) -- capture group: everything after the hash and space
• $ -- end of line

Edge Cases

• Multiple H1 headings: Only the first one is used (the regex does not have the g flag)
• H1 with inline formatting: The entire text content is captured, including markdown formatting markers. These are not stripped -- the title is used for display, and the rendering pipeline handles formatting.
• Empty frontmatter title: title: "" is treated as absent, falling through to H1 or filename
• No body content: If there is no H1 and no frontmatter title, the filename stem is used

---

Phase 4: Category Classification

Input: OKF type field + page path Output: WikiCategory enum value

Classification Logic (categoryFor)

The classifier uses a priority cascade: first checking the type metadata, then falling back to path prefix matching.

flowchart TD
    A["type metadata"] --> B{Contains person/pessoa/contact?}
    B -->|yes| C["category: person"]
    B -->|no| D{Contains directive/diretiva?}
    D -->|yes| E["category: directive"]
    D -->|no| F{Contains research/pesquisa?}
    F -->|yes| G["category: research"]
    F -->|no| H{Contains project/projeto?}
    H -->|yes| I["category: project"]
    H -->|no| J{Contains memory/memoria/memória?}
    J -->|yes| K["category: memory"]
    J -->|no| L{Path prefix match?}
    L -->|match| M[Category from prefix]
    L -->|no match| N["category: other"]

Complete Mapping

Category	Type Match (case-insensitive)	Path Prefix
person	person, pessoa, contact	pessoas/
directive	directive, diretiva	diretivas/
research	research, pesquisa	pesquisas/, kaggle/, fitdnuvo/, clickfix/
project	project, projeto	infraestrutura/, claude_desktop/, ai_modelos/, mcp/, hacks/
memory	memory, memoria, memória	memorias/
other	(default)	(no prefix match)

Priority Rules

1. Type match takes precedence over path prefix: A file at pesquisas/paper.md with type: directive is classified as directive, not research.
2. Case-insensitive matching: type: "Research" matches the research category.
3. Portuguese variants: The classifier supports both English and Portuguese type names (e.g., pesquisa for research, diretiva for directive).
4. First match wins: The type is checked against categories in the order shown above. If a type matches multiple categories (unlikely but possible), the first match wins.

Impact on UI

Categories drive:

• Graph visualization: Each category has a distinct color and X-position in the D3 force layout wikifita-site-graph-construction
• LLM index: The /llms.txt endpoint organizes pages by category with Portuguese section headers wikifita-site-llm-integration
• Search index: Category is included in the searchable text, so searching "pesquisa" matches research pages

---

Phase 5: Link Extraction and Resolution

Input: Parsed pages with body content Output: Pages with links[], outlinks[], and backlinks[]

This is the most complex phase. It is documented in full detail in wikifita-site-graph-construction.

Summary

1. Extract links from the markdown AST (markdown links, wiki links, image references)
2. Resolve each link target through the 6-stage cascade (external check, fragment-only, exact match, extension variants, relative path, fuzzy slug)
3. Build outlinks: Unique resolved destinations, excluding self-links
4. Build backlinks: Inverse mapping from outlinks
5. Build edge list: Flat {source, target} array for graph visualization and PageRank

---

Phase 6: Media Extraction

Input: Parsed pages with body content Output: Pages with media[] array

Extraction Sources

Media references are collected from two sources in the markdown content:

Source A: Markdown AST

The extractMediaReferences function (lines 378-404) visits link and image AST nodes:

Node Type	Extracted Data
link	URL + text content as label
image	URL + alt text as label

Source B: Raw HTML in Body

A regex captures src and poster attributes from HTML media elements:

Pattern: /<(iframe|video|audio|source|img)\b[^>]*\s(?:src|poster)=["']([^"']+)["'][^>]*>/gi

This catches media elements written as raw HTML in the markdown, which the AST parser may not represent as standard link/image nodes.

URL Classification (classifyMediaUrl)

Each extracted URL is classified by lib/media.ts:

flowchart TD
    A[URL] --> B{HTTPS?}
    B -->|no| C["allowed: false, reason: remote_media_requires_https"]
    B -->|yes| D{Extension match?}
    D -->|".avif,.gif,.jpg,.jpeg,.png,.svg,.webp"| E["kind: image, allowed: true"]
    D -->|".m4v,.mov,.mp4,.webm"| F["kind: video, allowed: true"]
    D -->|".flac,.m4a,.mp3,.ogg,.wav"| G["kind: audio, allowed: true"]
    D -->|YouTube/Vimeo URL| H["kind: embed, allowed: true, embedUrl: privacy-rewritten"]
    D -->|no match| I["Not collected (ignored)"]

YouTube/Vimeo URL Rewriting

Input URL	Output Embed URL
youtube.com/watch?v=ID	youtube-nocookie.com/embed/ID
youtu.be/ID	youtube-nocookie.com/embed/ID
youtube.com/shorts/ID	youtube-nocookie.com/embed/ID
vimeo.com/ID	player.vimeo.com/video/ID

Deduplication

Media references are deduplicated by the composite key ${kind}:${url}:${label}. If the same URL appears multiple times in a page (e.g., an image used twice), only one reference is stored.

Security

• HTTPS only: Non-HTTPS URLs are blocked with allowed: false
• No local assets: All media must be remote URLs. There is no local asset storage, upload pipeline, or bucket.
• Privacy-enhanced embeds: YouTube URLs use youtube-nocookie.com to prevent tracking

---

Phase 7: PageRank Computation

Input: Pages with resolved links Output: Pages with pagerank field

The final phase computes PageRank over the directed link graph. This is documented in full detail in wikifita-site-pagerank.

Summary

1. Build outlinksByPath from each page's outlinks array
2. Run iterative PageRank with $d = 0.85$, tolerance $10^{-10}$, max 200 iterations
3. Normalize ranks to sum to 1.0
4. Store each page's rank in pagerank field

---

Corpus Assembly

After all 7 phases, the final WikiCorpus object is assembled:

type WikiCorpus = {
  status: WikiSourceStatus;         // Source metadata
  pages: WikiPage[];                // All processed pages, sorted by path
  pageByPath: Map<string, WikiPage>; // O(1) lookup by path
  links: WikiGraphLink[];           // {source, target} edge list
};

WikiSourceStatus

type WikiSourceStatus = {
  repository: string;     // "aleffita/wikifita"
  branch: string;         // "main"
  commit: string;         // Git SHA
  commitDate: string;     // ISO date of commit
  loadedAt: string;       // ISO date of corpus build
  source: "github" | "snapshot";
  state: "live" | "cached" | "degraded";
  error?: string;         // Error message if degraded
};

Client Data Serialization

Before sending to the browser, the corpus is serialized to WikiClientData:

type WikiClientPage = Omit<WikiPage, "raw" | "body" | "links"> & {
  links: Array<Pick<WikiLink, "target" | "label" | "fragment" | "destination" | "state">>;
};

This strips:

• raw (full source markdown -- too large for client)
• body (markdown without frontmatter -- unnecessary on client)
• links[].origin (internal tracking data)

The client receives enough data for graph visualization, search, and link display without exposing the full markdown content.

---

Caching Strategy

Cache	Scope	Key	Invalidation	Location
liveCache	Module-level	{repo}:{branch}:{sha}	New commit SHA	Memory
snapshotCache	Module-level	{commit}:{state}:{error}	New snapshot	Memory
blobCache	Module-level	Blob SHA	Never (within process)	Memory
searchCache	WeakMap	Corpus reference	New corpus instance	Memory

In-Memory Caching by Commit SHA

The primary caching mechanism is SHA-based deduplication:

if (liveCache.key === `${repo}:${branch}:${sha}`) {
  return liveCache.corpus;  // Cache hit: no API calls
}

This means:

• The first request after a commit builds the full corpus (~100-500ms depending on corpus size)
• Subsequent requests with the same commit SHA return in <1ms
• A new commit invalidates the cache and triggers a full rebuild

Blob-Level Caching

Individual blob content is cached by SHA. When a new commit changes only some files, unchanged files are served from the blob cache. This reduces GitHub API calls on incremental updates.

---

Testing

The corpus pipeline is covered by unit tests in tests/wiki-pipeline.test.ts:

Test	What It Verifies
Frontmatter parsing	OKF fields extracted correctly, YAML errors captured as diagnostics
YAML error handling	Malformed YAML produces diagnostics, not crashes
Link resolution (markdown)	Standard markdown links resolve correctly
Link resolution (wiki)	WIKI_LINK_TARGET_LABEL patterns resolve correctly
Code fence ignoring	Links inside code blocks are not extracted
MDX support	.mdx files are processed identically to .md
Media classification	URLs classified correctly by extension and provider
PageRank convergence	Ranks sum to 1.0, dangling nodes behave correctly
BM25 accent-insensitive	"cafe" matches "Café" after normalization
Alias resolution	Fuzzy slug matching resolves to correct page
Live/snapshot equivalence	Both sources produce identical corpus for the same content

---

Related Pages

• wikifita-site-pagerank -- Phase 7 deep-dive
• wikifita-site-graph-construction -- Phase 5 deep-dive
• wikifita-site-search-system -- How the corpus feeds the search index
• wikifita-site-rendering-pipeline -- How page.body is rendered to HTML
• wikifita-site-deployment -- How the corpus is loaded in Cloudflare Workers
• wikifita-site-architecture -- Full technical architecture