---
name: wikifita-site-corpus-pipeline
type: analysis
title: "Corpus Building Pipeline"
description: "The 7-phase pipeline that transforms raw wiki files into a fully indexed WikiCorpus: normalization, frontmatter parsing, title resolution, category classification, link extraction, media extraction, and PageRank computation."
tags: [wikifita, corpus, pipeline, frontmatter, yaml, category, media, github-api, snapshot]
timestamp: 2026-07-21
---

# Corpus Building Pipeline

The `buildCorpus` function (`lib/wiki.ts`, lines 457-520) is the central processing pipeline of the Wikifita Atlas. It transforms an array of `RawWikiFile` objects -- sourced from either the GitHub API or a pre-built snapshot -- into a fully indexed `WikiCorpus` with resolved links, computed PageRank, and classified pages.

This page documents all 7 phases in detail, including the data loading strategies and fallback mechanisms.

---

## Pipeline Overview

```mermaid
flowchart TD
    A["Data Source"] -->|"RawWikiFile[]"| B["Phase 1: Normalization"]
    B --> C["Phase 2: Frontmatter Parsing"]
    C --> D["Phase 3: Title Resolution"]
    D --> E["Phase 4: Category Classification"]
    E --> F["Phase 5: Link Extraction & Resolution"]
    F --> G["Phase 6: Media Extraction"]
    G --> H["Phase 7: PageRank Computation"]
    H --> I["WikiCorpus"]

    subgraph "Data Sources"
        J["GitHub Live (API)"] --> A
        K["Snapshot (JSON)"] --> A
    end

    subgraph "Outputs"
        I --> L["pages: WikiPage[]"]
        I --> M["pageByPath: Map"]
        I --> N["links: WikiGraphLink[]"]
        I --> O["status: WikiSourceStatus"]
    end
```

---

## Data Sources

Before the 7-phase pipeline runs, raw files must be acquired. The Wikifita Atlas supports two data sources with a fallback chain.

### Source 1: GitHub Live (`loadGitHubCorpus`)

The primary data source fetches wiki files from the `aleffita/wikifita` GitHub repository via the REST API.

#### Fetch Sequence

```mermaid
sequenceDiagram
    participant WikiLib as lib/wiki.ts
    participant GH as GitHub API

    WikiLib->>GH: GET /repos/{owner}/{repo}/commits/{branch}
    GH-->>WikiLib: { sha, commit.date }

    alt SHA matches cache
        WikiLib->>WikiLib: Return cached corpus
    else New SHA
        WikiLib->>GH: GET /repos/{owner}/{repo}/git/trees/{sha}?recursive=1
        GH-->>WikiLib: tree entries

        WikiLib->>WikiLib: Filter .md/.mdx, exclude AGENTS.md

        loop For each blob (max 8 concurrent)
            WikiLib->>GH: GET /repos/{owner}/{repo}/git/blobs/{sha}
            GH-->>WikiLib: base64 content
            WikiLib->>WikiLib: Decode to UTF-8, cache by blob SHA
        end

        WikiLib->>WikiLib: buildCorpus(rawFiles, status)
    end
```

#### SHA Caching

The in-memory `liveCache` stores the most recent corpus keyed by `{repo}:{branch}:{sha}`. If the GitHub API returns the same commit SHA as the cached key, the entire corpus is returned without any additional API calls. This makes repeated requests within the same commit essentially free.

#### Blob Fetching

Blobs are fetched with a concurrency limit of 8 simultaneous requests (`mapLimit`). Each blob is:
1. Fetched from the GitHub API (returns base64-encoded content)
2. Decoded from base64 to UTF-8
3. Cached in `blobCache` (a `Map<string, string>` keyed by blob SHA)

The `blobCache` persists for the lifetime of the worker process. If the same blob SHA appears in a future corpus (e.g., unchanged files across commits), it is served from cache without an API call.

#### Tree Filtering

The recursive git tree is filtered to:
- Only `.md` and `.mdx` files
- Excluding files whose path ends with `AGENTS.md`
- Sorted by path (deterministic ordering)

### Source 2: Snapshot (`loadSnapshotCorpus`)

The fallback data source reads from a pre-built JSON file: `data/wiki-snapshot.json`.

#### Snapshot Generation

The snapshot is generated by `scripts/build-wiki-snapshot.mjs`:

```bash
npm run wiki:snapshot
```

This script:
1. Fetches all `.md`/`.mdx` files from the GitHub repository
2. Encodes them as `RawWikiFile` objects (path, raw content, blobSha, byteLength)
3. Writes to `data/wiki-snapshot.json`

#### Snapshot Loading

```javascript
const snapshot = JSON.parse(readFileSync('data/wiki-snapshot.json', 'utf-8'));
const rawFiles = snapshot.rawFiles || snapshot.pages;
return buildCorpus(rawFiles, { source: 'snapshot', ... });
```

The snapshot format is flexible -- it accepts either `rawFiles` (the preferred key) or `pages` (legacy).

### Fallback Chain

The `loadWikiCorpus` function (lines 625-634) implements the fallback logic:

```
1. If WIKIFITA_SOURCE=snapshot:
     Always use snapshot

2. If WIKIFITA_SOURCE=github (default):
     Try loadGitHubCorpus()
     If success: return live corpus
     If failure + WIKIFITA_FALLBACK=snapshot:
       Return snapshot corpus with state="degraded"
       Include error message in status
     If failure + no fallback:
       Throw error
```

The `degraded` state is important: it signals to the UI and API responses that the data may be stale. The `X-Wikifita-Source-State` header exposes this state.

---

## Phase 1: Raw File Normalization

**Input:** `RawWikiFile[]` from data source
**Output:** Filtered, normalized `RawWikiFile[]`

### Processing Steps

```mermaid
flowchart LR
    A[RawWikiFile[]] --> B[Filter by extension]
    B --> C[Exclude AGENTS.md]
    C --> D[Normalize paths]
    D --> E[Compute SHA/bytes]
    E --> F[Sort by path]
    F --> G[Normalized RawWikiFile[]]
```

### Filtering

Only files with `.md` or `.mdx` extensions pass through. Files named `AGENTS.md` (at any path depth) are excluded -- these are symlinks to `CLAUDE.md` in the wikifita repository and are not wiki content.

### Path Normalization (`normalizePosix`)

```
1. Convert backslashes to forward slashes:  "dir\\file" -> "dir/file"
2. Remove leading "../":  "../dir/file" -> "dir/file"
3. Remove leading "./":  "./dir/file" -> "dir/file"
4. Remove leading "/":  "/dir/file" -> "dir/file"
```

This ensures all paths are POSIX-normalized relative paths, regardless of the source operating system.

### SHA and Byte Length

| Field | Source | Fallback |
|-------|--------|----------|
| `blobSha` | Git blob SHA (from GitHub API) | SHA-256 hash of `raw` content |
| `byteLength` | Provided by source | `new TextEncoder().encode(raw).byteLength` |

The `blobSha` is used for content identification and caching. The `byteLength` is the exact UTF-8 byte count, not the character count -- this matters for Portuguese text with multi-byte characters.

---

## Phase 2: Frontmatter Parsing

**Input:** Normalized `RawWikiFile`
**Output:** `WikiFrontmatter` + `body` string

### YAML Frontmatter Extraction

The parser extracts YAML frontmatter delimited by `---`:

```
---             ← Must be the first line (byte 0)
type: reference
title: "Page Title"
description: "One-line description"
tags: [tag1, tag2]
timestamp: 2026-07-21
custom_field: value
---             ← Closing delimiter
Body content here...
```

### Parsing Rules

1. **Delimiter detection**: The frontmatter must start at byte 0 with `---\n` or `---\r\n`
2. **Parser**: Uses `YAML.parseDocument()` (not `YAML.parse()`) to capture errors as diagnostics instead of throwing
3. **Root type**: Must be an object (not array, scalar, or null). Non-object roots produce a `frontmatter_root_is_not_object` diagnostic
4. **Missing close delimiter**: If the closing `---` is missing, a diagnostic is recorded and the entire file is treated as body

### OKF Field Extraction

| Field | YAML Path | Type | Notes |
|-------|-----------|------|-------|
| `type` | `metadata.type` | `string` | OKF type (e.g., "reference", "analysis") |
| `title` | `metadata.title` | `string` | Human-readable title |
| `description` | `metadata.description` | `string` | One-line summary |
| `tags` | `metadata.tags` | `string[]` | Parsed from array, comma-separated string, or bracket-enclosed string |
| `timestamp` | `metadata.timestamp` | `string` | ISO 8601 date |

### Tag Parsing

The `tags` field accepts three formats and normalizes them all to `string[]`:

```yaml
# Array (preferred)
tags: [tag1, tag2, tag3]

# Comma-separated string
tags: "tag1, tag2, tag3"

# Bracket-enclosed string
tags: "[tag1, tag2, tag3]"
```

### Metadata Preservation

The entire raw YAML object is preserved in `frontmatter.metadata`. This allows arbitrary custom fields beyond the OKF standard to be accessed programmatically.

```yaml
---
type: research
title: "Research Page"
custom_field: value
nested:
  key: value
---
```

`frontmatter.metadata` would be `{ type: "research", title: "Research Page", custom_field: "value", nested: { key: "value" } }`.

### Diagnostic Accumulation

Parse errors are accumulated (never thrown):

| Diagnostic | Condition |
|-----------|-----------|
| YAML parse error | Malformed YAML syntax |
| Missing close delimiter | No closing `---` found |
| Root is not object | YAML root is array or scalar |
| Frontmatter not found | No opening `---` at byte 0 |

Diagnostics are available on `page.frontmatter.diagnostics` for debugging and quality checks.

---

## Phase 3: Title Resolution

**Input:** Parsed frontmatter + body
**Output:** Resolved `title` string

### Resolution Cascade

```
Priority 1: frontmatter.okf.title
  └── If present and non-empty, use as title

Priority 2: First H1 heading in body
  └── Pattern: /^#\s+(.+)$/m
  └── Searches the body (after frontmatter) for the first markdown H1

Priority 3: Filename stem
  └── Extract filename without extension
  └── Replace hyphens and underscores with spaces
  └── "my-page_name.md" -> "my page name"
```

### `titleFromBody` Function

```javascript
function titleFromBody(body: string): string | undefined {
  const match = body.match(/^#\s+(.+)$/m);
  return match ? match[1].trim() : undefined;
}
```

The regex `/^#\s+(.+)$/m` matches:
- `^` -- start of line (with `m` flag)
- `#` -- literal hash
- `\s+` -- one or more whitespace characters
- `(.+)` -- capture group: everything after the hash and space
- `$` -- end of line

### Edge Cases

- **Multiple H1 headings**: Only the first one is used (the regex does not have the `g` flag)
- **H1 with inline formatting**: The entire text content is captured, including markdown formatting markers. These are not stripped -- the title is used for display, and the rendering pipeline handles formatting.
- **Empty frontmatter title**: `title: ""` is treated as absent, falling through to H1 or filename
- **No body content**: If there is no H1 and no frontmatter title, the filename stem is used

---

## Phase 4: Category Classification

**Input:** OKF `type` field + page path
**Output:** `WikiCategory` enum value

### Classification Logic (`categoryFor`)

The classifier uses a priority cascade: first checking the `type` metadata, then falling back to path prefix matching.

```mermaid
flowchart TD
    A["type metadata"] --> B{Contains person/pessoa/contact?}
    B -->|yes| C["category: person"]
    B -->|no| D{Contains directive/diretiva?}
    D -->|yes| E["category: directive"]
    D -->|no| F{Contains research/pesquisa?}
    F -->|yes| G["category: research"]
    F -->|no| H{Contains project/projeto?}
    H -->|yes| I["category: project"]
    H -->|no| J{Contains memory/memoria/memória?}
    J -->|yes| K["category: memory"]
    J -->|no| L{Path prefix match?}
    L -->|match| M[Category from prefix]
    L -->|no match| N["category: other"]
```

### Complete Mapping

| Category | Type Match (case-insensitive) | Path Prefix |
|----------|-------------------------------|-------------|
| `person` | `person`, `pessoa`, `contact` | `pessoas/` |
| `directive` | `directive`, `diretiva` | `diretivas/` |
| `research` | `research`, `pesquisa` | `pesquisas/`, `kaggle/`, `fitdnuvo/`, `clickfix/` |
| `project` | `project`, `projeto` | `infraestrutura/`, `claude_desktop/`, `ai_modelos/`, `mcp/`, `hacks/` |
| `memory` | `memory`, `memoria`, `memória` | `memorias/` |
| `other` | (default) | (no prefix match) |

### Priority Rules

1. **Type match takes precedence over path prefix**: A file at `pesquisas/paper.md` with `type: directive` is classified as `directive`, not `research`.
2. **Case-insensitive matching**: `type: "Research"` matches the `research` category.
3. **Portuguese variants**: The classifier supports both English and Portuguese type names (e.g., `pesquisa` for research, `diretiva` for directive).
4. **First match wins**: The type is checked against categories in the order shown above. If a type matches multiple categories (unlikely but possible), the first match wins.

### Impact on UI

Categories drive:
- **Graph visualization**: Each category has a distinct color and X-position in the D3 force layout [[wikifita-site-graph-construction]]
- **LLM index**: The `/llms.txt` endpoint organizes pages by category with Portuguese section headers [[wikifita-site-llm-integration]]
- **Search index**: Category is included in the searchable text, so searching "pesquisa" matches research pages

---

## Phase 5: Link Extraction and Resolution

**Input:** Parsed pages with body content
**Output:** Pages with `links[]`, `outlinks[]`, and `backlinks[]`

This is the most complex phase. It is documented in full detail in [[wikifita-site-graph-construction]].

### Summary

1. **Extract** links from the markdown AST (markdown links, wiki links, image references)
2. **Resolve** each link target through the 6-stage cascade (external check, fragment-only, exact match, extension variants, relative path, fuzzy slug)
3. **Build outlinks**: Unique resolved destinations, excluding self-links
4. **Build backlinks**: Inverse mapping from outlinks
5. **Build edge list**: Flat `{source, target}` array for graph visualization and PageRank

---

## Phase 6: Media Extraction

**Input:** Parsed pages with body content
**Output:** Pages with `media[]` array

### Extraction Sources

Media references are collected from two sources in the markdown content:

#### Source A: Markdown AST

The `extractMediaReferences` function (lines 378-404) visits `link` and `image` AST nodes:

| Node Type | Extracted Data |
|-----------|---------------|
| `link` | URL + text content as label |
| `image` | URL + alt text as label |

#### Source B: Raw HTML in Body

A regex captures `src` and `poster` attributes from HTML media elements:

```
Pattern: /<(iframe|video|audio|source|img)\b[^>]*\s(?:src|poster)=["']([^"']+)["'][^>]*>/gi
```

This catches media elements written as raw HTML in the markdown, which the AST parser may not represent as standard `link`/`image` nodes.

### URL Classification (`classifyMediaUrl`)

Each extracted URL is classified by `lib/media.ts`:

```mermaid
flowchart TD
    A[URL] --> B{HTTPS?}
    B -->|no| C["allowed: false, reason: remote_media_requires_https"]
    B -->|yes| D{Extension match?}
    D -->|".avif,.gif,.jpg,.jpeg,.png,.svg,.webp"| E["kind: image, allowed: true"]
    D -->|".m4v,.mov,.mp4,.webm"| F["kind: video, allowed: true"]
    D -->|".flac,.m4a,.mp3,.ogg,.wav"| G["kind: audio, allowed: true"]
    D -->|YouTube/Vimeo URL| H["kind: embed, allowed: true, embedUrl: privacy-rewritten"]
    D -->|no match| I["Not collected (ignored)"]
```

### YouTube/Vimeo URL Rewriting

| Input URL | Output Embed URL |
|-----------|-----------------|
| `youtube.com/watch?v=ID` | `youtube-nocookie.com/embed/ID` |
| `youtu.be/ID` | `youtube-nocookie.com/embed/ID` |
| `youtube.com/shorts/ID` | `youtube-nocookie.com/embed/ID` |
| `vimeo.com/ID` | `player.vimeo.com/video/ID` |

### Deduplication

Media references are deduplicated by the composite key `${kind}:${url}:${label}`. If the same URL appears multiple times in a page (e.g., an image used twice), only one reference is stored.

### Security

- **HTTPS only**: Non-HTTPS URLs are blocked with `allowed: false`
- **No local assets**: All media must be remote URLs. There is no local asset storage, upload pipeline, or bucket.
- **Privacy-enhanced embeds**: YouTube URLs use `youtube-nocookie.com` to prevent tracking

---

## Phase 7: PageRank Computation

**Input:** Pages with resolved links
**Output:** Pages with `pagerank` field

The final phase computes PageRank over the directed link graph. This is documented in full detail in [[wikifita-site-pagerank]].

### Summary

1. Build `outlinksByPath` from each page's `outlinks` array
2. Run iterative PageRank with $d = 0.85$, tolerance $10^{-10}$, max 200 iterations
3. Normalize ranks to sum to 1.0
4. Store each page's rank in `pagerank` field

---

## Corpus Assembly

After all 7 phases, the final `WikiCorpus` object is assembled:

```typescript
type WikiCorpus = {
  status: WikiSourceStatus;         // Source metadata
  pages: WikiPage[];                // All processed pages, sorted by path
  pageByPath: Map<string, WikiPage>; // O(1) lookup by path
  links: WikiGraphLink[];           // {source, target} edge list
};
```

### WikiSourceStatus

```typescript
type WikiSourceStatus = {
  repository: string;     // "aleffita/wikifita"
  branch: string;         // "main"
  commit: string;         // Git SHA
  commitDate: string;     // ISO date of commit
  loadedAt: string;       // ISO date of corpus build
  source: "github" | "snapshot";
  state: "live" | "cached" | "degraded";
  error?: string;         // Error message if degraded
};
```

### Client Data Serialization

Before sending to the browser, the corpus is serialized to `WikiClientData`:

```typescript
type WikiClientPage = Omit<WikiPage, "raw" | "body" | "links"> & {
  links: Array<Pick<WikiLink, "target" | "label" | "fragment" | "destination" | "state">>;
};
```

This strips:
- `raw` (full source markdown -- too large for client)
- `body` (markdown without frontmatter -- unnecessary on client)
- `links[].origin` (internal tracking data)

The client receives enough data for graph visualization, search, and link display without exposing the full markdown content.

---

## Caching Strategy

| Cache | Scope | Key | Invalidation | Location |
|-------|-------|-----|-------------|----------|
| `liveCache` | Module-level | `{repo}:{branch}:{sha}` | New commit SHA | Memory |
| `snapshotCache` | Module-level | `{commit}:{state}:{error}` | New snapshot | Memory |
| `blobCache` | Module-level | Blob SHA | Never (within process) | Memory |
| `searchCache` | WeakMap | Corpus reference | New corpus instance | Memory |

### In-Memory Caching by Commit SHA

The primary caching mechanism is SHA-based deduplication:

```javascript
if (liveCache.key === `${repo}:${branch}:${sha}`) {
  return liveCache.corpus;  // Cache hit: no API calls
}
```

This means:
- The first request after a commit builds the full corpus (~100-500ms depending on corpus size)
- Subsequent requests with the same commit SHA return in <1ms
- A new commit invalidates the cache and triggers a full rebuild

### Blob-Level Caching

Individual blob content is cached by SHA. When a new commit changes only some files, unchanged files are served from the blob cache. This reduces GitHub API calls on incremental updates.

---

## Testing

The corpus pipeline is covered by unit tests in `tests/wiki-pipeline.test.ts`:

| Test | What It Verifies |
|------|-----------------|
| Frontmatter parsing | OKF fields extracted correctly, YAML errors captured as diagnostics |
| YAML error handling | Malformed YAML produces diagnostics, not crashes |
| Link resolution (markdown) | Standard markdown links resolve correctly |
| Link resolution (wiki) | `WIKI_LINK_TARGET_LABEL` patterns resolve correctly |
| Code fence ignoring | Links inside code blocks are not extracted |
| MDX support | `.mdx` files are processed identically to `.md` |
| Media classification | URLs classified correctly by extension and provider |
| PageRank convergence | Ranks sum to 1.0, dangling nodes behave correctly |
| BM25 accent-insensitive | "cafe" matches "Café" after normalization |
| Alias resolution | Fuzzy slug matching resolves to correct page |
| Live/snapshot equivalence | Both sources produce identical corpus for the same content |

---

## Related Pages

- [[wikifita-site-pagerank]] -- Phase 7 deep-dive
- [[wikifita-site-graph-construction]] -- Phase 5 deep-dive
- [[wikifita-site-search-system]] -- How the corpus feeds the search index
- [[wikifita-site-rendering-pipeline]] -- How page.body is rendered to HTML
- [[wikifita-site-deployment]] -- How the corpus is loaded in Cloudflare Workers
- [[wikifita-site-architecture]] -- Full technical architecture