WikifitaGitHub live67e8de5
outro · wikifita-site-architecture/wikifita-site-llm-integration

Wikifita Site LLM Integration

How the Wikifita Atlas exposes structured plain-text endpoints for LLM consumption: site-wide indexes, per-page summaries, and full lossless context dumps.

Baixar raw

Wikifita Site LLM Integration

The Wikifita Atlas provides a layered set of plain-text endpoints designed for consumption by LLMs and AI agents. These endpoints follow the emerging llms.txt convention (inspired by robots.txt and sitemap.xml) to make the knowledge base machine-readable without requiring HTML parsing.

The design philosophy aligns with the wikifita principle that knowledge should be accessible to both humans and agents. The site serves as a bridge: humans interact with the rendered UI, while agents consume the structured text endpoints.

Endpoint Hierarchy

flowchart TD
    A[LLM Agent] --> B{/llms.txt}
    A --> C{/llms-full.txt}
    A --> D{/wiki/page/llms.txt}
    A --> E{/wiki/page/llms-full.txt}

    B -->|Reduced index| F[Page list by category with titles + URLs]
    C -->|Full dump| G[All raw markdown with metadata headers]
    D -->|Per-page summary| H[Page metadata + links + backlinks + excerpt]
    E -->|Per-page full| I[Page metadata + raw markdown + link resolution]

    F -->|Entry point| J[Agent discovers pages]
    J -->|Picks page| D
    D -->|Needs full content| E
    C -->|Bulk ingestion| K[Agent loads entire corpus]

Endpoint 1: /llms.txt (Site Index)

Content-Type: text/plain; charset=utf-8 Cache-Control: no-store

A reduced site index listing all pages organized by category. This is the entry point for agents discovering the knowledge base.

Output Format

# Wikifita Atlas

> Atlas editorial da Wikifita, a base privada e versionada de memória persistente...

## Principais
- [Atlas](https://wikifita.aleffita.dev)
- [Busca](https://wikifita.aleffita.dev/search)
- [Contexto completo lossless](https://wikifita.aleffita.dev/llms-full.txt)

## Memórias
- [Memory Index](https://wikifita.aleffita.dev/wiki/memorias/MEMORY.md): Memory index page

## Projetos
- [Project Alpha](https://wikifita.aleffita.dev/wiki/projetos/alpha): Project description

## Pesquisas
- [Research Topic](https://wikifita.aleffita.dev/wiki/pesquisas/topic): Research description

Suppression Rules

Pages are excluded from /llms.txt if their filename stem (without extension) matches:

  • claude
  • agents
  • prefill
  • template

This prevents internal configuration files from appearing in the public LLM index.

Section Names

Categories are translated to Portuguese section headers:

CategorySection Name
memoryMemórias
projectProjetos
researchPesquisas
directiveDiretivas
personPessoas
otherOutros

Endpoint 2: /llms-full.txt (Full Corpus Dump)

Content-Type: text/plain; charset=utf-8 Cache-Control: no-store

A complete dump of every wiki page's raw markdown, delimited by structured metadata headers. This is the "lossless" endpoint -- the raw content is preserved byte-for-byte, including frontmatter, whitespace, and newlines.

Output Format

# Wikifita Atlas — Full Lossless Context

> Extensão lossless da Wikifita para leitores autorizados e agentes de pesquisa.

Cada documento abaixo é delimitado por path, URL, byte length e hash SHA-256...

--- WIKIFITA_DOCUMENT path="memorias/MEMORY.md" url="https://wikifita.aleffita.dev/wiki/memorias/MEMORY.md" bytes="1234" sha256="abc123..." ---
# Memory Index
---
type: memory
title: "Memory Index"
...
---

Body content here...

--- END_WIKIFITA_DOCUMENT path="memorias/MEMORY.md" ---

--- WIKIFITA_DOCUMENT path="projetos/alpha.md" url="https://wikifita.aleffita.dev/wiki/projetos/alpha" bytes="5678" sha256="def456..." ---
...
--- END_WIKIFITA_DOCUMENT path="projetos/alpha.md" ---

Document Delimiters

Each document is wrapped in structured delimiters that enable agents to:

  • Parse individual documents from the stream
  • Verify content integrity via SHA-256 hash
  • Know the canonical URL and byte length
  • Extract the raw markdown including frontmatter

The delimiter format:

--- WIKIFITA_DOCUMENT path="{path}" url="{canonicalUrl}" bytes="{byteLength}" sha256="{hash}" ---
{raw markdown}
--- END_WIKIFITA_DOCUMENT path="{path}" ---

Use Cases

  • Bulk ingestion: An agent can fetch /llms-full.txt once and have the entire knowledge base in structured form
  • RAG pipeline seeding: The document delimiters enable clean chunking for vector store insertion
  • Content verification: SHA-256 hashes allow agents to verify they have the canonical version

Endpoint 3: /wiki/{path}/llms.txt (Per-Page Summary)

Content-Type: text/plain; charset=utf-8 Cache-Control: no-store

A structured summary of a single page, designed as a lightweight context window for agents working with a specific document.

Output Format

# Page Title

> Wikifita page context for https://wikifita.aleffita.dev/wiki/path/to/page

- canonical: https://wikifita.aleffita.dev/wiki/path/to/page
- raw: https://wikifita.aleffita.dev/raw/path/to/page.md
- path: path/to/page.md
- type: reference
- category: project
- sha256: abc123...
- byte_length: 1234
- description: One-line description
- tags: tag1, tag2

## Summary

First 240 characters of the page body or description...

## Links
- [Link Label](https://wikifita.aleffita.dev/wiki/target) -> resolved
- [Broken Link](missing-page) -> missing

## Backlinks
- [Source Page](https://wikifita.aleffita.dev/wiki/source)

## Media References
- image: https://cdn.example.com/photo.png (Photo caption)
- embed: https://youtu.be/ID (Video title)

## Full Context

- [Lossless raw page context](https://wikifita.aleffita.dev/wiki/path/to/page/llms-full.txt)

Metadata Block

The header provides machine-readable metadata:

  • canonical: Full URL for the page
  • raw: URL to download the raw markdown
  • path: Internal file path
  • type: OKF type from frontmatter
  • category: Computed category (person, project, research, directive, memory, other)
  • sha256: Content hash for integrity verification
  • byte_length: Exact byte count
  • description: From frontmatter (if present)
  • tags: Comma-separated tag list

Link Resolution

Links in the output are resolved to their canonical state:

  • Resolved links show the canonical URL
  • Missing links show the state string "missing"
  • External links show the state string "external"

Backlinks

The backlinks section lists all pages that link to the current page, with their canonical URLs. This enables agents to understand the document's position in the knowledge graph.

Endpoint 4: /wiki/{path}/llms-full.txt (Per-Page Full Context)

Content-Type: text/plain; charset=utf-8 Cache-Control: no-store

The full lossless representation of a single page, including the raw markdown and complete link resolution data.

Output Format

# Page Title — Full Lossless Context

--- WIKIFITA_PAGE path="path/to/page" url="https://..." raw="https://..." bytes="1234" sha256="abc..." blob="def..." ---
source="github" state="live" commit="abc123" loaded_at="2026-07-21T..."
links=[{"target":"...","label":"...","state":"resolved","destination":"target.md","canonicalUrl":"https://..."}]
media=[{"url":"https://...","kind":"image","allowed":true}]

# Full raw markdown content including frontmatter...

--- END_WIKIFITA_PAGE path="path/to/page" ---

Structured Metadata Header

The WIKIFITA_PAGE header contains:

  • path: Internal file path
  • url: Canonical URL
  • raw: URL to raw markdown
  • bytes: Byte length
  • sha256: Content hash
  • blob: Git blob SHA

The second line provides source metadata:

  • source: github or snapshot
  • state: live, cached, or degraded
  • commit: Git commit SHA
  • loaded_at: ISO timestamp of corpus load

The links and media fields are JSON-encoded arrays with full resolution data, enabling agents to:

  • Walk the link graph programmatically
  • Understand which links are resolved, missing, or external
  • Access canonical URLs for each linked page
  • Identify media references and their types

Routing Mechanism

Proxy Rewrite

Per-page LLM endpoints are accessible via two URL patterns:

  1. Proxy pattern: /wiki/{path}/llms.txt and /wiki/{path}/llms-full.txt
  2. API pattern: /api/page-llms/llms/{path} and /api/page-llms/llms-full/{path}

The proxy.ts middleware intercepts requests matching the proxy pattern and rewrites them to the API pattern. This allows agents to discover per-page LLM endpoints by appending /llms.txt to any wiki page URL, following a consistent convention.

// proxy.ts
const match = pathname.match(/^\/wiki\/(.+)\/(llms(?:-full)?\.txt)$/);
// Rewrites: /wiki/path/to/page/llms.txt -> /api/page-llms/llms/path/to/page

Content Negotiation

The per-page endpoint uses the mode path segment to determine output format:

ModeFunctionOutput
llmspageLlmsText()Structured summary with metadata, links, backlinks
llms-fullpageLlmsFullText()Full lossless context with raw markdown

Integration with Wikifita Philosophy

Agent-Consumable Knowledge

The LLM endpoints embody the wikifita principle that a knowledge base should be accessible to both humans and agents:

  • Humans use the rendered UI (Next.js + D3 graph + search)
  • Agents use the plain-text endpoints (/llms.txt, /llms-full.txt, per-page)
  • Both consume the same canonical source (the git repository)

Layered Access

The endpoint hierarchy provides layered access:

LayerEndpointContentUse Case
Discovery/llms.txtPage list by categoryAgent finds relevant pages
Summary/wiki/{path}/llms.txtMetadata + links + excerptAgent understands page context
Full/wiki/{path}/llms-full.txtRaw markdown + full metadataAgent ingests complete content
Bulk/llms-full.txtAll pages concatenatedAgent loads entire corpus

Content Integrity

Every endpoint includes integrity signals:

  • X-Wikifita-Commit: Git commit SHA for the current corpus
  • X-Wikifita-Source-State: Whether the data is live, cached, or degraded
  • ETag: Deterministic hash for cache validation
  • SHA-256 hashes in document delimiters for content verification

This enables agents to verify they are working with canonical, unmodified content.

Filtering and Suppression

Site-Wide Index (/llms.txt)

Suppresses pages with stems: claude, agents, prefill, template.

Per-Page Summaries (/wiki/{path}/llms.txt)

Suppresses pages with stems: claude, agents, prefill, template.

Note: The full corpus dump (/llms-full.txt) does NOT suppress any pages -- it includes everything for completeness.

Rationale

The suppressed filenames correspond to internal configuration files (like CLAUDE.md, AGENTS.md) that are part of the repository but not meaningful content for external agents. They are excluded from discovery endpoints but remain accessible via direct URL.

Example Agent Workflow

An agent discovering and consuming a wiki page:

1. GET /llms.txt
   -> Discovers "Pesquisas" section
   -> Finds "Unit Distance Conjecture" page

2. GET /wiki/pesquisas/unit-distance/llms.txt
   -> Reads metadata: type=research, tags=[math, conjecture]
   -> Reads summary: first 240 chars
   -> Sees links to related pages
   -> Decides to read full content

3. GET /wiki/pesquisas/unit-distance/llms-full.txt
   -> Reads complete raw markdown
   -> Parses WIKIFITA_PAGE header for integrity
   -> Walks link array to find related documents

4. GET /raw/pesquisas/unit-distance.md
   -> Downloads raw markdown for local processing
   -> Verifies SHA-256 matches from step 3

Limitations

  1. No authentication: The LLM endpoints are publicly accessible. Sensitive content must be on the personal branch (never deployed).

  2. No pagination: /llms-full.txt returns the entire corpus. For very large wikis, this could exceed agent context windows. The per-page endpoints provide a more granular alternative.

  3. No streaming: All endpoints return the complete response at once. There is no SSE or chunked transfer for large payloads.

  4. No content transformation: The raw markdown is served as-is. There is no summarization, chunking, or embedding generation at the endpoint level. Agents are expected to process the raw content themselves.

  5. Portuguese section names: The /llms.txt index uses Portuguese section headers (Memórias, Projetos, etc.) reflecting the wiki's primary language. Agents should be aware of this for category-based navigation.

Related Pages