WikifitaGitHub live67e8de5
projeto · infraestrutura/litellm-proxy-migration

Migration: Custom Gateway to LiteLLM Proxy Server

Documentation: Migration: Custom Gateway to LiteLLM Proxy Server

Baixar raw

Migration: Custom Gateway to LiteLLM Proxy Server

Migration journey from custom FastAPI gateway to LiteLLM Proxy Server as the main app.

Timeline

Phase 1: Custom FastAPI (Jun 2026)

Original gateway in gateway.py:

  • FastAPI app with manual Tier x Effort routing
  • router.py (304L) — (Tier, Effort) → Backend table
  • handlers/messages.py — manual HTTP forwarding
  • services/forwarding.py + services/fallback.py — custom proxy
  • core/translation.py — manual Anthropic to OpenAI translation
  • MiMo provider registered via monkey-patch in ProviderConfigManager

Phase 2: LiteLLM Router as Library

  • litellm_router.py — singleton loading model_list from YAML
  • router.acompletion() replaces manual forwarding
  • Hooks registered in hooks/pipeline.py (pre_call, post_call)
  • Still a separate FastAPI app using LiteLLM as a library

Phase 3: Proxy Server as Main App (Jul 2026)

  • app = litellm.proxy.proxy_server.app — the proxy IS the gateway
  • 65+ native routes: /v1/messages, /v1/models, /health, /ui, /key/*, /user/*, /spend/*
  • Custom routes added: voice WS, MCP, search, API handlers
  • ~2400 lines of dead code deleted
  • Functional admin UI with PostgreSQL

Critical Discoveries

CONFIG_FILE_PATH

The proxy reads CONFIG_FILE_PATH at startup, NOT LITELLM_CONFIG_PATH. This was the bug that caused "Model list not initialized" for hours.

# proxy_server.py, linha ~786:
env_config_yaml = get_secret_str("CONFIG_FILE_PATH")
if env_config_yaml is not None and os.path.isfile(env_config_yaml):
    llm_router, llm_model_list, general_settings = await proxy_config.load_config(
        router=llm_router, config_file_path=env_config_yaml
    )

Prisma + PostgreSQL

  • LiteLLM's schema.prisma uses provider = "postgresql" — SQLite doesn't work
  • prisma generate needs to run with the litellm_proxy_extras schema
  • libatomic1 required in Dockerfile (Node.js binaries)
  • Migrations: prisma migrate deploy creates 71 tables

Model list in YAML vs model_info

  • model_list in config YAML defines the models
  • model_info in each entry defines capabilities (supports_vision, supports_function_calling, supports_1m)
  • Proxy serves /v1/models in OpenAI format; middleware translates to Anthropic
  • Capabilities derived from model_info + tier (haiku→low/med, sonnet→+high, opus→all)

Admin UI

  • Served at /ui by the proxy
  • Login: UI_USERNAME / UI_PASSWORD from .env
  • Internal routes (/v2/model/info, /model/cost_map/source, /key/list, /spend/logs/ui)
  • allowed_routes is an Enterprise feature — remove to avoid blocking internal routes

MiMo Provider

Registered BEFORE the proxy app import:

from providers.mimo import register_mimo_provider
register_mimo_provider()  # patch ProviderConfigManager
import litellm.proxy.proxy_server
app = litellm.proxy.proxy_server.app

MiMoAnthropicMessagesConfig:

  • Uses x-api-key header (not Authorization: Bearer)
  • API base: https://token-plan-sgp.xiaomimimo.com/anthropic
  • Strips [1m] from model name before sending
  • Full URL: .../anthropic/v1/messages

DeepSeek Provider:

  • Uses deepseek/deepseek-v4-pro (not deepseek/anthropic/)
  • The /anthropic/ prefix in model name was rejected by the API
  • API base: https://api.deepseek.com/anthropic

References