Modern applications use multiple LLM providers: OpenAI for GPT-4o, Anthropic for Claude, Google for Gemini, self-hosted LLaMA via vLLM. Each has a different API, different pricing, different rate limits, and different failure modes. Without a gateway, your application code is littered with provider-specific logic, retry handling, and cost tracking. An LLM gateway sits between your application and the LLM providers, providing: unified API (one interface for all providers), routing (send requests to the right model), fallbacks (switch providers on failure), rate limiting (prevent quota exhaustion), caching (avoid duplicate calls), and observability (logging, cost tracking, latency metrics).

LiteLLM is the most popular open-source LLM gateway. It provides an OpenAI-compatible API that proxies to 100+ LLM providers (OpenAI, Anthropic, Google, Cohere, Mistral, Azure, Bedrock, self-hosted vLLM, Ollama, and more). Key features: unified interface (call any model with the same completion() function), load balancing (distribute across multiple API keys or deployments), fallbacks (automatic retry with alternative models on failure), budget management (set spend limits per user, team, or API key), and virtual keys (create API keys with custom rate limits and budgets). LiteLLM is MIT-licensed and can be self-hosted as a proxy server or used as a Python library.

Portkey is a managed LLM gateway with enterprise-grade features. Beyond basic routing and fallbacks, Portkey offers: semantic caching (cache responses for semantically similar queries, reducing costs by up to 40%), guardrails (built-in content filtering and PII detection), detailed analytics (cost, latency, token usage dashboards per model, user, and team), request/response logging (full audit trail for compliance), and conditional routing (route based on request metadata, user tier, or content type). Portkey is SaaS-first with a self-hosted enterprise option. It’s the right choice for teams that want a managed solution with advanced observability.

Not every request needs GPT-4o. Smart routing saves money and reduces latency: Complexity-based routing — simple questions go to a cheap, fast model (GPT-4o-mini, Claude Haiku); complex reasoning goes to a powerful model (GPT-4o, Claude Sonnet). Cost-based routing — route to the cheapest model that meets quality requirements. Latency-based routing — route to the fastest available provider. Load-based routing — distribute across providers to stay under rate limits. Content-based routing — code questions to one model, creative writing to another. A classifier model (small, fast) can analyze the request and decide which model to route to.

Every LLM provider enforces rate limits: RPM (requests per minute), TPM (tokens per minute), and RPD (requests per day). Hit the limit and you get HTTP 429 errors. Without management, a single user or feature can exhaust your entire quota. The gateway handles this with: per-user rate limits (each user gets a fair share), per-team budgets (engineering gets $5K/month, marketing gets $2K/month), token-level throttling (limit tokens per minute, not just requests), queue-based smoothing (queue excess requests instead of rejecting), and multi-key distribution (spread requests across multiple API keys to increase effective limits).

LLM calls are expensive and slow. Caching avoids redundant calls: Exact-match caching — cache the response for identical prompts. Simple, effective for deterministic queries (temperature=0). Semantic caching — embed the prompt, find similar cached prompts within a threshold, return the cached response. Catches paraphrases. Prompt prefix caching — some providers (Anthropic, OpenAI) cache the KV states of long system prompts, so only the user message is processed. Reduces latency and cost for apps with long, static system prompts. RAG caching — cache retrieved documents so the same query doesn’t hit the vector database again. Layer these: prompt prefix caching (provider-side) + semantic caching (gateway-side) + application-level caching.

LLM costs are per-token, which makes them hard to predict. A gateway provides visibility: cost per request (input tokens × input price + output tokens × output price), cost per user/team (who’s spending the most?), cost per feature (which product feature drives the most LLM spend?), and cost trends (is spend growing linearly or exponentially?). Optimization levers: model selection (use cheaper models where quality allows), prompt optimization (shorter prompts = fewer tokens = lower cost), caching (avoid duplicate calls), output length limits (set max_tokens appropriately), and batch processing (some providers offer 50% discounts for async batch API calls).

Choose LiteLLM if: you want open-source, self-hosted control, need to support many providers, and have engineering capacity to operate it. Choose Portkey if: you want a managed service with semantic caching, advanced analytics, and enterprise compliance features. Choose OpenRouter if: you want a simple pay-per-use proxy without running infrastructure. Build your own if: you have very specific routing logic, need deep integration with internal systems, or have strict data residency requirements. Most teams should start with LiteLLM (free, self-hosted, covers 90% of needs) and evaluate Portkey if they need advanced observability or semantic caching.