Intro
The moment your app calls more than one model provider, the plumbing gets ugly. Different SDKs, different auth, no shared view of spend, and when one provider has a bad afternoon your bot just goes dark. The fix is an LLM gateway: one endpoint in front of every provider that handles routing, fallbacks, cost control, and governance so your app talks to one thing instead of a dozen. There are a lot of options now, commercial and open source, and they serve different teams. So lately I ask myself: “how do you pick the right one?” Let’s break it down.
Note: We left out the cloud hyperscaler gateways (Bedrock, Vertex, Azure AI,etc) as they’re too rigid for the multi-provider, multi-cloud setup this post is about.
The field at a glance
| Tool | Camp | License | Built on | Best for |
|---|---|---|---|---|
| LiteLLM | open-source | MIT | Python | Widest provider coverage, self-hosted |
| Helicone | open-source | Apache 2.0 | Rust | One-line observability + caching |
| Bifrost | open-source | Apache 2.0 | Go | Microsecond overhead at scale |
| LLM Gateway (llmgateway.io) | open-source | AGPLv3 + EE | TypeScript | Self-hosted OpenRouter alternative |
| Portkey | managed-first | Apache 2.0 core | TypeScript | Config-driven reliability + observability |
| Kong AI Gateway | managed-first | OSS core + Konnect | Nginx + plugins | Enterprise governance on an existing API mesh |
| TrueFoundry | managed | Proprietary | Kubernetes | Full MLOps suite, not just routing |
| Merge Gateway | managed-first | Proprietary | Hosted SaaS | Governance + audit by default |
| OpenRouter | marketplace | Proprietary | Hosted marketplace | Breadth: one key, 300+ models |
Open Source
These lead with self-hosting. The code is the product; the paid tier (if any) is for support, SSO, audit and compliance.
I. LiteLLM: The Community Default
A lightweight Python library and proxy (MIT) that calls 100+ providers in the OpenAI format. pip install it or run the container and you own the gateway end to end. Think “host your own OpenRouter.” It translates API parameters and error codes into one consistent shape, so swapping Claude for Gemini is trivial. Out of the box: virtual keys, per-team budgets, spend tracking, load balancing, guardrails, and an admin dashboard. Current stable line is 1.87.x.
📒 Docsdocs.litellm.ai↗Pros
- Widest provider coverage and a neutral, vendor-agnostic abstraction.
- Full ownership: runs anywhere, no log limits or request quotas.
- Core is free; Enterprise tier (SSO, audit, JWT) starts around
$250/mo.
Cons
- Ops burden at scale: wants PostgreSQL + Redis plus ongoing patching.
- Python overhead: GIL and interpreter cost mean multiple instances behind a load balancer past a few hundred to a few thousand RPS (per competitor benchmarks).
Best for developers and startups wanting a customizable proxy, with the DevOps capacity to run it.
II. Helicone: The One-Liner
Observability-first. Change your base URL and you instantly get cost, latency, token usage, and user-level attribution, with routing and failover layered on. The cost lever is caching: for repetitive workloads, a cache hit serves instantly and kills token spend. Apache 2.0 and self-hostable via Docker or Helm for VPC deployments.
📒 Docsdocs.helicone.ai↗$79/mo, Team $799/mo.Pros
- Fastest path to observability: one URL change.
- Caching cuts cost on repetitive traffic.
- Open source, self-hostable for data control.
Cons
- Shallower tracing than dedicated observability tools (request-level, not span-level).
- Ownership change to factor into long-term plans.
Best for product-led teams needing granular cost attribution and developer-friendly debugging.
III. Bifrost: The Speed Demon
The newer, performance-obsessed entrant: open source (Apache 2.0), written in Go by Maxim AI, built so the gateway adds near-zero overhead. Ships as a single binary, starts with zero config (npx or Docker), covers ~20+ providers behind an OpenAI-compatible API. Loaded for production: adaptive load balancing, hierarchical governance, clustering for HA, dual-layer semantic caching (hash + vector, default threshold ~0.8), MCP gateway, vault, and audit logs.
📒 Docsdocs.getbifrost.ai↗Pros
- Microsecond overhead and stable memory under sustained load.
- Single-binary deploy, straightforward setup.
- Good semantic caching and full governance/clustering.
Cons
- Smaller community than LiteLLM.
- Fewer integrations.
- Unclear paywall line on enterprise features (see heads-up).
Best for production deployments where performance and stability at scale are the priority.
IV. LLM Gateway (llmgateway.io): The OSS OpenRouter
The cleanest open-source answer to OpenRouter. The project at llmgateway.io (by theopenco) gives you one OpenAI-compatible endpoint across 35+ providers and 280+ models (per its docs), with API-key management, usage analytics (requests, tokens, latency, cost), caching, and automatic failover. It goes beyond text routing, too: standard REST endpoints for embeddings, speech, moderation, and image generation/editing, plus an MCP server you can wire straight into Claude Code, Cursor, or Codex.
ee/ directory need a paid Enterprise license. Two notes. First, AGPLv3 is strong copyleft: if you modify it and offer it as a network service, you must publish your changes, a different posture from LiteLLM (MIT) or Helicone and Bifrost (Apache 2.0). Second, some capabilities are gated: Enterprise covers advanced billing, unlimited data retention (the open core keeps 30 days), team and organization management, custom provider-key configs, and priority support, with hosted plans adding SSO/RBAC, white-label, and custom SLAs.Pros
- True OSS aggregator: one API for many providers, self-host or hosted, BYOK free.
- Multimodal, not just text: image gen (Qwen, Gemini), speech, embeddings, and an MCP server for Claude Code or Cursor.
- Analytics built in: per-request tokens, latency, and cost out of the box.
Cons
- Young and small: ~1.3k stars, limited track record next to LiteLLM.
- AGPLv3 copyleft may not suit teams that modify and serve.
- Key features gated: retention, team management, and SSO sit behind the Enterprise license.
Best for teams that want OpenRouter’s “one key, many models” convenience but self-hosted and open source, and who can live with AGPL or buy the enterprise tier.
Commercial / Managed-First
These are SaaS. The center of gravity is the managed platform, the support contract, and the compliance story.
V. Portkey: The Control Plane
Based on its open source self hosted version, this middleware that adds reliability and visibility without touching app code. Its signature is the Config object: you declare routing logic (retries with exponential backoff, multi-model fallbacks) as configuration, not code. Around it sit deep observability, Virtual Keys (scoped per team or environment so one experiment can’t drain the org budget), prompt versioning, and an MCP gateway.
📒 Docsportkey.ai/docs↗ 📦 Repogithub.com/Portkey-AI/gateway↗$49/mo included. Model/provider counts are vendor-reported and inconsistent across Portkey’s own docs.Pros
- Config-driven reliability: fallbacks and retries without code changes.
- Strong observability: single pane for cost, latency, errors across providers.
- Open-source core now under Apache 2.0.
Cons
- Now owned by Palo Alto (folded into Prisma AIRS): roadmap may drift from developer DX toward enterprise security.
- Vendor-reported catalog numbers; verify the live model list.
Best for SRE and DevOps teams building resilient, high-availability AI systems with deep monitoring.
VI. Kong AI Gateway: The Enterprise Incumbent
If you already run Kong for API traffic, the AI Gateway is the path of least resistance. It is a plugin layer on Kong’s Nginx core, so you govern LLM traffic with the same machinery you use for web traffic. The AI plugins are the draw: a PII Sanitizer (20 categories, 9 languages) applied before a prompt leaves for an external provider, Semantic Prompt Guard, semantic caching, AI rate limiting, and semantic load balancing that routes by the meaning of the input. Recent releases added MCP and agent-to-agent governance.
📒 Docsdeveloper.konghq.com/ai-gateway↗Pros
- Enterprise governance out of the box: PII redaction, policy enforcement, audit.
- Reuses your existing Kong mesh and plugins.
- Open-source self-hosted core.
Cons
- Heavy: assumes Kong infrastructure underneath.
- Opaque pricing: full features run through Konnect (multi-dimensional; third-party estimates exceed $30 per million requests at scale).
- Plugin velocity: custom logic has historically meant Lua (a knock raised mostly by competitors).
Best for large organizations and platform engineers managing AI traffic alongside a complex microservices ecosystem.
VII. TrueFoundry: The Whole Platform
Not really a gateway, but a full LLMOps platform with gateway functionality inside it. Kubernetes-native and bring-your-own-cloud: it deploys into your own AWS, GCP, Azure, on-prem, or air-gapped account. Alongside the AI Gateway you get GPU model serving (vLLM, SGLang), fine-tuning, agent frameworks, MCP, and prompt management. The gateway reports roughly 3 to 4 ms latency and 350+ RPS on one vCPU (vendor-reported).
Recently it has pushed into the agent layer: an Agent Gateway and MCP Gateway alongside the LLM Gateway, plus the June 2026 acquisition of Seldon AI to pull real-time ML serving under one Kubernetes-native control plane.
Pros
- Full lifecycle: serving, fine-tuning, agents, gateway in one platform.
- BYO-cloud, including air-gapped, good for regulated industries.
- Kubernetes-native for teams already deep in K8s.
Cons
- Overkill if you only need routing.
- Heavy setup and learning curve; platform tax on everything.
- Aimed at large enterprises, not startups.
Best for ML teams that need a full platform and already use Kubernetes extensively.
VIII. Merge Gateway: The Governance Play
The new kid on the block, launched March 31, 2026 by Merge, the unified-API company behind Merge Unified and Agent Handler. It sits between your app and the providers (OpenAI, Anthropic, Google, xAI, etc) behind one OpenAI-compatible endpoint at gateway.merge.dev/v1, a base-URL swap, with a native SDK if you want a deeper integration.
Build Your Own Router feature. Instead of trusting a black-box ML router, you define what “best” means by weighting benchmarks (HumanEval, MMLU, leaderboards like LMSYS Arena Elo or your own scores). The logic lives in your config, and every decision comes with a plain-language explanation of which model won and why.
Governance: budgets per account, team, or customer, semantic caching, PII filtering and DLP, per-customer region and model policies, and an audit log of every request. Addressing the “governance failures” according to CEO Shensi Ding.
📒 Docsmerge.dev/gateway↗$0.05 per million tokens. What the steady-state platform fee looks like after the promo is not spelled out, and that deadline is days away as of writing, so check the live pricing page before planning around it.Pros
- Transparent routing: config-based benchmark weighting, not a black-box model, with a plain-language reason for every pick.
- Governance-first: per-customer budgets, region and model policies, DLP, and a full audit trail by default.
- Enterprise credentials early: SOC 2, ISO 27001, GDPR, and HIPAA, plus a one-line base-URL swap or native SDK.
Cons
- Very new: limited production track record.
- Proprietary, hosted-only; no self-host option.
- Post-promo pricing unclear: the steady-state platform fee after the 12-month offer is not published.
Best for teams whose primary driver is governance and audit, especially regulated or multi-tenant SaaS, that want budgets and DLP without building them.
IX. OpenRouter: The Marketplace
The odd shape in this lineup: less a control plane, more a model marketplace. One API key and one balance buy access to 300+ models from 60+ providers behind an OpenAI-compatible endpoint, with automatic routing, fallback (billed only for the successful run), and per-model price and latency comparison. Founded in 2023, it is now the default aggregation layer for a lot of developers: 8 million users, ~100T tokens per month, a $113M Series B at ~$1.3B in May 2026.
Its latest MCP server for Claude Code, Cursor, or Codex: hands free lookups to let your agent pick a model from live benchmarks and pricing, and chat-send runs the job through it (billed against a scoped key). Handy for choosing models; for production you still call the API directly.
Pros
- Breadth: 300+ models, 60+ providers, one key, unified billing.
- MCP picker and runner: an official server ranks models on live data (free lookups) and runs the job through the chosen one via chat-send (billed).
- No inference markup; pay-as-you-go, no minimum commitment.
Cons
- Marketplace fees (5.5% credit fee, 5% BYOK tail past 1M req/mo) need to be in your unit economics.
- Lighter governance than a control plane: request-level visibility, not deep per-team policy.
- No client-side retry/backoff or free-tier SLA; you build resilience around it.
Best for developers who want the widest model catalog behind one key, fast, for prototyping and breadth-first apps where governance depth is secondary.
🟢Summary
Head-to-head at a glance
| Tool | Standout strength | Main trade-off | Entry price |
|---|---|---|---|
| LiteLLM | Widest coverage, neutrality | Python overhead + ops burden | Free → ~$250/mo |
| Helicone | One-line observability + caching | Shallow tracing; Mintlify-owned | Free → $79/mo |
| Bifrost | Microsecond overhead, single binary | Smaller community, fewer integrations | Free → enterprise |
| LLM Gateway (llmgateway.io) | Open-source OpenRouter alternative, analytics built in | Young/small; AGPLv3 copyleft | Free self-host → enterprise |
| Portkey | Config-driven reliability + observability | Now Palo Alto-owned (Prisma AIRS) | Free → $49/mo hosted |
| Kong AI Gateway | Enterprise governance, PII, semantic routing | Heavy; opaque pricing | Free self-host → Konnect |
| TrueFoundry | Full MLOps suite | Overkill + platform tax for routing alone | Custom |
| Merge Gateway | Transparent routing + governance/audit by default | Brand new, unproven; no self-host | Zero-fee 12mo promo → usage |
| OpenRouter | 300+ models, one key, unified billing | Marketplace fees; lighter governance/SLA | Pass-through + 5.5% fee |
Prices, model counts, and performance figures are vendor-reported and move fast. Re-confirm before you budget.
💡When to Pick Which
The first fork is one question: do you want to own the infrastructure, or not?
- Own it, Python-first → LiteLLM. Widest coverage, flexible custom plugin logic. You own the backend, and pager.
- Own it, latency-critical → Bifrost. Go, single binary, built for high RPS. Younger ecosystem.
- Own it, cost is the pain → Helicone. One URL change to start logging spend, with caching to cut the bill.
- Own it, want an OSS OpenRouter → LLM Gateway. One key, many providers, self-hosted, AGPL or enterprise.
- Don’t own it, need reliability + governance as a product → Portkey. Eyes open on the acquisition.
- Don’t own it, governance /audit focus → Merge Gateway. Built around budgets, DLP, and audit logs, but very new.
- Don’t own it, already run Kong → Kong AI Gateway. Extend the mesh you operate.
- Don’t own it, need the whole ML lifecycle → TrueFoundry. Right call only if you need more than routing.
- Don’t own it, want every model behind one key fast → OpenRouter. Breadth over governance + marketplace fees.
A common pattern:
Pair an open-source gateway for routing with a dedicated observability tool for deep tracing, rather than buying one box that claims to do everything.
Final Thoughts
The day your stack touches a second provider or cluster, the gateway stops being optional. Good news: whatever you need exists, from a single binary you fully control to a managed control plane with a contract behind it.
The big lesson today: the open-core and commercial line is blurred. Most “commercial” tools open-source their core, and most “open-source” ones sell an enterprise tier on top. So pick for your constraints, not the leaderboard, the fastest gateway on a slide is the wrong choice if your real bottleneck is cost attribution or a compliance audit.
Hoping this gave you enough breadth in LLM routing options.
What’s next:
Next, we’ll dive into a practitioner comparison of LLM guardrails solutions, open source vs commercial. Stay tuned!