LiteLLM is an open-source LLM gateway with about 61,000 GitHub stars. It gives your code one OpenAI-style endpoint for more than 100 providers. Teams usually look for a LiteLLM alternative for one of five reasons:
- A feature they need is only in the paid Enterprise license.
- They want tracing and evaluations in the same tool as the gateway.
- The proxy adds too much time per request at their traffic level.
- They do not want to run a gateway server.
- The March 2026 PyPI incident made them review their stack.
The right replacement depends on the reason. This guide gives the tool for each reason, then compares five alternatives in one table.
The short answer
| Why you are leaving LiteLLM | Look at |
|---|---|
| Audit logs, roles or budget alerts need an Enterprise license | AcruxCore |
| Your prompts live in LangSmith or Langfuse, separate from LiteLLM | AcruxCore |
| Gateway overhead matters at thousands of requests per second | Bifrost, or LiteLLM's Rust beta |
| You do not want to run a gateway server | Cloudflare AI Gateway |
| You want one bill for many model providers | OpenRouter |
| Your platform team already runs Kubernetes and Envoy | Agent Router (formerly Envoy AI Gateway) |
If none of these reasons apply to you, staying on LiteLLM is a reasonable choice. The next section lists the LiteLLM features that a replacement must also cover.
What LiteLLM does well
These are the LiteLLM features teams use most, taken from the LiteLLM docs:
- Virtual keys. Each key can carry its own budget and its own limits on requests and tokens per minute. Spend is tracked per key, user and team.
- Caching. Responses can be cached in Redis or Valkey, in S3 or GCS, on disk, or in a semantic cache backed by Redis or Qdrant.
- Reliability. Three kinds of fallback are built in: to another model, to a model with a larger context window, and to a model with a different content policy. Retries, cooldowns and load balancing across deployments are also included.
- Logging callbacks. One line of config sends every call to Langfuse, LangSmith and many other tools.
- A prompt store. It is marked experimental in the dashboard, and only proxy admins can edit prompts.
The code is MIT, except the enterprise/ folder. Virtual keys need Postgres, and the docs say to add Redis once you run more than one proxy instance.
Reason 1: the features you need are in the Enterprise license
LiteLLM's enterprise page lists the features that need a paid license. These are the ones a growing team usually needs first:
- Audit logs
- Role-based access control
- SSO for more than five users
- Tag-based budgets and budgets per model on a virtual key
- Email alerts for soft budgets
A two-person team rarely needs these. A team needs them when a customer or an auditor asks who changed a key, and when.
AcruxCore is an Apache-2.0 gateway with no enterprise folder: the self-hosted version has every feature. It includes roles, an audit log of every change, and budgets that send an alert at 80% of the limit and again when the limit is reached. AcruxCore is the first tool in the comparison below.
Reason 2: you need tracing and evaluation, not only a gateway
A gateway logs each request. It does not show the steps of an agent run as one tree, store test datasets, or score answers. So many LiteLLM setups send traces to a second tool, usually Langfuse or LangSmith, through LiteLLM's callbacks. This Langfuse vs LangSmith comparison, with one app tested on both covers license, self-hosting, price and data retention side by side. If you want to replace LangSmith, Our Code World also has a list of open-source LangSmith alternatives for startups.
In the same setup, the prompts often live in a third place. A common stack keeps prompts in LangSmith or Langfuse and calls the model through LiteLLM. The app pulls the prompt from the prompt store, then sends the model call to LiteLLM. One user request then depends on two services, often on two different networks.
Both prompt SDKs keep pulled prompts in memory, so the extra round trip does not happen on every request. The LangSmith SDK keeps a prompt for 5 minutes by default and the Langfuse SDK for 60 seconds, and both refresh it in the background. The round trip still happens when the cache is empty: on the first call in each process, and on every cold start of a serverless function. If the prompt store is down at that moment, the request fails.
AcruxCore keeps the prompt and the gateway in one service. The app sends the prompt's name, its label and the variables in the same request as the model call:
curl -X POST $ACRUXCORE_BASE_URL/gateway/chat/completions \
-H "Authorization: Bearer $ACRUXCORE_API_KEY" \
-H "Content-Type: application/json" \
-d '{"model":"gpt-4o-mini","prompt":{"name":"weather-brief","alias":"production","variables":{"city":"Lisbon"}}}'
The gateway renders the prompt and calls the model, so the app never fetches the prompt separately. The call is recorded as a trace that names the prompt version it used, with no extra tracing code. The prompts, traces, datasets and evaluations are stored in the same database.
Reason 3: gateway overhead at high request rates
LiteLLM's proxy is written in Python. At a few requests per second, the time the gateway adds is small next to the model's own response time. At thousands of requests per second, the proxy's added time and its memory use become a real cost.
LiteLLM is moving its proxy to Rust. Its June 2026 launch post describes an early beta, with routes moving one at a time and the full Rust server planned for 1 December 2026. LiteLLM reports about 0.05 ms of overhead per request on the Rust path and about 7.5 ms on the Python path, measured against a mock upstream. If overhead is your only reason to leave, test the Rust beta first.
Bifrost, from Maxim AI, is a gateway written in Go. Its README reports 11 microseconds of added time per request at a steady 5,000 requests per second. Maxim's website says Bifrost is "50x faster than LiteLLM" at the 99th percentile. These are the vendor's own numbers, and the website gives only one line on how Maxim measured them. Run your own test on your own traffic before you rely on them.
Reason 4: you do not want to run a server
Two hosted services remove the gateway server from your stack.
Cloudflare AI Gateway is a hosted proxy between your code and the providers you already pay. Analytics, caching and rate limiting are free on every Cloudflare plan. It also has spend limits, retries, fallbacks and request logs.
OpenRouter sells access to hundreds of models from many providers under one account and one bill. It does not add a markup to model prices. It charges a 5.5% fee when you buy credits.
Reason 5: the March 2026 PyPI incident
On 24 March 2026, two LiteLLM releases on PyPI, versions 1.82.7 and 1.82.8, contained a credential stealer. LiteLLM's security post says PyPI quarantined them after about 40 minutes. Users of the official Docker image and anyone who stayed on 1.82.6 or earlier were not affected. LiteLLM then moved to a new release pipeline and signs its images, starting with 1.83.0.
Switching tools does not remove supply-chain risk, because every gateway has dependencies. For any tool on this list, run the gateway as a pinned container image, separate from your application's own Python packages.
The five alternatives compared
Every fact in this table comes from the vendor's own docs, license file or GitHub repository, read on 18 October 2026. "Not checked" means we did not find the feature in the pages we read, not that it is missing.
| Tool | Self-host | License | Budgets | Response cache | Prompt store | Traces and evals |
|---|---|---|---|---|---|---|
| LiteLLM (for reference) | Yes: Postgres, plus Redis for more than one instance | MIT, except enterprise/ | Yes | Yes | Yes, experimental | Sent to other tools |
| AcruxCore | Yes: Postgres and Redis | Apache-2.0, no enterprise folder | Yes, with alerts | Yes | Yes, versioned | Built in |
| Bifrost | Yes: with SQLite, Postgres or only a config file | Apache-2.0, some features Enterprise | Yes | Yes, semantic | Yes | Logs and metrics |
| Cloudflare AI Gateway | No | Closed source | Spend limits | Yes | No | Logs and analytics |
| OpenRouter | No | Closed source | Not checked | Not checked | Not checked | Activity log |
| Agent Router | Yes: Kubernetes and Envoy Gateway | Apache-2.0 | Token rate limits and quotas | No | No | Metrics and tracing export |
1. AcruxCore
AcruxCore puts a gateway, versioned prompts, tracing, a tool catalog and evaluations in one open-source app. A self-hosted install needs only Postgres and Redis.
What replaces LiteLLM. The gateway accepts OpenAI-style requests, so you change the base URL and the API key. It holds your provider keys encrypted, caches responses, retries a failed call, and falls back to a second model. Each key can have a spend budget and limits on requests and tokens per minute.
What it adds. Every call becomes a trace with its model, tokens, cost and latency. Your app reads the prompt version that has the production label, so a prompt change needs no redeploy. Prompt templates support conditionals and loops. Feedback on traces becomes a dataset, and evaluations score a prompt against it. Roles and the audit log are part of the free self-hosted version.
What you give up. AcruxCore has native adapters for OpenAI, Anthropic and Google Gemini, and any other provider works through an OpenAI-compatible base URL. LiteLLM has native adapters for far more providers. AcruxCore's repository went public in August 2026, so its community is still small.
Pick it if you were about to buy a LiteLLM Enterprise license for roles, audit logs or budget alerts, or you run a separate tracing tool next to LiteLLM today.
2. Bifrost
Bifrost is an Apache-2.0 gateway in Go, with about 8,900 GitHub stars. Its latest release came out on 13 October 2026. It can run with no database at all, from a config file, or keep its config in SQLite or Postgres.
What replaces LiteLLM. Virtual keys have budgets and limits. Keys belong to teams, and teams belong to customers. Bifrost also has automatic fallbacks, load balancing, a semantic cache, Prometheus metrics and request logs. It also has a versioned prompt repository. Maxim publishes a migration guide from LiteLLM to Bifrost.
What you give up. Role-based access control, audit logs, guardrails, clustering and adaptive load balancing are Enterprise features. LiteLLM also keeps role-based access control and audit logs in its Enterprise license.
Pick it if you have measured LiteLLM's added time on your own traffic, it is too high, and you need a Go gateway in production now rather than after LiteLLM's Rust server ships.
3. Cloudflare AI Gateway
Cloudflare AI Gateway is a hosted proxy on Cloudflare's network. You create a gateway in the Cloudflare dashboard and point your provider's base URL at it.
What replaces LiteLLM. Cloudflare AI Gateway has caching, rate limiting, spend limits, retries and fallbacks, request logs and cost analytics. You can use your own provider keys. You can also pay through Cloudflare's unified billing, which adds 5% to credit purchases.
What you give up. It is closed source and cannot be self-hosted. It has no prompt store and no evaluations. Log storage depends on your Workers plan.
Pick it if you only used LiteLLM as a proxy and a cost log, and you already use Cloudflare.
4. OpenRouter
OpenRouter is a hosted service that gives you hundreds of models from many providers behind one OpenAI-compatible API, one account and one bill.
What replaces LiteLLM. OpenRouter gives you the single endpoint and routes each call to a provider. You do not need an account with each model provider.
What you give up. It is a paid service, not a gateway you control, and it has no evaluations. If you use your own provider keys with OpenRouter, the first $25,000 of usage each month has no fee. Above that, OpenRouter charges 5% of the call's normal OpenRouter price.
Pick it if what you wanted from LiteLLM was access to many models without many provider accounts.
5. Agent Router (formerly Envoy AI Gateway)
Envoy AI Gateway was renamed Agent Router in September 2026 and moved to the Agentic AI Foundation. The code, license and maintainers did not change. It is an Apache-2.0 gateway built on Envoy and Envoy Gateway.
What replaces LiteLLM. Agent Router has token-based rate limits, usage quotas, provider fallback with retries, and OpenTelemetry metrics and tracing. It also routes MCP and agent-to-agent traffic.
What you give up. Production use needs Kubernetes 1.32 or newer and Envoy Gateway. A standalone mode exists, but its docs call it experimental. We found no response cache and no prompt store in its docs.
Pick it if your platform team already runs Envoy Gateway on Kubernetes and wants model traffic handled the same way.
Tools that older lists still recommend
Several "LiteLLM alternatives" lists from 2025 name tools whose situation has since changed:
- Portkey was acquired by Palo Alto Networks, a deal that closed on 29 May 2026. Its MIT gateway repository has had no new commits since 25 May 2026.
- TensorZero is archived on GitHub. Its website says the project is no longer maintained.
- Helicone joined Mintlify in March 2026 and is in maintenance mode. It still gets security fixes and new models, but no new features. To see Helicone tested next to other platforms, read these ten LLM observability tools compared side by side.
How to move off LiteLLM
- List what your config uses. Write down your model groups, fallbacks, cache settings, callbacks, and the budget and limits on each virtual key.
- Recreate the keys in the new tool before you switch traffic. Virtual keys do not carry over, so each team or service gets a new key with the same budget and limits.
- Replace the callbacks. If LiteLLM sent traces to Langfuse or LangSmith, either point the new gateway at the same tool or use one that records traces itself.
- Change the base URL and the key in one environment first. Compare a day of cost, latency and error rate with LiteLLM's numbers before you move production.
