Self-Hosting an LLM Gateway: When It's Right, When It's Not

TL;DR

An LLM gateway sits on your critical path with your provider keys, your prompts, and often admin access to spend caps and routing rules. When you self-host it, you own the security posture, the patch cadence, and the full ops stack: Postgres, Redis, monitoring, on-call. The CVE-2026-35029 privilege escalation in LiteLLM is a reminder that middleware is not something you can install once and forget. This post is about what self-hosting actually costs on a working month, when it is still the right call, and when a managed gateway makes more sense. It is not an argument that managed is better than self-hosted. It is an argument that the choice deserves an honest cost model instead of a default.

What Just Happened

A few days ago a privilege-escalation issue landed in LiteLLM (CVE-2026-35029, along with a few related chains). In short: a read-only viewer account could reach a config path that let it modify configuration, block keys or endpoints, and in some chains escalate further inside the gateway. The maintainers patched it. LiteLLM users who apply the update quickly are fine.

That is not a LiteLLM-specific problem. Every popular OSS middleware has shipped security fixes in the last two years. Envoy, Kong, Traefik, Consul, even Nginx itself. Complex middleware with an admin surface and a lot of moving pieces is a place where privilege boundaries slip. It always has been.

The reason this one caught attention is that LLM gateways are a newer category. Many teams installed one during 2024-2025 because they needed cost tracking and provider failover, and then never revisited the operational side. The CVE is a good moment to revisit it.

Why LLM Middleware Is a Higher-Value Target Than It Looks

Three reasons LLM gateways are worth attacking:

1. The gateway holds every provider key in your stack.

OpenAI, Anthropic, Google, xAI, DeepSeek, whatever else. If those keys have no per-model spend cap on the upstream side (most do not have granular caps), stealing them is worth real money to the attacker. A compromised OpenAI key can be used to burn thousands of dollars in a few hours before rate limits kick in.

2. The gateway sees every prompt.

Prompts routinely contain data the sending team did not intend to expose: customer PII pasted into a system prompt, secrets embedded in a code snippet, internal document content in a RAG retrieval, drafted emails that reveal deal terms. The gateway is a natural exfiltration point because it has all of it in one place.

3. Gateways often ship with weaker default auth than the app calling them.

The gateway was often built as an internal tool: it started life on a private network with basic-auth admin access, and the auth surface never got hardened when the tool became critical infrastructure. Many self-hosted deployments still have their admin UI reachable from the public internet on the same origin as the API, protected only by a shared bearer token stored in an env var.

Web API gateways went through the same maturation curve between 2017 and 2020. Kong, Envoy, and Traefik all had multiple privilege-escalation CVEs in that window. What is happening to LLM middleware now is that same curve, compressed into a shorter window because the category matured faster.

The Real Cost of Self-Hosting: An Honest Checklist

Here is what you actually take on when you decide to self-host an LLM gateway. Not the marketing version, the working-month version.

Patch cadence. Someone on your team needs to be watching the upstream repo for CVE announcements, evaluating whether they apply to your deployment, testing patches in staging, and deploying them fast. Estimate: 2-4 hours per month in a quiet month, 10+ hours in a month with a critical CVE.

Secret handling. BYOK keys for every provider, an admin key, database passwords, Redis credentials, JWT signing keys if you added auth. Rotation policy, storage (Vault? sealed secrets? env vars?), audit trail on who accessed what. Estimate: one-time setup of 8-16 hours, ongoing 1-2 hours per month for rotation.

Database operations. Postgres for request logs, usage aggregations, per-org config. You are now responsible for backups, restore drills, connection pooling, index maintenance, and cost tracking as the log table grows. Estimate: 2-4 hours per month once stable.

Redis operations. Cache, rate limits, spend counters. Persistence config, memory limits, monitoring for eviction pressure. Estimate: 1-2 hours per month.

Monitoring. Uptime, latency, error rates, upstream provider health, cache hit rates, spend against per-org caps. Someone needs to configure alerting thresholds and respond to pages. Estimate: 4-8 hours one-time setup, 1-3 hours per month tuning.

On-call. LLM outages happen. OpenAI has multi-hour incidents roughly monthly. Anthropic has intermittent 5xx storms. When those happen, your gateway is retrying, failing over, hopefully caching, and generally under load in ways it is not under load the rest of the time. Someone needs to be reachable. Estimate: rotating burden across 2+ engineers, or the founder is on-call 24/7.

Config changes without downtime. Adding a new provider, changing routing rules, updating guardrail policies, adjusting spend caps. All these need to happen without a restart if you have real traffic. Estimate: implicit in the platform team's ongoing time, hard to break out.

Adds up to: For a mature self-hosted LLM gateway serving real production traffic, budget 15-25 engineer-hours per week across your team. Not per month. Per week.

At a rough fully-loaded engineer cost of $150k/year in the US or $75k in APAC, that is $22-56k per year of engineering time going into keeping the gateway working. That is the number to compare against managed pricing, not the "$0 for OSS" number.

When Self-Hosting IS the Right Call

None of the above means managed always wins. There are real cases where self-hosting is correct:

You have data residency requirements no managed vendor can meet. Some regulated verticals (regional banking, defense, government) require that inference metadata never leave a specific jurisdiction, and no managed gateway currently has infrastructure in that jurisdiction. Self-host in-region.

You already have a platform team that runs this class of software. If you have SREs who already operate Envoy or Kong for other services, adding an LLM gateway to their scope is marginal cost. The delta is smaller than for a team that has never run middleware before.

You need deep compliance controls that require end-to-end audit. Some SOC 2 or ISO 27001 audits require that you can prove which humans touched which config knobs and when. A managed vendor gives you their audit trail, not yours. If your auditor wants your audit trail, you host.

You are at a volume where the managed markup exceeds a dedicated engineer. If you are pushing tens of billions of tokens per month, a 5-10% markup on inference is real money. At some threshold (often around $500k/year in LLM spend), a dedicated platform engineer to run the gateway becomes cheaper than the vendor's take. Do the math.

When Managed IS the Right Call

The other side, also honestly:

Small team, under 10 engineers. You do not have a dedicated platform person, and the founder-engineers who would end up owning the gateway have higher-leverage work to do. Managed is the right call by a wide margin.

You do not already run OSS middleware in production. If nobody on your team currently gets paged for Envoy or Kong, adding an LLM gateway is not marginal. It is a new class of on-call that will crowd out other work.

You do not want to be paged when OpenAI goes down. Managed gateways handle upstream provider outages for you: they retry, fail over, degrade gracefully. When you self-host, you get to watch your own retries.

The vendor is on-call for their infrastructure and you are on-call for your app. That split is often the right one at your stage.

If You Are Currently on LiteLLM and Considering a Switch

For a Leanroute-specific answer: the migration path is short because Leanroute exposes an OpenAI-compatible endpoint. In most cases the change is:

Sign up, mint a runtime key, add your provider keys via BYOK.
Change your OPENAI_BASE_URL from your LiteLLM instance to https://api.leanroute.dev/v1.
Keep your existing model IDs (openai/gpt-5.6-luna, anthropic/claude-sonnet-5, etc.) — they work as-is.
Verify with a canary request. Check the x-gateway-provider response header to prove routing.
Cut over the rest of your traffic when the canary is clean.

For a typical deployment that takes an afternoon. For a deployment with heavy per-org config, custom guardrails, or unusual auth, budget a day or two. Full docs at leanroute.dev/docs. If you want a hand, [email protected] will reply personally.

The Meta Point

An LLM gateway is a trust boundary. When you own the gateway, you own the trust: your team's patching, your team's secret handling, your team's audit trail. When a vendor owns it, they own the trust, and their business exists on being trusted. Both models work. Both models fail when the owner is not paying attention.

The wrong answer is not making an active choice. The teams that get hurt by CVEs like the LiteLLM one are the teams that installed the gateway eighteen months ago, moved on, and never revisited whether they had the capacity to keep it patched. That is true of any middleware, not just LLM middleware.

If you are self-hosting, make sure someone on your team owns the operational surface, and give them the time to actually own it. If you are on managed, make sure the vendor's security posture and status page match your risk tolerance. Either way, revisit the choice once a year. The stakes are quietly getting higher on both sides.

If you are on the fence between the two, the honest question to ask is: do we have the team, and are we willing to give that team the hours? If either answer is no, managed is the right call.

Sources: