Five questions to ask before you pick an AI gateway

Every AI gateway demo looks good. They all route a request to a model and show you a latency number.

The interesting part is the five minutes after that, when a prospect starts asking how things actually work. I watched one of those recently. Five questions, and every answer was some version of "you build that part."

Here are the five, what a good answer sounds like, and the one I think almost nobody is asking yet.

1. How does routing actually work?

Ask it like this: if I want the cheapest model that can handle this request, what do I configure?

There are two shapes of answer.

The first is a workflow builder: drag an LLM connector onto a canvas, wire if/else and switch branches, express your routing policy as a flowchart. This is flexible in the way that a blank file is flexible. It is also yours to maintain forever, and it will drift the moment a provider ships a new model or retires an old one.

The second is a policy you declare: name a quality tier, and the gateway picks inside it. That only works if the vendor is willing to take a position on which models are equivalent, publish it, and keep it current. That is real ongoing work, which is why many products push it back to you.

Neither is wrong. But be clear which you are buying, because "flexible routing" in a demo often means "you will be building routing."

Follow-up worth asking: what happens when the model I picked returns a 5xx? If the answer is "the request fails and you retry," failover is also something you are building.

2. How do token prices stay current?

Ask it like this: show me where the per-token rates live.

If the answer is a form with text boxes, stop and think about what that means.

It does not mean your cost reporting is approximate. It means it is wrong in ways you cannot detect. Providers reprice without announcements. Promotional rates lapse silently. A model moves to a new tier. Nothing in the product tells you, so you keep quoting confident numbers off a table that quietly stopped being true.

This is the question I think is most underweighted in gateway evaluations right now. Teams are building FinOps dashboards, chargeback models and per-team budgets on top of numbers that nobody has verified since setup.

A concrete example, from our own catalog, this week.

DeepSeek charges double during two windows on weekdays. Our routing layer knew that. Our billing calculation did not, and had been charging the off-peak rate for every request inside those windows. Separately, a Groq cache discount was sitting at a placeholder of zero, which meant we billed cache hits at the full input rate.

We found both through automation rather than by reading: a daily refresh against a live catalog, plus a canary that dispatches a real request to every model we list. A hand-maintained table would still have both errors today, and we would still be quoting cost numbers off it.

Follow-up worth asking: when a model is retired upstream, what happens to a request naming it? "A 502" is a worse answer than "it is rewritten to a named replacement and you get a deprecation header."

3. What do spend caps actually cap?

Ask it like this: if a bug in my code sends a million requests tonight, what stops it, and when?

The weak version is a quota per consuming application, checked after the fact, reported the next day. That is an invoice explanation, not a control.

The version worth having enforces before dispatch, at more than one level. An org-wide ceiling stops the whole account. A per-key limit contains the blast radius to one environment, so a runaway staging job cannot spend your production budget.

Follow-up worth asking: is the cap checked before the upstream call or after the response? Only the first prevents spend.

4. Are guardrails included or are they a plugin?

Ask it like this: what does it cost to turn on PII redaction?

A surprising number of gateways treat guardrails as a separate purchase. Sometimes a plugin, sometimes a tier, sometimes a per-request fee on top of the per-request fee.

There is a defensible argument for that, which is that serious content filtering is a real product in its own right. But if the reason you are buying a gateway is that you do not want to build this yourself, discovering it is a separate line item halfway through procurement is a bad surprise.

Follow-up worth asking: can a rule redact rather than only block? Blocking a request because it contains an email address is often useless. Stripping the address and letting it through is usually what you actually wanted.

5. Is semantic caching built, or is it a pattern?

Ask it like this: what do I need to provision to turn on semantic caching?

If the answer involves bringing your own vector database, you are not buying semantic caching. You are buying a diagram of semantic caching, and the assembly is yours: embedding model, vector store, similarity threshold, eviction, and tenant isolation so one customer's cached answer never reaches another's request.

That last one is the part that gets skipped in demos and matters most in production.

Follow-up worth asking: how is the cache isolated between tenants, and what similarity threshold is used? If neither has a crisp answer, it has not been run in anger.

The pattern underneath all five

Every one of these questions is really the same question: which parts am I still building?

A gateway that hands you primitives is not a bad product. For a team with platform engineers and strong opinions, it may be the right one. But the reason most teams reach for a gateway is to stop maintaining this category of plumbing, and a product that returns the plumbing with a nicer UI has not solved that.

So when you evaluate, sort the answers into two columns: what ships, and what you assemble. Then price the second column in engineer-weeks, because that is the real comparison.

Where we land

We are not neutral, so treat this as disclosure rather than argument.

Leanroute ships routing by tier with automatic failover, prices refreshed daily from a live catalog with a weekly reviewed sweep on top, hard spend caps per org and per key enforced before dispatch, a seven-rule guardrail library included rather than sold separately, and semantic caching pre-built with per-org isolation. All of it in the flat subscription, no per-request fee.

We are also honest about the gaps. The daily refresh covers what the upstream catalog carries; first-party models that it does not carry are maintained by us on a weekly cycle. Three BYOK-only providers cannot be probed by our canary at all, because we cannot buy credit on them, and that is written down rather than hidden.

If you are evaluating gateways, ask us these five. Ask the others too. The answers are more informative than any feature matrix.