LEANROUTE

Caching

Prompt and semantic caching

Repeat and near-repeat requests are answered from cache without calling the provider. Cache hits bill at $0. There is nothing to set up: both tiers are on by default, on every plan, and each organization's cache is isolated from every other's.

Two tiers

Every non-streaming request checks two caches in order, and stops at the first hit.

  1. Exact match. The request (model, messages, parameters, tools) is hashed. If an identical request was answered recently, that answer is returned. Free, fast, and it cannot produce a false positive.
  2. Semantic. If there is no exact match, the prompt is embedded and compared against recent prompts from your organization. A match at cosine similarity 0.92 or higher returns the earlier answer. This catches the same question asked with different wording. The threshold is deliberately strict: a wrong cached answer costs more than a missed saving.

On a miss, the request goes to the provider as normal, and the response is stored in both tiers for next time.

What you can see

Every cache hit is marked on the response, so you never have to guess:

  • x-gateway-cache: exact or x-gateway-cache: semantic. Absent on a miss.
  • x-gateway-cache-similarity, on semantic hits only, with the matched score.
  • The request log in your dashboard records the hit and bills it at $0, and the overview shows your cache hit rate.

Full header reference at /docs/headers.

Limits

  • Lifetime: entries expire after 15 minutes. Caching is for bursts of repeated traffic (retries, fan-out, popular questions), not long-term storage.
  • Size: the semantic tier keeps up to 1,000 recent entries per organization, oldest evicted first.
  • Isolation: every entry is keyed to your organization. Another customer's traffic can never satisfy your request, or yours theirs.

What is never cached

  • Streaming requests (stream: true). They always go to the provider.
  • Requests with mcp_servers. The answer depends on what your tools return when the request runs, and tools usually return live data. A cached answer to "what is on my calendar today?" would be wrong tomorrow, so MCP requests skip both tiers for reads and writes. See /docs/mcp.
  • Errors. Only successful responses are stored, so a provider outage can never be replayed from cache.

Turning it off

Enable no-persistence mode in your dashboard settings. Both tiers are skipped entirely, and nothing about your prompts is stored beyond the lifetime of the HTTP request. Every response carries x-gateway-prompts-persisted: true or false so you can confirm which mode a request ran in. Billing, logging and spend caps work the same either way; only the cache is affected.

What is stored

The exact tier stores a hash of the request and the response body. The semantic tier also stores the prompt's embedding, which is a list of numbers rather than your text. The request log in our database never stores prompt or response content at all, only metadata such as model, tokens, cost and latency. More at /compliance.