LEANROUTE

Blog

Spend caps and rate limits for LLM APIs, per team and per key

5 min readLeanroute Team

The LLM bill horror stories all have the same shape. An agent gets stuck in a retry loop overnight. A key ends up in a public repo. A batch job runs against the expensive model instead of the cheap one. Nobody notices until the invoice arrives, because provider dashboards report spend after the fact and most provider budget alerts only email you.

A spend cap is different from an alert: it stops the next request. This post shows how to set caps and rate limits in Leanroute, what your code sees when one is hit, and how to pick sensible numbers.

Four limits, at two levels

Leanroute has four limits:

Limit What it stops
Daily spend cap (USD) Runaway cost within a day. Resets at 00:00 UTC.
Monthly spend cap (USD) Total cost in a calendar month (UTC).
Requests per minute (RPM) Tight loops and request floods.
Tokens per minute (TPM) A few very large requests eating the budget.

You can set each one at two levels:

  • Organization, under Dashboard, Settings. Applies to every key in the org.
  • Per API key, under Dashboard, API keys. Applies to that key only.

When both are set, the stricter one wins. A common setup is a generous org-wide monthly cap as the backstop, plus tight per-key caps for anything risky: a key for an experimental agent, a key handed to a contractor, a key used in CI.

Leave a field empty for no limit. Changes take effect within about a minute.

Caps cover your own provider keys too

If you bring your own provider keys, requests on those keys still count toward your caps, priced at the provider's list price. That matters, because a cap that ignores BYOK traffic would not protect the bill you actually pay. If you set a $50 daily cap, an agent looping on your own OpenAI key stops at roughly $50 of OpenAI usage, the same as it would on ours.

What your code sees

When a spend cap is reached, the next request is refused before it reaches any provider, with HTTP 402:

{
  "error": {
    "code": "spend_cap_exceeded",
    "message": "Daily spend cap of $50.00 reached for this organization. Resets at 00:00 UTC.",
    "scope": "org_daily",
    "limit": 50,
    "current": 50.12
  }
}

Rate limits return HTTP 429 with code rate_limit_exceeded. Either way, the x-gateway-throttle-scope response header and the scope field tell you exactly which limit fired:

Scope Limit Typical handling
org_daily, key_daily Daily spend cap Stop until 00:00 UTC, or raise the cap
org_monthly, key_monthly Monthly spend cap Stop and page a human
org_rpm, key_rpm Requests per minute Back off and retry within the minute
org_tpm, key_tpm Tokens per minute Back off, or send smaller requests

The important part for client code: do not retry 402s in a loop. A spend cap will not clear in a few seconds, and an SDK that blindly retries on errors will just hammer the gateway. Retry 429s with backoff; treat 402s as a stop signal.

from openai import OpenAI, APIStatusError

client = OpenAI(base_url="https://api.leanroute.dev/v1", api_key=LEANROUTE_API_KEY)

try:
    reply = client.chat.completions.create(model=model, messages=messages)
except APIStatusError as e:
    scope = e.response.headers.get("x-gateway-throttle-scope", "")
    if e.status_code == 402 and "daily" in scope:
        schedule_for_tomorrow(messages)    # cap resets at 00:00 UTC
    elif e.status_code == 402:
        alert_on_call(scope)               # monthly cap or balance: needs a human
    elif e.status_code == 429:
        retry_with_backoff(messages)
    else:
        raise

Alerts before the wall

Hitting a cap with no warning is only slightly better than not having one. When a daily or monthly cap is set, Leanroute emails you once your spend passes 80% of it, at most once a day per cap. On the prepaid credits plan you also get an email when your balance runs out.

How precise is a cap?

Caps are checked before each request, and a request's cost is only known once it finishes. So a cap stops the next request after you cross it; it does not cut off a request already in progress. In practice the overshoot is at most the cost of the requests that were in flight at that moment.

Two cases worth knowing:

  • MCP tool loops run as one request. The whole loop's cost counts toward your caps, but a loop that has already started finishes rather than stopping halfway.
  • Streams are billed for the tokens actually generated, including when your client disconnects early. Disconnecting does not stop the provider from generating, so set max_tokens to bound long responses.

Picking numbers

Start from your actual traffic. The usage page in the dashboard shows daily cost per provider, which gives you a baseline.

  • Org monthly cap: around 1.5 to 2 times your expected monthly spend. It is a backstop, not a budget.
  • Org daily cap: around 3 times a normal day. A bad loop gets stopped within a day instead of within a month.
  • Per-key caps on anything experimental: small and specific. A prototype agent with a $5 daily cap can fail as loudly as it likes.
  • RPM per key: a little above that key's real peak. If a key normally peaks at 20 requests a minute, 60 gives headroom and still stops a tight loop quickly.

Revisit the numbers once a month. Caps that are always nearly hit get raised quietly, and caps that are never close stop being useful.

Plan limits

Separate from your own caps, the BYOK plans include a request rate limit: 30 requests per minute on Starter and 150 on Pro. Your own RPM setting can be stricter than the plan's, never looser. Details are on the pricing page.

Set your first caps under Dashboard, Settings, and see the API docs and headers reference for the full error format.

LLM GatewayFinOpsspend capsrate limiting
← All posts