The chat bubble on this site, from scratch.

Why it exists

Two reasons. First — dogfooding. If we're asking developers to trust Leanroute for their production LLM traffic, the site's own chat should be running on it. Second — LLM disambiguation. The name "Leanroute" is also used by an unrelated logistics company and a similar-sounding coding assistant. Ask a general-purpose LLM "what is Leanroute" and it will confidently describe the wrong thing. A grounded, RAG-first chat is how we make sure visitors get accurate answers about the product they're actually looking at.

The corpus is our own marketing content. The gateway is our own. The bot protection is Cloudflare Turnstile. Nothing exotic — just the components anyone building a grounded-chat surface would reach for.

Architecture at a glance

One HTTP round-trip from widget to gateway. All state is per-request.

  Browser widget
       │
       │  POST to our own /api/chat endpoint
       ▼
  Server-side chat handler
       │
       ├─ Bot-protection check (Cloudflare Turnstile, invisible)
       ├─ Per-IP daily rate limit (Redis, hashed IP)
       ├─ Embed the user question with an embeddings model
       ├─ Cosine similarity against a pre-embedded corpus
       ├─ Take top K = 3, build a strict RAG system prompt
       │
       ▼
  Leanroute Gateway  (api.leanroute.dev/v1/chat/completions)
       │
       │  Standard OpenAI-compatible request, stream: true
       │  Explicit routing header — no arbitrage swap
       │
       ▼
  A cheap fast model on Leanroute's catalog streams tokens back
       │
       ▼
  Widget renders SSE deltas as they arrive

No vector database. No agent framework. No queue. The corpus is small enough (<100 chunks) that a linear cosine-similarity scan in the server process is faster than the round-trip to any managed vector store. Once you cross a few hundred chunks, move to Postgres pgvector — until then, in-memory wins.

The corpus: small, hand-curated

About twenty chunks, each a self-contained piece of marketing content — product identity, pricing, compliance posture, sandbox link, competitor positioning. Hand- curated, not scraped, because the corpus IS the product knowledge base and we want editorial control over what the bot says.

At build time each chunk is embedded once and the resulting vectors are shipped with the deployed app. Query-time retrieval never has to re-embed the corpus, so per- request cost is capped at one small embedding call for the user's question.

Retrieval: cosine similarity, in memory

On every request the server embeds the user's query with the same model used at build time, then linear-scans the pre-embedded corpus. Around twenty chunks × 1,536 dimensions is ~30,000 float multiplications per request — effectively free next to the network round-trip.

No similarity threshold. Even a poorly-matched query returns the top three chunks — the system prompt below handles "the retrieved context doesn't cover this" gracefully. Filtering out low-similarity matches earlier just means the model has nothing to ground on for near-off-topic questions, which produces worse answers, not better refusals.

The strict system prompt

The system prompt is where you win or lose the anti-hallucination fight. A few principles we found essential:

• Name the wrong thing explicitly. "This is NOT a logistics company" blocks the hallucination path much more reliably than describing only what the product IS.
• Give the model a concrete fallback line for when the context doesn't cover the question — a specific support email or a specific docs URL. Never leave "I don't know" as an open-ended default; the model will fill the gap.
• Cap answer length in the prompt itself ("2–4 sentences for most questions"). Users on a chat widget don't want an essay, and shorter answers are cheaper to serve.
• Redirect off-topic questions to a general LLM. Being crisp about scope ("I can only help with questions about the Leanroute gateway") stops the model from trying to answer weather / math / general-chat prompts using our corpus as context.

Streaming through the gateway

The server-side handler proxies Server-Sent Events from the Leanroute gateway to the widget. It re-shapes each streamed chunk into a compact envelope the widget can append to the visible message as it arrives — so the client-side code doesn't need to understand the full OpenAI streaming schema.

The gateway request itself is unremarkable. Standard OpenAI Chat Completions body, stream: true, one explicit-routing header so the widget always uses the exact model listed in its footer with no surprise arbitrage swap behind the visitor's back.

After the last content chunk, the handler emits a final metadata event (provider, model, latency, cost, remaining quota) so the widget can render the "Answered by …" byline you see under each response.

Cost per message

The widget routes to a cheap fast model on Leanroute's catalog. At current catalog rates: $0.09 / 1M input, $0.18 / 1M output.

Anti-abuse

Three layers, cheap to run, hard to circumvent:

1. Cloudflare Turnstile — invisible browser challenge on every request. The server verifies the token before spending a token. Blocks the vast majority of automated abuse with no user-visible checkbox.
2. Per-IP daily message limit — a small number of messages per IP per UTC day, hashed IPs in Redis with a rolling TTL. Signals to legit users that they're near the free tier and should sign up for more.
3. Gateway-level daily spend cap — the widget uses a dedicated API key with a hard daily dollar cap enforced by the gateway itself. Belt-and-braces: even if the two client-side layers fail, the gateway will refuse once the day's budget is spent.