# Leanroute robots.txt
# Spec: RFC 9309 / https://www.robotstxt.org/
#
# Strategy: allow every polite search and AI crawler on the marketing
# site, disallow the app surfaces (dashboard, admin, login, invite links,
# API). Cloudflare's "managed robots.txt" is intentionally disabled at the
# zone level: it injected a non-standard `Content-Signal:` directive that
# Bingbot flagged as unrecognized syntax.
#
# 2026-10-06: collapsed into ONE group. Previously each named crawler had
# its own `Allow: /` group. Under RFC 9309 a crawler obeys only the most
# specific group that names it and ignores `*`, so Googlebot, ClaudeBot
# and the rest were never told about the Disallow lines below. Listing
# every user-agent in a single group gives them all the same rules.
# Also added Anthropic's current agents (Claude-SearchBot, Claude-User),
# which power Claude's web search and on-demand fetching.

# Search engines
User-agent: Googlebot
User-agent: Bingbot
User-agent: DuckDuckBot
User-agent: YandexBot
User-agent: Baiduspider
User-agent: Applebot
# OpenAI
User-agent: GPTBot
User-agent: OAI-SearchBot
User-agent: ChatGPT-User
# Anthropic
User-agent: ClaudeBot
User-agent: Claude-SearchBot
User-agent: Claude-User
User-agent: Claude-Web
User-agent: anthropic-ai
# Perplexity
User-agent: PerplexityBot
User-agent: Perplexity-User
# Others
User-agent: Google-Extended
User-agent: Applebot-Extended
User-agent: Bytespider
User-agent: CCBot
# Everyone else
User-agent: *
Allow: /
Disallow: /dashboard/
Disallow: /admin/
Disallow: /api/
Disallow: /login
Disallow: /invite/

Sitemap: https://leanroute.dev/sitemap.xml