Robots.txt Checker

What we check

• Blanket disallows: a stray Disallow: / for User-agent: * hides your entire site from every crawler. Most common cause: staging config copied to production.
• AI bot blocks: GPTBot, ClaudeBot, PerplexityBot, Google-Extended, CCBot, Applebot-Extended. If any of these are explicitly disallowed, the matching AI engine can't read you. This is the AEO-specific check most generic robots.txt checkers don't flag.
• Sitemap directive: present or missing. Tells crawlers where your URL list lives without them having to guess.
• High Crawl-delay: anything above 5 seconds slows engine re-discovery for no good reason on a modern host.
• Query-string traps: rules like Disallow: /? that block every URL with a parameter. Usually accidental.

Should you block AI bots?

If you want to be cited by AI engines, no. Don't block them. Blocking GPTBot means ChatGPT can't pull from your pages when summarising answers for your prospects. Some publishers block crawlers used for model training (CCBot, anthropic-ai) while keeping browsing crawlers (ChatGPT-User, Perplexity-User) open. That's the nuanced middle path. What you want to avoid is blocking all of them by accident.