What we check
Checks robots.txt for rules targeting GPTBot, ClaudeBot, PerplexityBot, and others.
Why it matters
GPTBot, ClaudeBot, PerplexityBot decide whether your content shows up in AI answers, make a deliberate call.
Common causes
- The robots.txt contains no rules for AI crawlers, so the default permission applies to GPTBot and friends, often unknowingly.
- A blanket block from the copyright debate also locks out the crawlers that feed AI search and citations.
- The rules are outdated, new bots with new user agents are not covered.