Check explainer
AI bot policy
GPTBot, ClaudeBot, PerplexityBot decide whether your content shows up in AI answers, make a deliberate call.
- note
What we check
Checks robots.txt for rules targeting GPTBot, ClaudeBot, PerplexityBot, and others.
Why it matters
GPTBot, ClaudeBot, PerplexityBot decide whether your content shows up in AI answers, make a deliberate call.
Common causes
- The robots.txt contains no rules for AI crawlers, so the default permission applies to GPTBot and friends, often unknowingly.
- A blanket block from the copyright debate also locks out the crawlers that feed AI search and citations.
- The rules are outdated, new bots with new user agents are not covered.
How to fix it
Decide with marketing + legal, then add User-agent sections in robots.txt.
What does a failing AI bot policy finding mean?
GPTBot, ClaudeBot, PerplexityBot decide whether your content shows up in AI answers, make a deliberate call. Decide with marketing + legal, then add User-agent sections in robots.txt.
How do you fix AI bot policy?
Decide with marketing + legal, then add User-agent sections in robots.txt.