Check explainer
robots.txt
A misconfigured robots.txt makes the site invisible to Google.
- to review
- ok
What we check
Checks robots.txt reachability and whether it blocks everything.
Why it matters
A misconfigured robots.txt makes the site invisible to Google.
Common causes
- A "Disallow: /" from the staging phase went live with the site.
- The file returns an error or redirect, some crawlers treat that as a block.
- The sitemap reference is missing or points to a dead URL.
How to fix it
robots.txt at the web root: "User-agent: *" + "Allow: /" + "Sitemap: <URL>".
What does a failing robots.txt finding mean?
A misconfigured robots.txt makes the site invisible to Google. robots.txt at the web root: "User-agent: *" + "Allow: /" + "Sitemap: <URL>".
How do you fix robots.txt?
robots.txt at the web root: "User-agent: *" + "Allow: /" + "Sitemap: <URL>".