What we check
Checks robots.txt reachability and whether it blocks everything.
Why it matters
A misconfigured robots.txt makes the site invisible to Google.
Common causes
- A "Disallow: /" from the staging phase went live with the site.
- The file returns an error or redirect, some crawlers treat that as a block.
- The sitemap reference is missing or points to a dead URL.