What is a Robots.txt Checker?
The staging robots.txt with “Disallow: /” shipping to production is a classic, expensive accident: traffic quietly evaporates.
This checker fetches the live file, breaks down rules per user-agent, lists sitemap lines, and flags site-wide blocks loudly.
Frequently asked questions
Is a missing robots.txt a problem?
Not inherently: crawlers then assume everything is allowed. But a 5xx error on the URL can make Google pause crawling entirely.
What is the worst pattern?
“User-agent: * / Disallow: /” in production: it tells every crawler to leave. This checker flags it in red.
Should the sitemap line be here?
Yes: it is the standard discovery hint alongside Search Console submission.
Do all crawlers obey robots.txt?
Reputable ones do. It is guidance, not security: never rely on it to hide anything.