What is a Robots.txt Checker?
The staging robots.txt with “Disallow: /” shipping to production is a classic, expensive accident — traffic quietly evaporates.
This checker fetches the live file, breaks down rules per user-agent, lists sitemap lines, and flags site-wide blocks loudly.
How to use the Robots.txt Checker
- Enter your domain.
- Read the per-agent rule table and warnings.
- Fix and re-check after every major deploy.
Frequently asked questions
Is a missing robots.txt a problem?
Not inherently — crawlers then assume everything is allowed. But a 5xx error on the URL can make Google pause crawling entirely.
What is the worst pattern?
“User-agent: * / Disallow: /” in production — it tells every crawler to leave. This checker flags it in red.
Should the sitemap line be here?
Yes — it is the standard discovery hint alongside Search Console submission.
Do all crawlers obey robots.txt?
Reputable ones do. It is guidance, not security — never rely on it to hide anything.