What is a Robots.txt Generator?
robots.txt syntax is small but unforgiving — one wrong Disallow can deindex a site (it has happened to large companies).
Two clear modes: production (allow everything except listed paths) and staging (block all), with the sitemap line search engines want.
How to use the Robots.txt Generator
- Pick the policy mode.
- List paths to block (production mode) and your sitemap URL.
- Generate, copy, save as robots.txt at your site root.
Frequently asked questions
Does robots.txt actually block access?
No — it is a politeness convention. Crawlers you care about obey it; anything sensitive needs real access control.
Will Disallow remove pages from Google?
It stops crawling, not necessarily indexing (blocked pages can appear URL-only). Use noindex or auth for true removal.
Where must the file live?
Exactly at the domain root: example.com/robots.txt. Subdirectory copies are ignored.
Common mistake to avoid?
Shipping the staging “Disallow: /” file to production — check yours with the Robots.txt Checker after deploys.