Robots.txt Generator
Create a robots.txt that allows the right crawlers and blocks the rest.
Rules that crawlers actually follow
A robots.txt is a short text file with a narrow job: tell crawlers which parts of the site they may request. It is not a security control and it does not hide pages from search results.
This generator writes a valid file and handles the fiddly parts, including the AI crawler presets that are easy to get wrong.
How to use it
- Choose a preset: allow everything, block AI crawlers, or custom.
- Add your sitemap URL.
- Add paths to disallow, one per line.
- Copy the result to
/robots.txton your domain.
Presets explained
Allow everything. User-agent: * with Allow: /. The right choice for most sites that want visibility.
Block AI training. Blocks GPTBot, ClaudeBot, Google-Extended, CCBot and similar agents. Use this if you do not want your content in model training data.
Custom. Full control over each group.
The common mistake
Disallow: / blocks every crawler, including Google, and removes the site from search. This single line has taken more sites offline than any other configuration error.
Check it after publishing
Fetch https://yourdomain.com/robots.txt in a browser to confirm it is served as plain text. Then use the robots.txt report in Search Console to see whether any important page is blocked.
For the page level version, use a noindex meta tag. Meta Tag Generator builds those tags, and Slug Generator keeps URLs consistent.
Frequently asked
Does robots.txt remove a page from Google?
No. It stops crawling, not indexing. A blocked page can still appear in results if other sites link to it. Use a noindex tag to remove a page.
Should I block AI crawlers?
That depends on whether you want your content used for model training. Blocking GPTBot and similar crawlers keeps content out of training data but also out of AI answers.
Where does robots.txt go?
At the root of the domain, for example https://example.com/robots.txt. It only controls that host.
Do I need a sitemap line?
It helps discovery. Add the full URL of your sitemap so crawlers find it without guessing.