How to use the Robots.txt generator
- 1Choose All crawlers: Allow, with exceptions below for a normal site.
- 2Tick WordPress defaults if you use WordPress, and add Allow and disallow rules for paths to keep out of crawling.
- 3Under AI crawlers, tick the bots to block; each one says what it does.
- 4Add your Sitemap URLs, then Download the file and upload it to your domain's root. Use Test these rules in the robots.txt tester to check URLs against it.
Example: WordPress site blocking three AI crawlers
All crawlers: Allow · WordPress defaults: on Disallow: /cart/ · Disallow: search AI crawlers: GPTBot, CCBot, Google-Extended Sitemap: https://www.example.com/sitemap.xml
User-agent: * Disallow: /wp-admin/ Allow: /wp-admin/admin-ajax.php Disallow: /cart/ Disallow: /search # AI crawlers User-agent: GPTBot User-agent: CCBot User-agent: Google-Extended Disallow: / Sitemap: https://www.example.com/sitemap.xml
The path search was typed without a slash; the generator adds it. Several user-agent lines above one rule form a single group, as RFC 9309 allows.
User-agent groups and rules
A robots.txt file is a list of groups. Each group starts with one or more User-agent: lines naming crawlers, followed by Allow: and Disallow: rules. A crawler follows the group that names it most specifically and ignores the rest; User-agent: * covers every crawler without its own group. Within a group, the longest matching rule wins, so Allow: /wp-admin/admin-ajax.php beats Disallow: /wp-admin/. Paths are case-sensitive; * matches any characters and $ marks the end of the URL.
Blocking or allowing AI crawlers
AI companies use separate user agents for training, search and user-triggered fetches, so you can block one without the others:
| Token | Purpose |
|---|---|
| GPTBot, ClaudeBot, CCBot, Meta-ExternalAgent | Collect training data |
| OAI-SearchBot, Claude-SearchBot, PerplexityBot | Index pages for AI search answers |
| ChatGPT-User, Claude-User | Fetch a page a user asked about |
| Google-Extended, Applebot-Extended | Control tokens: not crawlers; they govern AI use of content Googlebot/Applebot already crawl |
Blocking Google-Extended doesn't affect Google Search rankings or AI Overviews in Search. robots.txt is a request; well-behaved crawlers follow it, but it isn't access control.
Adding your sitemap
Sitemap: https://www.example.com/sitemap.xml can go anywhere in the file and applies to all crawlers. Use the full URL, and list several sitemaps or a sitemap index if you have more than one. It's the easiest way for crawlers other than Google and Bing (where you can also submit sitemaps directly) to find your sitemap. Need one? Use the XML sitemap generator.
Crawl-delay: who respects it
Crawl-delay: 10 asks a crawler to wait 10 seconds between requests. Bing and Yandex honor it; Google ignores it and adjusts its crawl rate automatically based on how your server responds (slow responses and 5xx/429 errors slow it down). If Googlebot really overloads your server, return 503 or 429 temporarily rather than relying on crawl-delay.
Mistakes that block a whole site
Disallow: /underUser-agent: *, often left over from a staging site.- Blocking CSS and JavaScript folders, which stops Google rendering pages properly.
- Using robots.txt to hide a page from search: a blocked URL can still be indexed (without content) if other sites link to it. Use
noindexinstead, and don't block the page, or crawlers will never see the noindex. - Typos in field names (
Disalow), which crawlers silently ignore. - Serving robots.txt with a 5xx error, which makes Google pause crawling the whole site.
Is anything I enter stored?
This tool runs entirely in your browser. Your text is not uploaded to our servers.