How to use the Robots.txt tester
- 1Enter a domain under Site address and press Fetch robots.txt, or paste rules into robots.txt rules.
- 2Pick a User agent (or Other… for any crawler token).
- 3List the URLs or paths to test, one per line.
- 4Read Results: each URL says Allowed or Blocked and quotes the rule and line number that decided it. Check Syntax notes for ignored or misspelled lines.
Example: one file, five URLs (illustrative)
User-agent: * Disallow: /search Disallow: /*.pdf$ Allow: /search/help User-agent: Googlebot Disallow: /private/ Allow: /private/press/
As Bingbot: Blocked /search?q=bread ← Disallow: /search (line 2) Allowed /search/help ← Allow: /search/help (line 4), the longer match Blocked /files/menu.pdf ← Disallow: /*.pdf$ (line 3) Allowed /files/menu.pdf?v=2 ← no rule matches ($ needs the URL to end in .pdf) As Googlebot: Allowed /search?q=bread ← Googlebot has its own group, which has no /search rule Allowed /private/press/2026.html ← Allow: /private/press/ (line 9) Blocked /private/report.html ← Disallow: /private/ (line 8)
These are the tool's results for the Example button. Googlebot ignores the * group entirely because a group names it; Googlebot-News, which has no group, falls back to the Googlebot group.
How rule matching works (longest match wins)
- Pick the group. The crawler looks for groups whose
User-agentmatches its token (case-insensitive). If several groups name it, they're combined. Google's specialised crawlers fall back toGooglebot. If no group names it, the*group applies; if there's no*group either, everything is allowed. - Find matching rules. Each
Allow/Disallowpath is compared with the start of the URL path and query. - Longest wins. The rule with the longest path decides. If an Allow and a Disallow are equally long, Allow wins.
/robots.txtitself is always allowed, and an emptyDisallow:matches nothing.
Wildcards and $ anchors
| Pattern | Matches | Doesn't match |
|---|---|---|
/shop | /shop, /shop/, /shopping, /shop?id=2 | /Shop |
/shop/ | /shop/, /shop/shoes | /shop |
/*.pdf | /a.pdf, /a.pdf?x=1, /x/a.pdf.html | /a.PDF |
/*.pdf$ | /a.pdf | /a.pdf?x=1 |
/*? | any URL with a query string | /page |
Paths are case-sensitive and matched from the start. * matches any characters, including /.
Testing different user agents
The same file can treat crawlers very differently, which is why the tester asks for the user agent. A common surprise: adding a User-agent: Googlebot group makes Googlebot ignore every rule under User-agent: *, so rules meant for everyone have to be repeated in the Googlebot group. Test both your search crawlers (Googlebot, Bingbot) and any AI crawlers (GPTBot, ClaudeBot, PerplexityBot) you meant to block.
Syntax errors and unsupported lines
Crawlers skip lines they don't understand without telling you. The tester flags them: misspelled fields (Disalow), rules placed before any User-agent, sitemap lines without a full URL, and directives Google doesn't support (Crawl-delay, Noindex, Host). It also warns when a file is larger than 500 KiB, the most Google reads. When fetching a live file it explains the status: a 404 means everything is allowed; a 5xx makes Google pause crawling the site.
Blocked vs not indexed
Blocking a URL in robots.txt stops crawlers reading it; it doesn't remove it from search. Google can still index a blocked URL from links, shown without a description ("No information is available for this page"). If you want a page out of search, allow crawling and use a noindex meta tag, then check it with the meta tags analyzer. And if a page you want indexed shows as "Blocked by robots.txt" in Search Console, this tester shows which line is responsible.
Is my text private?
To check a URL, our server requests the address you enter and returns the result to you. We don't store the pages we fetch or keep a list of the addresses you check. Like any website, our hosting provider keeps short-lived technical request logs for security. See the privacy policy.