robots.txt tester
Test whether a crawler may fetch a path, by the rules crawlers actually use.
Free and no signup. This one fetches the URL from our server; nothing about it is stored.
This one runs from our server rather than your browser, so the address does leave your machine. Nothing about it is stored.
About this tool
The matching rules are counter-intuitive in exactly the way that produces wrong answers. The longest matching rule wins regardless of order, and Allow beats Disallow on an equal-length match — so a Disallow: /admin followed by Allow: /admin/public means the public path is crawlable. Reading the file top to bottom gets that backwards, which is what makes a tester worth having.
- Longest match wins, not first match. This is the single most misunderstood thing about the format, and it decides the answer on precisely the paths people are worried about.
- The * and $ wildcards are supported. Neither is in the original standard and both are honoured by every major crawler, so ignoring them gives the wrong answer on most real files.
- robots.txt controls crawling, not indexing. A blocked page can still be listed if others link to it — to keep it out of an index you need a noindex tag, which means leaving the page crawlable so the tag can be read.
- The file is shown as served, because that is usually what settles the argument.
Questions
- If two rules match, which one wins?
- The longest one, regardless of the order they appear in. On an equal-length match, Allow beats Disallow.
- Does robots.txt stop a page being indexed?
- No, it stops it being crawled. A blocked page can still appear in results without a description. Use a noindex meta tag and leave the page crawlable so the tag can be seen.
- Do wildcards work in robots.txt?
- They are not in the original standard, but * and $ are honoured by every major crawler and used constantly in real files.