Skip to content

robots.txt generator

Build a robots.txt, including a considered answer on AI crawlers.

Free, no signup, and it runs entirely in your browser.

AI crawlers
Output
User-agent: *
Disallow:

Save this as robots.txt in your site root.

About this tool

robots.txt controls crawling, not indexing — a page blocked here can still appear in results if something links to it, just without a description. To keep a page out of an index you need a noindex tag, which a crawler can only see if you let it fetch the page. Blocking a URL you are trying to remove is the classic own goal.

  • The AI question is two questions. Refusing training crawlers costs you nothing in traffic. Refusing the search and citation crawlers removes you from answers that would have cited and linked you — and most "block AI" advice conflates them.
  • Reference your sitemap here. It is the one place every crawler looks without being told, and it costs a single line.
  • The file is public. Listing /admin as disallowed advertises that /admin exists, so use it for crawl control, never for secrecy.
  • It must live at the domain root and be named exactly robots.txt. A subdirectory copy is ignored.

Questions

Does robots.txt stop a page being indexed?
No. It stops it being crawled. A blocked page can still be listed if other pages link to it. Use a noindex meta tag for that, and leave the page crawlable so the tag can be read.
Should I block AI crawlers?
It depends which. Training crawlers like GPTBot and ClaudeBot send you no traffic, so blocking them costs nothing. Search crawlers like OAI-SearchBot and PerplexityBot fetch pages to answer questions and cite you — blocking those loses referrals.
Where does robots.txt go?
At the root of the domain, as https://example.com/robots.txt. Each subdomain needs its own; one at the apex does not cover them.

Related tools