Does robots.txt keep a page out of Google?

No — and this is the most costly misunderstanding about the file. Disallow stops a page being crawled, not indexed; a disallowed URL with links pointing at it can still appear in results, listed without a description. To keep a page out of the index, allow the crawl and use a noindex meta tag.

Should I block AI crawlers?

It depends on what you sell. Blocking GPTBot, ClaudeBot and PerplexityBot keeps your content out of assistant answers — which protects original writing, and also removes you from the results people increasingly ask for instead of searching. For a tool or a product, being retrievable is usually worth more than being withheld.

Where does the file go?

The root of the domain, exactly at /robots.txt. It is per host and per protocol, so a subdomain needs its own file, and nothing at /folder/robots.txt is ever read.

Robots.txt Generator

Build a robots.txt by choosing which crawlers to allow, which paths to disallow, and where your sitemap lives — then copy the file into your site's root. Generated in your browser, with the AI crawlers listed separately so admitting or blocking them is a deliberate choice.

Runs in your browser — nothing is uploaded

Questions

Does robots.txt keep a page out of Google?: No — and this is the most costly misunderstanding about the file. Disallow stops a page being crawled, not indexed; a disallowed URL with links pointing at it can still appear in results, listed without a description. To keep a page out of the index, allow the crawl and use a noindex meta tag.
Should I block AI crawlers?: It depends on what you sell. Blocking GPTBot, ClaudeBot and PerplexityBot keeps your content out of assistant answers — which protects original writing, and also removes you from the results people increasingly ask for instead of searching. For a tool or a product, being retrievable is usually worth more than being withheld.
Where does the file go?: The root of the domain, exactly at /robots.txt. It is per host and per protocol, so a subdomain needs its own file, and nothing at /folder/robots.txt is ever read.

Robots.txt Generator

Questions

What people use next