Robots.txt generator

Pick what search engines and AI crawlers may read on your site. The file builds as you go, ready to copy or download.

All search engines and bots
Paths to block or allow

Each path starts with a slash, such as /wp-admin/ or /checkout/.

Block AI training crawlers

Each one gets its own rule that blocks it from the whole site.

Your robots.txt

Upload the file to the top folder of your site so it opens at yourdomain.com/robots.txt.

What robots.txt can and cannot do

Robots.txt is a set of requests for crawlers. It tells search engines which parts of your site they should skip, and where to find your sitemap.

Once crawlers can reach your pages, check how they will show in results with the Google search preview tool.

Common questions

Questions people ask about this tool

Where does robots.txt go?

It must sit at the root of your domain, so it opens at yourdomain.com/robots.txt. Search engines do not look for it in folders, and each subdomain needs its own file.

Will blocking a page in robots.txt remove it from Google?

No. It stops Google from crawling the page, but a blocked URL can still show in results if other sites link to it. To keep a page out of Google, let it be crawled and add a noindex tag.

Does blocking Google-Extended hurt my Google rankings?

No. Google-Extended only controls whether Google can use your content for its Gemini AI models. Google Search crawls with Googlebot, which this rule does not touch.

Do all bots obey robots.txt?

The big search engines and the named AI crawlers say they follow it, but it is a request, not a lock. Bots that scrape or spam often ignore it completely.

Want it done for you?

Not sure what your site should block?

We check crawling and indexing on small business sites and fix what keeps pages out of Google. Book a free call and we will look at yours.