Robots.txt generator
Pick what search engines and AI crawlers may read on your site. The file builds as you go, ready to copy or download.
Your robots.txt
Upload the file to the top folder of your site so it opens at yourdomain.com/robots.txt.
What robots.txt can and cannot do
Robots.txt is a set of requests for crawlers. It tells search engines which parts of your site they should skip, and where to find your sitemap.
- It is not a security tool. Anyone can open the file and read the paths you list, and a bot that ignores it can still visit them. Protect private pages with a login.
- It does not remove pages that are already in Google. A blocked page can stay in results. To take a page out, allow crawling and add a noindex tag, then wait for Google to visit again.
- One wrong line can hide your whole site. "Disallow: /" under "User-agent: *" tells every search engine to stay out.
- Do not block your CSS, JavaScript or image folders. Google needs them to see the page the way your visitors do.
Once crawlers can reach your pages, check how they will show in results with the Google search preview tool.
Common questions
Questions people ask about this tool
Where does robots.txt go?
It must sit at the root of your domain, so it opens at yourdomain.com/robots.txt. Search engines do not look for it in folders, and each subdomain needs its own file.
Will blocking a page in robots.txt remove it from Google?
No. It stops Google from crawling the page, but a blocked URL can still show in results if other sites link to it. To keep a page out of Google, let it be crawled and add a noindex tag.
Does blocking Google-Extended hurt my Google rankings?
No. Google-Extended only controls whether Google can use your content for its Gemini AI models. Google Search crawls with Googlebot, which this rule does not touch.
Do all bots obey robots.txt?
The big search engines and the named AI crawlers say they follow it, but it is a request, not a lock. Bots that scrape or spam often ignore it completely.
Want it done for you?
Not sure what your site should block?
We check crawling and indexing on small business sites and fix what keeps pages out of Google. Book a free call and we will look at yours.