Free Robots.txt Generator
Build a robots.txt file by filling in the paths you want kept out, choosing whether AI crawlers can read your site, and adding your sitemap. The file updates live as you type, and you can copy or download it when it looks right. Everything runs in your browser - the fetch-from-domain option in Test mode is the only thing that leaves it.
Ignored by Google. Honoured by Bing and several AI crawlers.
robots.txt
User-agent: * Disallow: /admin/ Disallow: /cart/ Disallow: /checkout/ User-agent: GPTBot Disallow: User-agent: ClaudeBot Disallow: User-agent: PerplexityBot Disallow: User-agent: Google-Extended Disallow:
What robots.txt controls, and what it does not
robots.txt controls crawling. It tells a bot which paths it may request. That is all it does.
It does not control indexing, and this is the single most common misunderstanding about the file. If you block a page in robots.txt and somebody links to it, Google can still list that URL in search results - it just shows the URL with no description, because it was never allowed to read the page. Blocking has made the listing worse rather than removing it.
To keep a page out of search results, you have to let Google crawl it and put a noindex tag on it. Those two instructions are in direct tension: a page blocked in robots.txt can never be crawled, so its noindex tag can never be read. Blocking a page you have already noindexed will strand it in the index indefinitely.
robots.txt is also advisory. Well-behaved crawlers honour it. Scrapers and bad actors ignore it entirely, and nothing in the file can stop them.
Every directive, and how conflicts resolve
User-agent names the bot the following rules apply to. An asterisk means every bot that does not have its own block.
Disallow is a path prefix that may not be crawled. Disallow slash blocks the entire site. Disallow with nothing after it blocks nothing, which is how you explicitly allow a bot everything.
Allow carves an exception out of a Disallow.
Sitemap gives the absolute URL of your sitemap. It sits outside any user-agent block and applies to the whole file. Don’t have one yet? The sitemap generator builds it in seconds.
Two rules decide what happens when directives conflict, and they surprise people. First, a bot obeys only the single most specific user-agent block that matches it - if there is a GPTBot block, GPTBot ignores the asterisk block entirely, including any Disallow lines you assumed were global. Second, within a block, the most specific matching rule wins regardless of order, and where an Allow and a Disallow are the same length, Allow wins.
Here /admin/private/ is blocked and /admin/public/ is crawlable, because the Allow path is longer and therefore more specific.
The AI crawlers, one by one
This is the part of robots.txt that has actually changed, and most guidance on the file has not caught up.
GPTBot is OpenAI. It crawls for training data. Blocking it does not remove you from ChatGPT answers that use live browsing - that is a separate agent.
ClaudeBot is Anthropic, and does the equivalent job for Claude.
PerplexityBot is Perplexity, which is more citation-driven than the others - it names and links its sources in answers, so blocking it removes you from a surface that sends real referral traffic.
Google-Extended is not a crawler at all. It is a permission flag that controls whether content Googlebot already fetched can be used to improve Gemini and other Google AI products. Setting it has no effect on Google Search crawling, indexing or ranking - those are Googlebot, controlled separately.
CCBot is Common Crawl, an open dataset that many model builders use as a source.
Bytespider is ByteDance.
If you’re deciding how AI systems should read your site more broadly, the llms.txt generator covers the complementary file built for language models.
Should you block AI crawlers?
There is a real trade here and the answer depends on what you sell.
The case for blocking: if your content IS the product - a publisher, a course, a research archive - then models trained on it can answer the question your visitor would otherwise have come to you for. You are supplying the substitute for your own product.
The case for allowing: if your content is how people find you rather than what they buy, blocking removes you from the place a growing share of shortlists now get made. Someone asking an assistant for the best three agencies in your category will get three names, and you cannot be one of them if the assistant has never read your site. For most B2B, SaaS, services and local businesses, allowing is the right call. Not sure where you stand today? Check your AI visibility first.
A middle path that works: allow the crawlers, and block only the paths that hold your genuinely proprietary material - gated resources, documentation you sell, member content. That is what Allow and Disallow are for.
Whatever you choose, decide deliberately. The common failure is not choosing wrongly - it is having a robots.txt written years ago that silently blocks everything, and never revisiting it.
Mistakes that take a site out of Google
Disallow slash left in production. It happens when a staging robots.txt gets deployed live, and it removes the entire site from search. If organic traffic drops off a cliff overnight, check this first, before anything else.
Blocking CSS and JavaScript. Google renders pages to judge them. A blocked stylesheet means it sees an unstyled page and may conclude the page is unusable on mobile.
Putting the file anywhere except the root. It must be at yourdomain.com/robots.txt. A file in a subfolder is not read, and a robots.txt on a subdomain governs that subdomain only.
Using it to hide private URLs. The file is public and anyone can read it, so listing /internal-pricing-2026/ tells the world exactly where to look. Use authentication.
Blocking a page you also want deindexed. Covered above - it strands the page in the index because the noindex tag can never be read.
Test before you publish
Paste a URL and a robots.txt - or let it fetch the live one from your domain - and see, for each named bot, whether that URL is allowed or blocked and which single rule decided it. That is the question a tester actually needs to answer, because precedence is where most robots.txt files behave differently from how their author expected.
Getting crawled by the right things is half of AI visibility. The other half is what happens after they read you - see how Zaprev’s generative engine optimization services work.
FAQ
Frequently Asked Questions
Where does robots.txt go?
At the root of the domain, at example.com/robots.txt, and nowhere else. A file in a subfolder is ignored completely.
Does blocking a page in robots.txt remove it from Google?
No. It stops Google crawling the page, but the URL can still be indexed from external links and shown with no description. To remove a page, allow the crawl and use a noindex tag.
Should I block GPTBot and ClaudeBot?
It depends on what you sell. If your content is the product, blocking protects it. If you want to be recommended when someone asks an assistant for a shortlist in your category, blocking removes you from that answer. Most B2B and service businesses should allow.
What is Google-Extended?
A separate control for whether your content is used to improve Gemini and other Google AI products. It does not affect Google Search crawling or ranking at all - Googlebot handles that and is controlled separately.
Can I use robots.txt to hide private pages?
No. The file is public - anyone can read it, and listing a private path there advertises it. Use authentication.
Do I need a robots.txt file at all?
Not strictly. Without one, everything is crawlable, which is fine for most small sites. You need one when you have paths worth excluding, or when you want to state a sitemap location or an AI crawler policy.