Robots.txt generator
Build a valid robots.txt: allow and disallow rules for each crawler, your sitemaps, and one-click blocking of AI training crawlers. The output is checked against an independent robots.txt parser.
Free · Runs in your browser · Updated 26 September 2026
Robots.txt generator
Rules
Start from:
- * means every crawler. Or names such as Googlebot or Bingbot, separated by spaces or commas.One path per line. / blocks the whole site. Leave empty to allow everything.Exceptions inside a disallowed path. The longest matching rule wins.Google ignores this. Some other crawlers honour it.
AI crawlers
Sitemaps
One full URL per line.
robots.txt
User-agent: *
Disallow: /admin/
Sitemap: https://example.com/sitemap.xml
- Save the file as
robots.txtin the root of your site, so it loads athttps://your-domain/robots.txt. It asks crawlers to stay out; it does not hide anything or stop a page from appearing in search results.
robots.txt generated.
How it works
Groups, and the most specific rule wins.
User-agent: *
Disallow: /admin/
Allow: /admin/help
User-agent: GPTBot
Disallow: /
Sitemap: https://example.com/sitemap.xml- A file is a set of groups. Each group starts with one or more
User-agentlines and is followed byAllowandDisallowrules. - A crawler obeys only the group that matches it best. Rules under
*do not carry over to a named group: in the example, GPTBot follows its own group and ignores the/admin/rule. - Within a group, the most specific (longest) matching rule wins. If an Allow and a Disallow match equally, Google uses the least restrictive one. That is why Allow: /admin/help works inside Disallow: /admin/.
*matches any run of characters and$marks the end of the URL, so/*.pdf$blocks addresses that end in .pdf.- An empty Disallow: means nothing is blocked. Disallow: / means everything is.
AI crawlers
Who the presets block.
Checked against each company’s own documentation on 26 September 2026. These lists change, so check them again before you rely on them.
| Preset | User-agent names | What they do |
|---|---|---|
| AI training crawlers | GPTBot, ClaudeBot, Google-Extended, Applebot-Extended, CCBot, meta-externalagent, Bytespider, Amazonbot | Collect pages that may be used to train AI models. |
| AI search crawlers | OAI-SearchBot, Claude-SearchBot, PerplexityBot | Build the index behind AI search answers. Blocking them can remove you from those answers. |
| User-triggered fetchers | ChatGPT-User, Claude-User, Perplexity-User | Fetch a page when a person asks an AI tool about it. Vendors say some of these do not follow robots.txt. |
Google states that blocking Google-Extended does not affect a site’s inclusion in Google Search and is not a ranking signal. Blocking Googlebot would.
Tested
Parsed by an independent parser.
The generated files were read by Protego, the robots.txt parser used by Scrapy, which follows Google’s rules for wildcards and precedence. Across seven rule sets, it allowed and blocked the URLs each file was meant to, applied a named group in place of the * group, read the sitemap lines and the crawl-delay. The last run was on 26 September 2026.
FAQ
Questions people ask first.
- Where do I put robots.txt?
- In the root of your site, so it loads at https://your-domain/robots.txt. Crawlers only look there, and each host (including each subdomain) needs its own file. It must be a plain text file saved as UTF-8.
- Does robots.txt hide a page or remove it from Google?
- No. It asks crawlers not to fetch a URL. Google says a page blocked this way can still be indexed from links elsewhere, shown without a snippet. To keep a page out of search results, leave it crawlable and use a noindex tag; if robots.txt blocks the page, the crawler never sees the noindex.
- Is robots.txt a security measure?
- No. The standard says these rules are not a form of access authorisation, and listing a path in the file makes it public. Protect private pages with a login, not with a Disallow line.
- Should I block AI crawlers?
- It is your decision, and a trade-off. Blocking training crawlers asks companies not to use your pages to train their models. Blocking AI search crawlers can keep you out of AI answers that cite sources. Blocking Google-Extended does not affect Google Search. Compliance is voluntary, and some fetchers that act for a user say they do not follow robots.txt.
- What does Crawl-delay do?
- It asks a crawler to wait a number of seconds between requests. Google ignores it; some other crawlers honour it. The generator writes it if you set it, and tells you Google does not use it.
- How do I check that Google reads my file the way I expect?
- Google Search Console has a robots.txt report that shows the file Google fetched and any problems it found. Also try a few URLs you care about after every change.
Related
More free tools.
From the team behind the tools
Building a product?
Daniotech builds web, mobile and real-time software for startups and teams. Tell us what you are making and who it is for.