Skip to content
Webmaster Tools
SEORobots.txt Generator

Robots.txt Generator

Build crawl rules and sitemaps.

Start from

A crawler uses only the most specific group that names it, so rules under * don't apply to a bot that has its own group. The longest matching path wins; on a tie Allow wins. * matches any characters and $ anchors the end of the URL. Google ignores Crawl-delay; Bing and some others honor it.

robots.txt


    
Upload the file to the root of the host, e.g. https://example.com/robots.txt; each subdomain needs its own. Robots.txt is advisory: well-behaved crawlers follow it, others don't, and it doesn't hide pages from search. A blocked URL can still be indexed from links elsewhere; to keep a page out of results, allow crawling and use a noindex meta tag or header. Test a live site's robots.txt.

  




    
    
    
About the Robots.txt Generator


This tool writes a robots.txt file: groups of User-agent lines, each with Allow and Disallow rules, an optional Crawl-delay, and Sitemap URLs. Start from a preset (allow everything, block everything, or block AI training crawlers) and add groups and rules as you need them. Paths start with /, and in a path * matches any characters and $ anchors the end of a URL. Sitemap URLs must be absolute.


The AI preset names crawlers whose operators document their tokens: GPTBot, ClaudeBot, CCBot, Google-Extended, Applebot-Extended, PerplexityBot and meta-externalagent. Google-Extended and Applebot-Extended don't crawl at all; they only control whether Google and Apple may use pages for AI models, so blocking them doesn't affect Google Search or Apple search results.


Robots.txt is advisory. Well-behaved crawlers follow it, others ignore it, and it doesn't hide pages: a blocked URL can still appear in search results if other sites link to it. To keep a page out of search, let it be crawled and add a noindex meta tag or X-Robots-Tag header.


How to use it


    

  1. Pick a preset or edit the first group.
    2. 

  2. Add rules and groups for specific crawlers.
    3. 

  3. Add your sitemap URLs.
    4. 

  4. Copy or download the file and upload it to the root of the host.
    5. 

  5. Test the live file with the Robots.txt Tester.
    6. 



Examples


The AI preset, shortened to three crawlers, looks like this:


User-agent: GPTBot
User-agent: ClaudeBot
User-agent: CCBot
Disallow: /

User-agent: *
Allow: /



Common questions


Does Google honor Crawl-delay?


No. Google ignores it and adjusts its crawl rate automatically. Bing and some other crawlers honor it.


Which rule wins when Allow and Disallow both match?


The most specific (longest) matching path. On a tie, Google uses the least restrictive, Allow.


Do subdomains share one robots.txt?


No. Each host, and each protocol and port, needs its own file at its root.