About the Robots.txt Tester
This tool fetches /robots.txt from the root of a site and parses it into User-agent groups, each with its Allow and Disallow rules and any Crawl-delay. It also lists the Sitemap URLs the file declares, collects directives it doesn't recognize (such as Host or Clean-param), and shows the raw file.
It flags common problems: a missing file (404 or 410, which crawlers treat as "allow everything"), a 5xx error (which Google treats as "disallow everything" until the server recovers), a Content-Type other than text/plain, a file over Google's 500 KiB limit, rules placed before any User-agent line, an invalid Crawl-delay, and Disallow: / for User-agent: *.
How to use it
- Enter a domain or any URL on the site; only the scheme and host are used.
- Select Fetch robots.txt.
- Review the warnings, then each group's rules and the sitemap list.
Examples
- Fetch the robots.txt for www.google.com: a long rule list plus a few crawler-specific groups.
- Fetch the robots.txt for en.wikipedia.org: dozens of groups for individual named crawlers.
- Fetch the robots.txt for example.com: a site with no file, so you see the 404 warning.
Common questions
Does Disallow remove a page from Google?
No. Disallow stops crawling, but a blocked URL can still be indexed if other pages link to it. To keep a page out of search results, allow crawling and add a noindex robots meta tag or X-Robots-Tag header.
Which group applies to a crawler?
A crawler uses the group with the most specific User-agent match for its name and ignores the others; User-agent: * applies only when no named group matches (RFC 9309).
Does Google honor Crawl-delay?
No. Googlebot ignores Crawl-delay; some other crawlers, including Bing, do respect it.
Related tools
- Meta Tag & Open Graph Checker: check a page's robots meta tag and canonical link.
- Redirect Checker: make sure the robots.txt URL doesn't redirect unexpectedly.
- Security Headers Checker: review the site's HTTP response headers.