Robots.txt Tester for Googlebot and AI Crawlers
Paste your robots.txt, enter a page path, and see which search and AI crawlers are allowed or blocked, and which exact rule decided it. It follows the standard matching rules, including wildcards and the longest-match rule.
Free, no account. By All I Need For My Website. Everything runs in your browser.
Paste the contents of yoursite.com/robots.txt. Nothing is uploaded; the test runs in your browser.
| Crawler | This path | Why |
|---|---|---|
| GooglebotGoogle Search | Allowed | No rule matches this path |
| BingbotBing Search and Copilot | Allowed | No rule matches this path |
| GPTBotOpenAI model training | Blocked | Disallow: / |
| OAI-SearchBotChatGPT search results | Allowed | No rule matches this path |
| ChatGPT-UserChatGPT visiting a page on request | Allowed | No rule matches this path |
| ClaudeBotAnthropic | Allowed | No rule matches this path |
| PerplexityBotPerplexity search index | Allowed | No rule matches this path |
| Google-ExtendedOpt-out token for Gemini training, not a separate crawler | Allowed | No rule matches this path |
| Applebot-ExtendedOpt-out token for Apple AI training | Allowed | No rule matches this path |
| CCBotCommon Crawl, a source for many AI datasets | Allowed | No rule matches this path |
Sitemap declared: https://yoursite.com/sitemap.xml
Crawler names change as vendors update them, so confirm current names in each vendor's documentation. A robots.txt rule is a request that well-behaved crawlers follow, not a security control.
How to use the Robots.txt Tester
- Open yoursite.com/robots.txt in your browser and copy everything on the page.
- Paste it into the box above. The results update as you type.
- Enter the path of a page you care about, such as /blog/my-post or /pricing.
- Read the table. Any crawler marked Blocked cannot fetch that page, and the Why column shows the rule responsible.
- Test your homepage (/) too. A rule that blocks / blocks everything for that crawler.
Why one wrong robots.txt line costs so much
Robots.txt looks like a small technical file, but a single misplaced Disallow rule can quietly remove an entire section of a site from Google, or block ChatGPT, Perplexity and Claude from ever reading it. The mistake usually happens during a redesign or migration, when a rule meant for a staging environment gets copied into the live file and never removed.
Googlebot, GPTBot, PerplexityBot and ClaudeBot are separate user agents, and a broad Disallow under User-agent: * applies to all of them unless they have their own group. A crawler that has its own group ignores the * group completely, which is why a file that looks safe can behave differently than expected. Test the exact paths that matter: your homepage, your top pages and your sitemap location.
Robots.txt controls crawling, not indexing. A blocked page can still appear in results if other sites link to it. To keep a page out of results, use a noindex tag on a page that crawlers are allowed to fetch.
Tap underlined terms to see their definition.
Common mistakes to avoid
Testing only the homepage
A rule can block /blog/ or /products/ while / is fine. Test at least one URL from each section of the site.
Blocking the folders that hold CSS and JavaScript
Disallowing something like /assets/ or /wp-content/ can stop Google from rendering the page the way a visitor sees it, which makes the page harder for it to understand.
Forgetting that rules match by prefix
"Disallow: /blog" also blocks /blog-news/ and /blogger. Add a trailing slash, or end the pattern with $, to be exact.
Assuming every crawler obeys the file
Robots.txt is a request. Google, OpenAI, Anthropic and Perplexity say their crawlers follow it, but others may not. To actually enforce a block you need a firewall or CDN rule.
Worked example: why one named group overrides the rest
User-agent: GPTBot Disallow: / User-agent: * Disallow: /private/
Test the path /blog/my-post against this file. GPTBot is blocked, because its own group says Disallow: /. Googlebot and PerplexityBot are allowed, because they fall back to the * group, which only blocks /private/. A crawler follows the most specific group that names it and ignores the others, which is why one named group can quietly override your general rules.
After you use it
When the results look right, upload the file to the root of your domain and open yoursite.com/robots.txt to confirm the live version matches what you tested. Google refreshes its copy of the file regularly, but a change can take up to a day to be picked up. Re-run the test after every redesign, migration or plugin change, because that is when files get overwritten.
Common questions
Does blocking GPTBot or ClaudeBot hide my site from ChatGPT or Claude?
It stops those crawlers from fetching your pages, which can reduce how often your content is used or cited. Some bots, such as ChatGPT-User, fetch a page only when a person asks for it, and vendors treat their different bots differently, so check each vendor's documentation for what a specific rule does.
What is Google-Extended?
It is a robots.txt token that lets you opt out of your content being used to train Google's Gemini models. It is not a separate crawler, and blocking it does not affect normal Google Search crawling.
Which rule wins when several match?
The longest matching pattern wins. If an Allow and a Disallow match with the same length, Allow wins.
Is my robots.txt uploaded anywhere?
No. The test runs entirely in your browser.
Related tools