INDEXED.
All tools

Robots.txt Tester for Googlebot and AI Crawlers

Paste your robots.txt, enter a page path, and see which search and AI crawlers are allowed or blocked, and which exact rule decided it. It follows the standard matching rules, including wildcards and the longest-match rule.

Free, no account. By All I Need For My Website. Everything runs in your browser.

Paste the contents of yoursite.com/robots.txt. Nothing is uploaded; the test runs in your browser.

Blocked from the entire site: GPTBot. If you want these tools to read or cite your content, remove the matching Disallow: / rule.
CrawlerThis pathWhy
GooglebotGoogle SearchAllowedNo rule matches this path
BingbotBing Search and CopilotAllowedNo rule matches this path
GPTBotOpenAI model trainingBlockedDisallow: /
OAI-SearchBotChatGPT search resultsAllowedNo rule matches this path
ChatGPT-UserChatGPT visiting a page on requestAllowedNo rule matches this path
ClaudeBotAnthropicAllowedNo rule matches this path
PerplexityBotPerplexity search indexAllowedNo rule matches this path
Google-ExtendedOpt-out token for Gemini training, not a separate crawlerAllowedNo rule matches this path
Applebot-ExtendedOpt-out token for Apple AI trainingAllowedNo rule matches this path
CCBotCommon Crawl, a source for many AI datasetsAllowedNo rule matches this path

Sitemap declared: https://yoursite.com/sitemap.xml

Crawler names change as vendors update them, so confirm current names in each vendor's documentation. A robots.txt rule is a request that well-behaved crawlers follow, not a security control.

How to use the Robots.txt Tester

  1. Open yoursite.com/robots.txt in your browser and copy everything on the page.
  2. Paste it into the box above. The results update as you type.
  3. Enter the path of a page you care about, such as /blog/my-post or /pricing.
  4. Read the table. Any crawler marked Blocked cannot fetch that page, and the Why column shows the rule responsible.
  5. Test your homepage (/) too. A rule that blocks / blocks everything for that crawler.

Why one wrong robots.txt line costs so much

Robots.txt looks like a small technical file, but a single misplaced Disallow rule can quietly remove an entire section of a site from Google, or block ChatGPT, Perplexity and Claude from ever reading it. The mistake usually happens during a redesign or migration, when a rule meant for a staging environment gets copied into the live file and never removed.

Googlebot, GPTBot, PerplexityBot and ClaudeBot are separate user agents, and a broad Disallow under User-agent: * applies to all of them unless they have their own group. A crawler that has its own group ignores the * group completely, which is why a file that looks safe can behave differently than expected. Test the exact paths that matter: your homepage, your top pages and your sitemap location.

Robots.txt controls crawling, not indexing. A blocked page can still appear in results if other sites link to it. To keep a page out of results, use a noindex tag on a page that crawlers are allowed to fetch.

Tap underlined terms to see their definition.

Common mistakes to avoid

  • Testing only the homepage

    A rule can block /blog/ or /products/ while / is fine. Test at least one URL from each section of the site.

  • Blocking the folders that hold CSS and JavaScript

    Disallowing something like /assets/ or /wp-content/ can stop Google from rendering the page the way a visitor sees it, which makes the page harder for it to understand.

  • Forgetting that rules match by prefix

    "Disallow: /blog" also blocks /blog-news/ and /blogger. Add a trailing slash, or end the pattern with $, to be exact.

  • Assuming every crawler obeys the file

    Robots.txt is a request. Google, OpenAI, Anthropic and Perplexity say their crawlers follow it, but others may not. To actually enforce a block you need a firewall or CDN rule.

Worked example: why one named group overrides the rest

User-agent: GPTBot
Disallow: /

User-agent: *
Disallow: /private/

Test the path /blog/my-post against this file. GPTBot is blocked, because its own group says Disallow: /. Googlebot and PerplexityBot are allowed, because they fall back to the * group, which only blocks /private/. A crawler follows the most specific group that names it and ignores the others, which is why one named group can quietly override your general rules.

After you use it

When the results look right, upload the file to the root of your domain and open yoursite.com/robots.txt to confirm the live version matches what you tested. Google refreshes its copy of the file regularly, but a change can take up to a day to be picked up. Re-run the test after every redesign, migration or plugin change, because that is when files get overwritten.

Common questions

Does blocking GPTBot or ClaudeBot hide my site from ChatGPT or Claude?

It stops those crawlers from fetching your pages, which can reduce how often your content is used or cited. Some bots, such as ChatGPT-User, fetch a page only when a person asks for it, and vendors treat their different bots differently, so check each vendor's documentation for what a specific rule does.

What is Google-Extended?

It is a robots.txt token that lets you opt out of your content being used to train Google's Gemini models. It is not a separate crawler, and blocking it does not affect normal Google Search crawling.

Which rule wins when several match?

The longest matching pattern wins. If an Allow and a Disallow match with the same length, Allow wins.

Is my robots.txt uploaded anywhere?

No. The test runs entirely in your browser.

Run a full free site check