INDEXED.
WordPress SEO guide

How to Allow AI Crawlers on WordPress

Published · Updated

If your robots.txt blocks crawlers such as GPTBot, ClaudeBot or PerplexityBot, those AI tools cannot read your pages, which can reduce how often you are cited. Rules like this are sometimes added by security plugins, hosts or templates without the owner realising.

The fix on WordPress

Go to Yoast SEO > Tools > File Editor or RankMath > General > Edit robots.txt. Remove Disallow rules for AI crawlers.

Builder menus change over time. If a label looks different, look for the closest option and check WordPress's help centre.

What this usually looks like on WordPress

WordPress itself rarely blocks AI crawlers. The usual causes sit around it: a robots.txt rule added by an SEO or security plugin, a one-click AI bot blocker at your CDN or host (Cloudflare offers one, for example), or a firewall plugin that returns an error to bots it does not recognize. That is why robots.txt can look open while a crawler is still turned away.

What to know about WordPress

Named AI crawler control

You choose one plugin

Some plugins let you allow or block specific AI crawlers, GPTBot, ClaudeBot, PerplexityBot, individually, without touching raw robots.txt syntax.

Why this matters

Blocking AI crawlers means ChatGPT, Perplexity, Claude and other AI tools cannot read your content. Even if they have already indexed some of it, keeping them blocked stops them from refreshing and citing your latest pages.

Common mistakes on WordPress

  • Mixing up training and search crawlers

    OpenAI runs GPTBot for model training and OAI-SearchBot for ChatGPT search results, and other AI companies split their crawlers in a similar way. Blocking one does not block the other, so decide what you want before you edit anything.

  • Editing the virtual file while a real one exists

    WordPress serves a virtual robots.txt unless a physical robots.txt file sits in the site root. If a physical file exists, changes made in a plugin screen show nothing on the live site. Open yoursite.com/robots.txt in a browser and compare it with what your plugin says.

  • Fixing robots.txt but not the firewall

    A rule that allows a crawler is useless if the CDN, host or a security plugin answers its requests with a 403 error. Check both layers.

How to check it worked

  1. Open yourdomain.com/robots.txt and look for lines naming GPTBot, ClaudeBot, PerplexityBot, Google-Extended or CCBot, or a blanket "Disallow: /" under "User-agent: *".
  2. Paste the file into the Robots.txt Tester, which shows the result for each crawler on any page path you enter.

Load yoursite.com/robots.txt and read it from top to bottom, then paste it into the Robots.txt Tester and test your homepage and one article for each crawler you care about. For the firewall layer, request a page with the crawler's name as the user agent, for example: curl -I -A "GPTBot" https://yoursite.com/. A 200 response is good and a 403 means something is still blocking it. This only tests user-agent rules, because some firewalls also verify the crawler's IP address. Allowing a crawler gives it permission to read your site. It does not guarantee any AI tool will mention or cite you.

Test AI crawler access with the Robots.txt Tester →Read the lesson behind this fix →

Common questions

Should I allow every AI crawler?

That is a business decision. Allowing them makes it possible for AI tools to read and cite your content. Blocking them keeps your content out of those tools but may reduce visibility. Different bots do different jobs, such as training versus live search, so check each vendor's documentation.

Is it my site or WordPress that is blocking them?

Check your robots.txt first. If the rule is there, it came from a setting, plugin or template. The steps above show where to change it on this platform.

Same fix on other builders

Check your WordPress site for free