How to Allow AI Crawlers on WordPress
If your robots.txt blocks crawlers such as GPTBot, ClaudeBot or PerplexityBot, those AI tools cannot read your pages, which can reduce how often you are cited. Rules like this are sometimes added by security plugins, hosts or templates without the owner realising.
The fix on WordPress
Go to Yoast SEO > Tools > File Editor or RankMath > General > Edit robots.txt. Remove Disallow rules for AI crawlers.
Builder menus change over time. If a label looks different, look for the closest option and check WordPress's help centre.
What to know about WordPress
Named AI crawler control
You choose one plugin
Some plugins let you allow or block specific AI crawlers, GPTBot, ClaudeBot, PerplexityBot, individually, without touching raw robots.txt syntax.
Why this matters
Blocking AI crawlers means ChatGPT, Perplexity, Claude and other AI tools cannot read your content. Even if they have already indexed some of it, keeping them blocked stops them from refreshing and citing your latest pages.
How to check it worked
- Open yourdomain.com/robots.txt and look for lines naming GPTBot, ClaudeBot, PerplexityBot, Google-Extended or CCBot, or a blanket "Disallow: /" under "User-agent: *".
- Paste the file into the Robots.txt Tester, which shows the result for each crawler on any page path you enter.
Common questions
Should I allow every AI crawler?
That is a business decision. Allowing them makes it possible for AI tools to read and cite your content. Blocking them keeps your content out of those tools but may reduce visibility. Different bots do different jobs, such as training versus live search, so check each vendor's documentation.
Is it my site or WordPress that is blocking them?
Check your robots.txt first. If the rule is there, it came from a setting, plugin or template. The steps above show where to change it on this platform.
More fixes for WordPress