How to Allow AI Crawlers on Webflow
If your robots.txt blocks crawlers such as GPTBot, ClaudeBot or PerplexityBot, those AI tools cannot read your pages, which can reduce how often you are cited. Rules like this are sometimes added by security plugins, hosts or templates without the owner realising.
The fix on Webflow
Go to Project Settings > SEO and check your robots.txt configuration. Remove blocks on AI crawlers (OpenAI, Anthropic, Perplexity, Google, Apple, Meta, Amazon) to allow AI tools to read your content.
Builder menus change over time. If a label looks different, look for the closest option and check Webflow's help centre.
What to know about Webflow
Native robots.txt control
Handled, and editable
Edit directives directly in Site Settings → SEO → Indexing, no code required. Block a staging subdomain or restrict specific sections without a developer.
Why this matters
Blocking AI crawlers means ChatGPT, Perplexity, Claude and other AI tools cannot read your content. Even if they have already indexed some of it, keeping them blocked stops them from refreshing and citing your latest pages.
How to check it worked
- Open yourdomain.com/robots.txt and look for lines naming GPTBot, ClaudeBot, PerplexityBot, Google-Extended or CCBot, or a blanket "Disallow: /" under "User-agent: *".
- Paste the file into the Robots.txt Tester, which shows the result for each crawler on any page path you enter.
Common questions
Should I allow every AI crawler?
That is a business decision. Allowing them makes it possible for AI tools to read and cite your content. Blocking them keeps your content out of those tools but may reduce visibility. Different bots do different jobs, such as training versus live search, so check each vendor's documentation.
Is it my site or Webflow that is blocking them?
Check your robots.txt first. If the rule is there, it came from a setting, plugin or template. The steps above show where to change it on this platform.
More fixes for Webflow