How to Check and Fix robots.txt on Webflow
Robots.txt is a plain-text file at the root of your site that tells crawlers which areas they may fetch. A missing file means no restrictions, which is often fine, but a wrong rule can quietly block Google or AI tools from your whole site.
The fix on Webflow
Go to Project Settings > SEO > Indexing - Webflow generates robots.txt automatically when you publish. Make sure "Disable Webflow subdomain indexing" is not blocking your custom domain.
Builder menus change over time. If a label looks different, look for the closest option and check Webflow's help centre.
What to know about Webflow
Native robots.txt control
Handled, and editable
Edit directives directly in Site Settings → SEO → Indexing, no code required. Block a staging subdomain or restrict specific sections without a developer.
Why this matters
Robots.txt is a file that tells search engines which parts of your site to crawl. Without it Google has no instructions and may waste crawl time on pages you don't want indexed, like admin or thank-you pages.
How to check it worked
- Open yourdomain.com/robots.txt in your browser. A short list of rules means the file exists.
- Paste the contents into the Robots.txt Tester to see which crawlers are allowed or blocked on a given page.
Common questions
Does every site need a robots.txt file?
No. Without one, crawlers assume they may fetch everything. It becomes important when you want to keep specific areas out of crawling or to point to your sitemap.
Can I edit robots.txt on Webflow?
It depends on the platform. See the steps and platform notes on this page for what is editable here.
More fixes for Webflow