How to Check and Fix robots.txt on WordPress
Robots.txt is a plain-text file at the root of your site that tells crawlers which areas they may fetch. A missing file means no restrictions, which is often fine, but a wrong rule can quietly block Google or AI tools from your whole site.
The fix on WordPress
Yoast SEO and RankMath both generate and manage robots.txt automatically. Check SEO > Tools > File editor (Yoast) or General > Edit robots.txt (RankMath).
Builder menus change over time. If a label looks different, look for the closest option and check WordPress's help centre.
Why this matters
Robots.txt is a file that tells search engines which parts of your site to crawl. Without it Google has no instructions and may waste crawl time on pages you don't want indexed, like admin or thank-you pages.
How to check it worked
- Open yourdomain.com/robots.txt in your browser. A short list of rules means the file exists.
- Paste the contents into the Robots.txt Tester to see which crawlers are allowed or blocked on a given page.
Common questions
Does every site need a robots.txt file?
No. Without one, crawlers assume they may fetch everything. It becomes important when you want to keep specific areas out of crawling or to point to your sitemap.
Can I edit robots.txt on WordPress?
It depends on the platform. See the steps and platform notes on this page for what is editable here.
More fixes for WordPress