How to Allow AI Crawlers on Lovable
If your robots.txt blocks crawlers such as GPTBot, ClaudeBot or PerplexityBot, those AI tools cannot read your pages, which can reduce how often you are cited. Rules like this are sometimes added by security plugins, hosts or templates without the owner realising.
The fix on Lovable
Ask your AI builder to "check the robots.txt file and remove any blocks on AI crawlers like GPTBot, ClaudeBot, PerplexityBot, Google-Extended, Applebot-Extended, Meta-ExternalAgent, and Amazonbot."
Builder menus change over time. If a label looks different, look for the closest option and check Lovable's help centre.
Why this matters
Blocking AI crawlers means ChatGPT, Perplexity, Claude and other AI tools cannot read your content. Even if they have already indexed some of it, keeping them blocked stops them from refreshing and citing your latest pages.
How to check it worked
- Open yourdomain.com/robots.txt and look for lines naming GPTBot, ClaudeBot, PerplexityBot, Google-Extended or CCBot, or a blanket "Disallow: /" under "User-agent: *".
- Paste the file into the Robots.txt Tester, which shows the result for each crawler on any page path you enter.
Common questions
Should I allow every AI crawler?
That is a business decision. Allowing them makes it possible for AI tools to read and cite your content. Blocking them keeps your content out of those tools but may reduce visibility. Different bots do different jobs, such as training versus live search, so check each vendor's documentation.
Is it my site or Lovable that is blocking them?
Check your robots.txt first. If the rule is there, it came from a setting, plugin or template. The steps above show where to change it on this platform.
More fixes for Lovable