Sitemaps, Robots.txt and Canonical Tags

An XML sitemap is a file that lists the important pages on a site, helping search engines find and prioritize them, especially useful for large sites or pages with few internal links pointing to them.

A robots.txt file tells search engine bots which parts of a site they should not crawl, such as admin pages or duplicate content used internally. It’s a set of instructions, not a security measure, since the file itself is publicly visible.

A canonical tag tells search engines which version of a page is the "main" one when multiple URLs show very similar or identical content, for example a product page accessible through several different URL parameters. Without it, search engines may split ranking signals across duplicates or pick the wrong version to show.

Robots.txt now has to account for an entire ecosystem of AI crawlers, not just Googlebot and Bingbot, and they are not all the same. Training bots, like Google-Extended or GPTBot, collect content to improve an AI model generally. Retrieval bots, like OAI-SearchBot or PerplexityBot, crawl closer to real time specifically to find and cite sources for a live question. Blocking a retrieval bot removes a site from that platform’s citations entirely; blocking a training bot is a separate decision about data use. Treating every AI crawler as one undifferentiated thing to block or allow means either losing visibility worth having or giving away data worth restricting.

One rendering detail matters more here than almost anything else: several major AI crawlers cannot execute JavaScript the way a browser does. A site relying heavily on client-side rendering, where content only appears after a script runs, may look empty to those crawlers even if it looks complete to a human visitor. Server-side rendering or static generation for public-facing content keeps the meaningful text in the initial HTML response, where every crawler, human-facing or AI, can actually read it.