Allow AI citations on robots.txt

A robots.txt that is too strict costs you twice. Search engines index less of the site, and AI assistants cannot read the pages they would otherwise cite. The video shows what that looks like on a real file.

You can be selective. AI companies run one crawler to collect training data and another to fetch a page when someone asks a question. Refusing the first and allowing the second keeps your pages out of training sets and still eligible for citations. Controlling what AI crawlers see lists the crawlers and says which group each one belongs to.

To see where you stand, the free AI crawler checker reports which AI bots your robots.txt blocks. A page has to be readable before it can be quoted, and Answer Engine Optimization covers what makes it quotable.