Documentation Index
Fetch the complete documentation index at: https://wiki.platelunchcollective.com/llms.txt
Use this file to discover all available pages before exploring further.
Technical implementation · AI Search Infrastructure
Definition
An AI crawler is an automated bot operated by an AI search platform to index web content for use in retrieval-augmented generation and AI-generated answers. Major AI crawlers include GPTBot (OpenAI), ClaudeBot (Anthropic), and Google-Extended (Google).
Why It Matters for AI Search
AI crawlers determine what content enters the retrieval pools that AI systems draw from when generating answers. A site that blocks AI crawlers via robots.txt is opting out of AI citation entirely. Understanding which crawlers exist, how to allow or block them selectively, and what signals they prioritize is foundational to any AI SEO strategy.
Common Misconception
Blocking AI crawlers prevents AI systems from using your content — but it does not prevent AI systems from referencing content they have already ingested from prior crawls or training data. Blocking is a forward-looking action, not retroactive.
Relevant Plate Lunch Collective Services
AI SEO Citation-Ready Content
