> ## Documentation Index
> Fetch the complete documentation index at: https://wiki.platelunchcollective.com/llms.txt
> Use this file to discover all available pages before exploring further.

# AI Crawler

> An AI crawler is an automated bot operated by an AI search platform to index web content for use in retrieval-augmented generation and AI-generated answers.

*Technical implementation* · *AI Search Infrastructure*

## Definition

An AI crawler is an automated bot operated by an AI search platform to index web content for use in retrieval-augmented generation and AI-generated answers. Major AI crawlers include GPTBot (OpenAI), ClaudeBot (Anthropic), and Google-Extended (Google).

## Why It Matters for AI Search

AI crawlers determine what content enters the retrieval pools that AI systems draw from when generating answers. A site that blocks AI crawlers via robots.txt is opting out of AI citation entirely. Understanding which crawlers exist, how to allow or block them selectively, and what signals they prioritize is foundational to any AI SEO strategy.

## Common Misconception

Blocking AI crawlers prevents AI systems from using your content — but it does not prevent AI systems from referencing content they have already ingested from prior crawls or training data. Blocking is a forward-looking action, not retroactive.

## Related Terms

<CardGroup cols={2}>
  <Card title="robots.txt" href="/ai-search-glossary/robotstxt" />

  <Card title="Crawl budget" href="/ai-search-glossary/crawl-budget" />

  <Card title="Indexability" href="/ai-search-glossary/indexability" />

  <Card title="Training corpus" href="/ai-search-glossary/training-corpus" />

  <Card title="Retrieval pipeline" href="/ai-search-glossary/retrieval-pipeline" />
</CardGroup>

## Relevant Plate Lunch Collective Services

[AI SEO](https://www.platelunchcollective.com/services/ai-seo)  [Citation-Ready Content](https://www.platelunchcollective.com/services/citation-ready-content)