> ## Documentation Index
> Fetch the complete documentation index at: https://wiki.platelunchcollective.com/llms.txt
> Use this file to discover all available pages before exploring further.

# Robots.txt

> Robots.

*Technical implementation* · *Technical SEO*

## Definition

Robots.txt is a text file placed in a website's root directory that instructs web crawlers — including AI bots — which pages or sections they are permitted or forbidden to access. Each major AI crawler (GPTBot, ClaudeBot) checks robots.txt before crawling, though some crawlers such as PerplexityBot have been observed to not consistently adhere to these directives.

## Why It Matters for AI Search

Robots.txt misconfiguration is one of the most common causes of AI citation invisibility. A brand that blocks all bots, or uses overly broad disallow rules, may inadvertently prevent AI crawlers from accessing key content. Reviewing robots.txt specifically for AI crawler rules — ensuring GPTBot, ClaudeBot, and PerplexityBot are permitted — is a foundational AI SEO audit step.

## Related Terms

<CardGroup cols={2}>
  <Card title="AI Crawler" href="/ai-search-glossary/ai-crawler" />

  <Card title="GPTBot" href="/ai-search-glossary/gptbot" />

  <Card title="ClaudeBot" href="/ai-search-glossary/claudebot" />

  <Card title="Indexability" href="/ai-search-glossary/indexability" />

  <Card title="AI-first indexing" href="/ai-search-glossary/ai-first-indexing" />
</CardGroup>

## Relevant Plate Lunch Collective Services

[AI SEO](https://www.platelunchcollective.com/services/ai-seo)