> ## Documentation Index
> Fetch the complete documentation index at: https://wiki.platelunchcollective.com/llms.txt
> Use this file to discover all available pages before exploring further.

# Crawl Budget

> Crawl budget is the number of pages a search engine or AI crawler will index from a site within a given time period.

*Technical implementation* · *AI Search Infrastructure*

## Definition

Crawl budget is the number of pages a search engine or AI crawler will index from a site within a given time period. It is determined by the crawler's assessment of crawl capacity (server health and speed) and crawl demand (popularity and staleness of content).

## Why It Matters for AI Search

For large sites, crawl budget determines which pages get indexed and which get ignored. Pages that are not crawled are not citeable. Optimizing crawl budget — through clean site architecture, fast load times, elimination of duplicate or thin content, and proper use of robots.txt — ensures that the pages most worth citing are the ones that actually get indexed.

## Related Terms

<CardGroup cols={2}>
  <Card title="AI Crawler" href="/ai-search-glossary/ai-crawler" />

  <Card title="Indexability" href="/ai-search-glossary/indexability" />

  <Card title="Technical SEO" href="/ai-search-glossary/technical-seo" />

  <Card title="Site architecture" href="/ai-search-glossary/site-architecture" />

  <Card title="Robots.txt" href="/ai-search-glossary/robotstxt" />
</CardGroup>

## Relevant Plate Lunch Collective Services

[AI SEO](https://www.platelunchcollective.com/services/ai-seo)