> ## Documentation Index
> Fetch the complete documentation index at: https://wiki.platelunchcollective.com/llms.txt
> Use this file to discover all available pages before exploring further.

# Web Crawl

> A web crawl is the automated process by which search engines and AI systems systematically browse the web to discover, fetch, and index web pages.

*Technical implementation* · *Technical SEO*

## Definition

A web crawl is the automated process by which search engines and AI systems systematically browse the web to discover, fetch, and index web pages. Crawlers follow links from known pages to discover new pages, download content, and add it to their indexes.

## Why It Matters for AI Search

Web crawls are how AI systems discover and index content. For brands publishing new content — glossary entries, service pages, blog posts — ensuring that content is linked from already-crawled pages and included in XML sitemaps accelerates crawl discovery. Understanding crawl behavior — which pages are crawled frequently versus infrequently — helps prioritize which pages receive the strongest internal link support for citation-critical content.

## Related Terms

<CardGroup cols={2}>
  <Card title="AI Crawler" href="/ai-search-glossary/ai-crawler" />

  <Card title="Crawl budget" href="/ai-search-glossary/crawl-budget" />

  <Card title="XML sitemap" href="/ai-search-glossary/xml-sitemap" />

  <Card title="Indexability" href="/ai-search-glossary/indexability" />

  <Card title="Internal linking" href="/ai-search-glossary/internal-linking" />
</CardGroup>

## Relevant Plate Lunch Collective Services

[AI SEO](https://www.platelunchcollective.com/services/ai-seo)