Documentation Index
Fetch the complete documentation index at: https://wiki.platelunchcollective.com/llms.txt
Use this file to discover all available pages before exploring further.
Technical implementation · AI Search Infrastructure
Definition
Crawl budget is the number of pages a search engine or AI crawler will index from a site within a given time period. It is determined by the crawler’s assessment of crawl capacity (server health and speed) and crawl demand (popularity and staleness of content).
Why It Matters for AI Search
For large sites, crawl budget determines which pages get indexed and which get ignored. Pages that are not crawled are not citeable. Optimizing crawl budget — through clean site architecture, fast load times, elimination of duplicate or thin content, and proper use of robots.txt — ensures that the pages most worth citing are the ones that actually get indexed.
Relevant Plate Lunch Collective Services
AI SEO
