Skip to main content
Technical implementation · AI Search Infrastructure

​
Definition

Crawl budget is the number of pages a search engine or AI crawler will index from a site within a given time period. It is determined by the crawler’s assessment of crawl capacity (server health and speed) and crawl demand (popularity and staleness of content). For large sites, crawl budget determines which pages get indexed and which get ignored. Pages that are not crawled are not citeable. Optimizing crawl budget — through clean site architecture, fast load times, elimination of duplicate or thin content, and proper use of robots.txt — ensures that the pages most worth citing are the ones that actually get indexed.

AI Crawler

Indexability

Technical SEO

Site architecture

Robots.txt

​
Relevant Plate Lunch Collective Services

AI SEO