It has two components. Crawl rate limit is how aggressively Google can crawl without slowing your server. Faster responses raise it. Crawl demand is how much Google wants your content, driven by popularity, freshness, and perceived quality.
For most sites under a few thousand pages, crawl budget truly isn't a concern, and worrying about it is a distraction. It becomes decisive on large sites (e-commerce catalogs, marketplaces, programmatic page sets) where thousands of URLs compete for finite attention.
The failure mode isn't that Google crawls too little overall. It's that crawl gets spent on parameter URLs, faceted navigation permutations, and thin pages, leaving important pages crawled rarely and updated slowly in the index.
Why Crawl Budget matters
On large sites, crawl allocation determines how quickly new pages get indexed and how fast changes are reflected. A programmatic page set with 4% indexation usually doesn't have a content problem alone. It has a crawl budget spread across thousands of equally-weighted, low-value URLs.
Where budget usually leaks
A retailer with 8,000 products generating 400,000 crawlable URLs through color, size and sort-order parameters. Google spends most of its allocation on near-duplicate variations, so truly new products take weeks to index. Canonical tags, parameter handling and robots directives redirect that budget to pages that matter, usually with no content work at all.
Common mistakes
Worrying about it on a small site
Under a few thousand URLs, crawl budget is almost never the constraint. Time spent here is time not spent on content or links, which is where the gain actually is.
Faceted navigation left uncontrolled
Filter combinations generate near-infinite URL space. Without canonicals, parameter rules, or robots directives, this alone can consume most of an allocation.
Flat sitemaps with uniform priority
Segment sitemaps by page type so you can monitor indexation rate per segment in Search Console. One flat file tells you nothing diagnostic.
Slow server responses
Crawl rate limit scales with response time. Server performance is an SEO input, not only a user experience one.
Where we work on this