Crawl depth vs crawl budget
Crawl depth and crawl budget are related, frequently confused, and fixed in different ways. Depth is a property of your link structure: how many hops from the homepage a page sits. Budget is a property of the crawler's willingness: how many requests it is prepared to spend on your site in a period. Depth decides which pages get visited within the budget; budget decides how much visiting happens at all. Most sites have a depth problem and think they have a budget problem.
Run the Crawl Depth Checker on your site — free, no account.
Two different things
crawl depth crawl budget ----------- ---------------------- ------------------------- what it is hops from homepage requests per period owned by your link structure the crawler measured in clicks URLs fetched per day you change it by adding/removing links by server speed + URL count symptom specific pages stale whole site crawled slowly
Google describes budget as the product of two things: crawl capacity (how fast your server responds without straining) and crawl demand (how much it wants your content, driven by popularity and staleness). Neither is directly settable — you influence capacity with performance and demand with quality and freshness.
Depth, by contrast, is entirely yours. It is a fact about your links and you can change it this afternoon.
How depth spends budget
The interaction is the part worth understanding, because it explains why depth problems present as budget problems.
A crawler works outward from known entry points. Within any given crawl, shallow pages are reached first and deep pages last. When the budget runs out mid-crawl, it runs out at the deep end — so the pages that go stale are always the deep ones, regardless of how important they are.
budget: 1,000 URLs/day, site: 5,000 pages deep structure flat-ish structure d1: 5 crawled daily d1: 50 crawled daily d2: 80 crawled daily d2: 900 crawled daily d3: 600 crawled ~2 days d3: 3,000 crawled ~4 days d4: 2,000 crawled ~weekly d4: 1,050 crawled ~weekly d5: 2,315 crawled ~monthly Same budget. The deep site's tail updates 4x slower.
This is why reducing depth often looks like gaining budget: you did not get more requests, you redistributed the ones you had toward pages you care about.
Which one is your bottleneck?
The diagnosis is a two-question decision, and both questions are answerable from the crawl stats report in Search Console plus one crawl of your own.
- Is the whole site crawled slowly, or only part of it? Uniform slowness across all depths points at budget. Fresh shallow pages with stale deep ones points at depth.
- How many indexable URLs do you have versus how many pages you meaningfully publish? A ratio far above 1 means something templated is generating URLs, and that is a budget problem you caused.
symptom likely cause fix ----------------------------------- ---------------- --------------------- new posts indexed in hours, depth internal links, hubs old deep pages months stale everything crawled slowly, budget: capacity server speed, TTFB server response times high indexable URLs >> real pages budget: waste facets, params, dupes low-quality pages crawled often, budget: demand prune, consolidate good pages ignored
The third row is the most common on large sites and the least often diagnosed. If a store has 40,000 products and 900,000 indexable URLs, no amount of server tuning fixes the crawl — the budget is being spent on URLs nobody decided to publish. That is the ecommerce faceted-navigation problem arriving in a different report.
Fixing each
If it is depth
- Add contextual links from shallow, well-linked pages to the deep pages that matter.
- Build hub pages that gather related deep content, which cuts depth for many pages with one page of work.
- Check that pagination is crawlable — an infinite scroll with no paginated links can leave an entire archive unreachable.
- See how to improve crawl depth for the full sequence.
If it is budget capacity
- Reduce server response time. Crawl rate is throttled by how fast you answer, so TTFB improvements translate fairly directly.
- Fix 5xx errors and timeouts. Errors cause crawlers to back off, and the backoff outlasts the errors.
- Watch redirect chains — each hop is a request spent on nothing.
If it is budget waste
- Reduce the number of indexable URLs: canonicalise parameter variants, noindex multi-facet combinations, stop indexing internal search results.
- Fix broken links and long redirect chains, which spend requests on nothing.
- Prune or consolidate genuinely thin pages rather than leaving them to absorb crawl attention.
Order of operations Waste first, depth second, capacity third. Cutting the indexable URL count is usually the largest single win and it makes the depth work easier, because a smaller site is a shallower site at the same link density. Measuring crawl efficiency covers how to quantify the waste.
When crawl budget genuinely does not matter
Google has said plainly that most sites do not need to think about crawl budget, and that is worth taking at face value. Under roughly a few thousand URLs, with a server that responds reliably, budget is not your constraint and optimising for it is wasted effort.
Depth still matters at that size, because depth is not about how many pages get crawled — it is about how internal authority is distributed among them. A 300-page site has no budget problem and can still bury its most valuable page five clicks down where it receives almost nothing. That is a structural issue and no amount of crawl capacity fixes it.
FAQ
Are crawl depth and crawl budget the same thing?
No. Crawl depth is how many link hops from the homepage a page sits — a property of your link structure that you control. Crawl budget is how many requests a crawler will spend on your site in a period, driven by your server capacity and the crawler's demand for your content.
Does reducing crawl depth increase crawl budget?
Not directly, but it changes who gets the budget. Crawlers work outward from entry points, so when the allocation runs out it runs out at the deep end. Bringing important pages closer means they are reached before the budget is exhausted, which usually looks like an increase.
How do I know if I have a crawl budget problem?
Compare your indexable URL count with the number of pages you meaningfully publish. If it is far higher, something templated is generating URLs and spending your allocation. Also check whether crawling is uniformly slow across all depths, which points at capacity, or only slow for deep pages, which points at depth.
Do small sites need to worry about crawl budget?
Generally no. Under a few thousand URLs with a reliably responding server, budget is not the constraint. Depth still matters at that size though, because it governs how internal authority is distributed rather than how many pages get fetched.
Does page speed affect crawl budget?
Yes, through crawl capacity. Google adjusts its request rate based on how fast your server responds and whether it shows signs of strain, so lowering response times and eliminating 5xx errors generally raises the ceiling. Errors are the sharper lever — crawlers back off after them and the backoff outlasts the errors.
Should I use robots.txt to save crawl budget?
Sparingly and precisely. Blocking genuinely worthless URL patterns saves requests, but a blocked URL cannot have its canonical tag read and passes nothing from links pointing at it. For pages that should not rank but should still consolidate signals, noindex or canonicalisation is the right tool.
Sources
Large site owner's guide to managing crawl budget — Google Search Central
Put this into practice
Run the Crawl Depth Checker to see this on your own site, or run the full structural audit for the complete picture — both free, no account required.