Crawl Budget: How to Control Google's Crawling

By Christoph Hein · Head of SEO & Search Consultant · Updated August 2026

Crawl budget is the number of URLs a search engine both wants to and is able to fetch from your site in a given period. It is not a ranking factor – but it is the precondition for everything else: what does not get crawled does not get indexed, and what is not indexed cannot rank. This guide explains what the budget is made of, how to measure it and which levers actually move it.

1. What is crawl budget?

"Crawl budget" is not an official figure you can look up anywhere. The term describes the outcome of two independent quantities that Google calculates separately for every host:

• Crawl capacity limit (formerly called the crawl rate limit): how many parallel connections and requests per second your server can handle without slowing down or returning errors.
• Crawl demand: how many of your URLs Google actually wants to fetch – driven by popularity, freshness and the estimated size of your inventory.

Your effective crawl budget is always the lower of the two. A blazing fast server buys you nothing if Google sees no reason to crawl more. Equally, high demand buys you nothing if your server buckles under load.

2. Crawl capacity and crawl demand in detail

Crawl capacity limit

Googlebot tries to crawl your site without overloading it, and continuously watches two signals to decide how hard it can push:

• Response time: if responses get slower, Googlebot lowers the crawl rate. If they stay fast, it raises the rate incrementally.
• Server errors: a run of 5xx responses or 429 Too Many Requests makes Googlebot back off sharply, sometimes within minutes.

The limit applies per host. shop.example.com and www.example.com have separate budgets – unless they sit behind the same server or IP, in which case they effectively share the capacity.

Crawl demand

Demand is driven by three factors:

1. Popularity: URLs with many internal and external links get refreshed more often. This is where your link equity feeds directly into crawl frequency.
2. Staleness: Google tries to keep the index from going out of date. Pages that demonstrably change often get fetched more often.
3. Perceived inventory: Google estimates how many valuable URLs you have. If a large share of your URLs are duplicates, filter combinations or empty pages, demand drops for the whole site.

Point three matters most and is the most underestimated: duplicates do not just cost you individual fetches, they lower Google's confidence in your entire URL inventory.

3. Who actually needs crawl budget optimisation?

For most sites, crawl budget is a non-issue. Google's own large-site guide names two concrete thresholds where the topic starts to earn attention:

A practical test beats any threshold: do new pages get crawled and indexed within a few days? If yes, you do not have a crawl budget problem. If important new URLs sit in "Discovered – currently not indexed" for weeks, you do. More on that in the indexing guide.

4. Analysing crawl budget in Search Console

The report you want is Settings → Crawl stats. It covers the last 90 days and is the fastest route to a diagnosis. Focus on these breakdowns:

• Total crawl requests: the absolute number says little, the trend says a lot. A sudden drop points to server trouble or newly blocked paths.
• Average response time: the single most important number in the report. Anything above roughly 600 ms is actively suppressing your capacity limit.
• By response: what share of your fetches return 200? If 301, 404 and 5xx together exceed about 20 %, you are measurably burning budget.
• By purpose: the ratio of discovery (new URLs) to refresh (known URLs). A healthy growing site shows a visible share of discovery. If refresh sits above 95 %, Google is barely finding anything new on your site.
• By file type: if images, CSS or JavaScript dominate your fetches, look at rendering and image SEO.
• Host status: checks robots.txt availability, DNS resolution and server connectivity. Anything flagged red here takes priority over everything else.

5. Server log files: the most precise data source

Search Console aggregates and rounds. Your own access logs show every single request – URL, status code, timestamp and user agent. A typical line in combined log format looks like this:

66.249.66.1 - - [18/Aug/2026:09:14:22 +0200] "GET /shoes/?color=red&size=42 HTTP/1.1" 200 18422 "-" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"

Three analyses almost always produce immediately actionable findings:

1. Top directories by crawl share: group all Googlebot requests by the first path segment. It frequently turns out that 60 % of fetches land on areas that are not organic traffic targets at all (search, cart, filters).
2. Crawled but never clicked: join crawled URLs with Search Console performance data. URLs with many fetches and zero impressions over 90 days are optimisation candidates.
3. Never crawled: join your sitemap against the logs. Anything listed in the sitemap but never fetched in 90 days is either poorly linked internally or considered worthless.

One caveat for the analysis: verify that requests are genuinely from Googlebot, because the user agent is trivial to fake. The official method is a reverse DNS lookup followed by a forward lookup – the IP must resolve to googlebot.com or google.com. Alternatively, Google publishes its IP ranges as a JSON file you can match against.

6. The eight biggest crawl budget wasters

From e-commerce audits, the same patterns cause the bulk of wasted budget almost every time – in this order:

1. Faceted navigation: five filters with five values each, freely combinable, produce over 7,700 URL variants per category. It is the number one reason sites with 5,000 products end up with millions of URLs.
2. Internal search result pages: /search?q=... generates unlimited URLs as soon as it is linked anywhere or ends up in a sitemap.
3. Session IDs and tracking parameters: every variation of ?utm_source= or ?sid= is a brand new URL to a crawler.
4. Redirect chains: every hop is a separate request. A three-step chain costs four times what a direct fetch would.
5. Soft 404s: empty categories or sold-out products returning 200 OK with "no results". Google keeps crawling them because they look technically valid.
6. Infinite calendars and pagination: a calendar widget with a "next month" link creates an endless URL chain.
7. Duplicates from case, trailing slash and protocol: four variants of the same page are four crawls. See URLs & status codes.
8. Heavy JavaScript bundles: with client-side rendering, Googlebot has to fetch and execute resources. That consumes capacity which is then unavailable for HTML fetches.

7. Picking the right tool: robots.txt, noindex or canonical

The most common mistake in crawl budget work is reaching for the wrong instrument. The three mechanisms act at completely different points:

The takeaway: only robots.txt saves crawl budget immediately. noindex and canonical have to be crawled before they can do anything. Google does lower the crawl frequency of such URLs over time, but they never fully leave the crawl cycle.

8. Controlling faceted navigation and parameter URLs

Facets are the single biggest lever for online shops. The approach: decide per filter type whether it should be indexable, crawlable but not indexable, or not crawlable at all.

A robust robots.txt for this scenario – note that Allow has to be more specific than Disallow, because Google applies the longest matching rule:

User-agent: *

# Block sorting, view modes and pagination duplicates
Disallow: /*?*sort=
Disallow: /*?*view=
Disallow: /*?*sid=
Disallow: /*?*utm_

# Block internal search entirely
Disallow: /search
Disallow: /*?q=

# Cart and checkout
Disallow: /cart
Disallow: /checkout/

# Explicitly allow resources needed for rendering
Allow: /*.css$
Allow: /*.js$

Sitemap: https://example.com/sitemap.xml

Two rules go with this. First, Crawl-delay does not apply – Google ignores the directive entirely (Bing and Yandex honour it). Second, every URL class you block must first disappear from internal links and sitemaps, or you will collect "Indexed, though blocked by robots.txt" warnings.

9. Server speed and status codes

Capacity is half the equation, and it is the half you control directly.

• Time to first byte: Googlebot judges raw server response time, not your users' Core Web Vitals. Server-side caching for category pages moves the needle far more here than any front-end work.
• HTTP/2: Googlebot has crawled over HTTP/2 since late 2020 where servers support it. Multiplexing measurably lowers the per-request connection overhead.
• Compression: gzip or brotli reduce transferred volume, which eases the byte budget shown as "total download size" in the crawl stats report.
• Conditional requests: serve correct ETag and Last-Modified headers. Googlebot can then request with If-Modified-Since and you answer 304 Not Modified – a fraction of the bytes for the same information.

On status codes: 404 is harmless and the correct code for removed content – Google automatically reduces how often it revisits. The dangerous ones are 5xx runs, because they lower the capacity limit for the whole domain, and 302 for permanent moves, because Google keeps crawling the old URL indefinitely.

10. Internal linking, click depth and sitemaps

Crawl demand follows links. Anything buried deep in your architecture gets fetched less often.

• Keep click depth under five: URLs more than five clicks from the homepage are demonstrably crawled far less. Sensible pagination on category pages plus hub pages are the most effective countermeasures.
• Eliminate orphan pages: URLs that exist only in the sitemap and are linked nowhere count as unimportant. A sitemap is a suggestion, not a substitute for internal links.
• Keep sitemaps honest: Google uses lastmod when it trusts the value. If your CMS stamps every date to "today" on each deployment, the field gets ignored. priority and changefreq are not used by Google at all.
• Segment your sitemaps: separate files for products, categories and editorial content make it visible in the sitemap report which area has indexing trouble.
• List indexable URLs only: no redirects, no noindex pages, no blocked paths. Details in the sitemap guide.

11. Crawl budget and AI crawlers

Since 2024 a significant share of bot traffic no longer comes from search engines. GPTBot, ClaudeBot, PerplexityBot, Bytespider and CCBot crawl aggressively at times – and they draw on the same server capacity as Googlebot. On load-sensitive systems that can indirectly lower your crawl budget for Google.

To be precise about the mechanism: these bots do not share a budget with Google. They compete only for your server resources. So restricting them helps Googlebot indirectly, through capacity.

# Example: exclude AI training crawlers, allow search engines
User-agent: GPTBot
Disallow: /

User-agent: CCBot
Disallow: /

User-agent: Bytespider
Disallow: /

User-agent: *
Disallow: /search
Allow: /

A common misconception: Google-Extended is not a crawler. The token only controls whether your content may be used for Gemini models – it has no effect on how much Googlebot crawls for Search. Details in the guide to llms.txt & AI crawlers.

12. Five myths that save no crawl budget

1. "Crawl-delay in robots.txt throttles Googlebot." No. Google ignores the directive completely and logs it as an unknown rule.
2. "More sitemaps mean more crawling." No. Sitemaps help with discovery; they raise neither capacity nor demand.
3. "priority and changefreq control crawl frequency." No. Google does not use either field.
4. "noindex saves crawl budget straight away." No. The URL has to be crawled for the directive to be read at all.
5. "More crawling means better rankings." No. Crawling is a prerequisite for indexing, not a quality or ranking signal.

13. Crawl budget audit checklist

1. Open crawl stats: host status green? Response time under 600 ms?
2. Check the share of 200 responses. Below 80 %? Clean up status codes.
3. Assess the discovery-to-refresh ratio.
4. Export 30 days of access logs and group Googlebot requests by directory.
5. Compare the top 10 directories by crawl share against actual clicks from Search Console.
6. Build a parameter inventory: which parameters exist, and which create standalone content?
7. Block sorting, view and session parameters in robots.txt.
8. Block internal search and cart paths.
9. Shorten redirect chains to a single hop.
10. Identify soft 404s and convert them to real 404 or 410 responses.
11. Clean up sitemaps: indexable URLs only, honest lastmod.
12. Link orphan URLs internally or remove them.
13. Enable ETag/Last-Modified and verify 304 responses.
14. Re-measure after four weeks: has the share of discovery crawls increased?

14. Frequently asked questions

Read next

About the Author

Christoph Hein

Head of SEO at Popken Fashion Group & independent Search Consultant

Christoph has spent 10+ years in search, currently steering organic strategy for 5 fashion brands across 13 countries and more than 30 domains. Alongside his in-house and consulting work, he founded niche content portals such as Angelmagazin.de and BaristaCompass.com, and built the Rank-O-Saur extension to make technical SEO audits effortless. Every guide here is grounded in hands-on, data-driven practice rather than theory.

christophhein.com ↗ LinkedIn ↗