Crawl budget only matters in 2026 if your site has 1 million or more unique pages changing weekly, 10,000 or more changing daily, or a large share of URLs stuck in Discovered, currently not indexed, per Google’s crawl budget guide; to check, open the Crawl Stats report in Google Search Console, confirm host status is green and most responses are 200, then cut duplicate and parameter URLs before you try to get more crawling. Google’s Crawl Stats documentation says a site with fewer than a thousand pages should not need that report at all, and Cloudflare Radar’s 2025 Year in Review found Googlebot alone sent 4.5% of HTML requests across Cloudflare’s network, more than all other AI bots combined at 4.2%.
Crawl budget is Google’s name for the set of URLs it can and wants to crawl on one hostname. It limits fetching, not ranking. Google revised the guide on July 22, 2026 under the title “Optimize your crawl budget.”
For most business sites, crawl budget is not the problem. Slow servers, broken robots.txt files and near duplicate URLs are, and they show up in the same report.
Not sure whether Googlebot or GPTBot is reaching the pages that win you customers? Get a free AI visibility audit.
Does crawl budget matter for your site in 2026?
Probably not, unless you run a very large or very fast changing site. Google’s guide names three cases: 1 million+ unique pages updated about weekly, 10,000+ unique pages updated daily, or many URLs listed as Discovered, currently not indexed in Search Console. Google calls these rough estimates, not exact thresholds.
The guide’s own opening line sets the bar: “If your site doesn’t have a large number of pages that change rapidly, or if your pages seem to be crawled the same day that they are published, you don’t need to read this guide.” For Google Search, it says a current sitemap and regular checks of the Page Indexing report are “adequate.” Gary Illyes made the same point in Google’s January 2017 post on crawl budget.
Who should care: retailers with faceted category pages, marketplaces, news publishers, job boards and any site that generates URLs from filters or calendars. Google’s crawl rate documentation names faceted navigation, calendars with a URL per date, and Dynamic Search Ad targets as the most common causes of crawl spikes.
A 40 page service business with slow indexing has a different problem. Our guide on fixing Crawled, currently not indexed covers that status, which usually points to page value rather than crawl limits.
How does Google decide how much to crawl?
Google sets crawl budget from two inputs: crawl capacity limit, which is how hard it can hit your server without harm, and crawl demand, which is how much it wants your URLs. Google’s guide defines the budget as the overlap: the set of URLs Google “can and wants to crawl.”
Crawl capacity limit. The July 2026 guide calls this “also known as hostload.” It caps how long your server spends holding connections open for Google, counting parallel connections and their duration. Every site starts at the same conservative default. If responses stay fast and stable, including time to first byte, the limit rises. If latency climbs, or the server returns 5xx errors or HTTP 429, it drops. Google’s 2017 post called the same idea the “crawl rate limit.”
Crawl demand. For Googlebot, demand depends on site size, update frequency, page quality and relevance. Google names three factors you can influence: perceived inventory (how many URLs it knows about), popularity, and staleness. Of these, the guide says inventory “is the factor that you can positively control the most.”
Two details most explainers miss:
- Budgets are per hostname. Google treats www.example.com and code.example.com as separate sites with separate crawl budgets.
- Capacity is shared across Google’s crawlers. Each crawler has its own demand, but the capacity limit is shared. AdsBot demand rises when you run dynamic ad targets, Google Shopping demand rises with merchant feed products, and that load can reduce what is left for Googlebot.
Google’s Myths and facts page adds that AMP and hreflang alternates, CSS, JavaScript and XHR fetches all count.
How do you check your crawl budget in Search Console?
Open Search Console, go to Settings, then Crawl stats. The report works only on a Domain property or a root level URL prefix property and shows total crawl requests, total download size, average response time and host status, plus breakdowns by response, file type, purpose and Googlebot type.
Read it in this order.
1. Host status
Green means Google found no significant availability problems in the past 90 days. Anything else, click through to robots.txt fetching, DNS resolution and server connectivity. Google draws a red line on each chart; for DNS, for example, failures on more than 5% of requests in a day count as an issue.
The robots.txt check matters most. Per Google’s documentation, if robots.txt returns a 429 or 5xx, Google stops crawling for the first 12 hours, then uses the last good copy for up to 30 days. A missing file (404) is fine and means “crawl everything.”
2. Crawl responses and file types
Most responses should be 200, 301 or 304. Each hop in a redirect chain counts as a separate request, so a two hop chain costs three fetches. In File type, look for many small images or resources served from a slower host.
3. Crawl purpose and Googlebot type
Discovery means a URL Google never crawled before; Refresh means a recrawl. After you publish a large new section or submit a sitemap, Google says you should see a bump in discovery crawls. In Googlebot type, a spike labeled AdsBot usually means new Dynamic Search Ads targets, and Google notes AdsBot recrawls those URLs about every 2 weeks.
Google says your crawl rate “should be relatively stable over the time span of a week or two.” A sudden drop usually traces to a new robots.txt rule, slower responses or a higher error rate. You cannot ask Google to raise the rate.
What wastes crawl budget, and how do you fix it?
Duplicate and low value URLs waste the most. Google’s 2017 analysis ranked faceted navigation and session identifiers first, then on-site duplicate content, soft error pages, hacked pages, infinite spaces and proxies, and low quality or spam content.
4. Shrink the URL inventory
Consolidate duplicates first. Where you cannot, block sorted, filtered or infinite scroll URLs with robots.txt. Google’s guide says not to use noindex for this, because Google “will still request, but then drop the page,” spending the fetch anyway. Its Myths page adds nuance: noindex can free some budget “in the long run” once URLs leave the index.
Do not use robots.txt to shuffle budget toward other pages. Google says it “won’t shift this newly available crawl budget to other pages unless Google is already hitting your site’s crawl capacity limit.”
5. Clean up dead ends
Return 404 or 410 for pages that are gone for good. Google calls a 404 “a strong signal not to crawl that URL again,” and its Myths page says 4xx responses, except 429, do not waste crawl budget. Soft 404s, pages that say “not found” but return 200, keep getting crawled. Collapse redirect chains into a single hop.
6. Make every fetch cheaper
Support HTTP 304 Not Modified so unchanged pages cost little bandwidth. Keep sitemaps current with an accurate lastmod. Google’s Myths page says faster loading and rendering lets it crawl more, but that it “might devote more time crawling a site that has more important information, even if it is slower.”
Google lists only two ways to get more budget: add server resources if you see Hostload exceeded in URL Inspection, and raise content value, meaning popularity, user value, uniqueness and serving capacity.
Do AI crawlers like GPTBot and ClaudeBot use up your crawl budget?
Not Google’s. Google’s shared capacity limit covers Google’s own crawlers. GPTBot, ClaudeBot and PerplexityBot are run by OpenAI, Anthropic and Perplexity, each with its own schedule. They do spend your server’s capacity, and Google lowers its limit when your server slows down or returns 5xx errors, whatever caused the load.
Cloudflare Radar’s 2025 Year in Review puts the load in proportion. Googlebot produced over 28% of Verified Bot traffic, OpenAI’s GPTBot about 7.5%, and Microsoft’s Bingbot 6%. For sites billed in North America in October 2025, Googlebot made 45.5% of crawler traffic, Bingbot 14.0% and Meta-ExternalAgent 7.7%. Cloudflare also found training crawls reached as much as 32 times user action crawling at peak.
What each AI company documents:
- OpenAI. OAI-SearchBot decides ChatGPT search visibility and GPTBot gathers training data. If you allow both, OpenAI says it “may use the results from just one crawl for both use cases.” Robots.txt changes take about 24 hours to apply.
- Anthropic. ClaudeBot, Claude-SearchBot and Claude-User honor robots.txt, and Anthropic supports the non standard Crawl-delay rule, for example Crawl-delay: 1 under User-agent: ClaudeBot.
- Perplexity. PerplexityBot builds search results and is not used for model training. Perplexity-User fetches pages for a person and “generally ignores robots.txt rules.” Changes can take up to 24 hours.
Google’s training opt out works differently. Google-Extended has no separate user agent: Google says crawling happens with its existing user agents and the robots.txt token works only as a control, with no effect on inclusion in Google Search.
Before blocking anything to save load, check which bots you need for AI answers. Our list of AI crawlers maps each user agent to its job, and how often AI engines recrawl your site covers how to see their visits in your logs.
How do you slow crawlers down without hurting your site?
For Google, return HTTP 500, 503 or 429 to crawl requests for a few hours, or one to two days at most. Google’s crawl rate documentation, updated October 6, 2026, says this lowers the crawl rate for the whole hostname and recovers once errors fall.
Past 1 to 2 days, URLs that keep returning those codes may drop from the index. With 503 or 429 you can send a Retry-After header, in seconds or as a UTC date. If you cannot serve errors, file a special request naming your preferred rate; Google says it can take several days. Google ignores the Crawl-delay rule.
Bing Webmaster Tools offers direct controls. Its Crawl Control setting lets you set Bingbot faster or slower than normal for each of the 24 hours in a day, so crawling can drop during your busy hours. A Crawl boost option lets Bing raise the rate “to boost index coverage,” with the hourly pattern still applied. Bing also points sites to IndexNow, and Cloudflare’s Crawler Hints can submit to IndexNow automatically.
For AI crawlers, use robots.txt rules per user agent, Crawl-delay for ClaudeBot, and the IP ranges OpenAI and Perplexity publish to verify requests before a firewall rule. We have not tested whether rate limiting an AI crawler changes how often it cites a site.
FAQ
Is crawl budget a ranking factor?
No. Google’s Myths and facts about crawling page says improving your crawl rate “won’t necessarily lead to better positions” and that crawling, while required for a page to appear in results, is not a ranking signal. Gary Illyes said the same in Google’s 2017 post. A page must be crawled before it can rank, but extra crawling does not lift it.
How many pages does a site need before crawl budget matters?
Google’s guide points to sites with 1 million or more unique pages that change about weekly, or 10,000 or more that change daily, plus sites with many URLs marked Discovered, currently not indexed. Google calls those rough estimates. Its 2017 post said sites with fewer than a few thousand URLs are crawled efficiently most of the time.
Does noindex save crawl budget?
Not directly. Google has to fetch a page to see its noindex rule, so the request still counts. Google’s crawl budget guide says to use robots.txt, not noindex, for URLs you never want crawled. Its Myths page adds that once noindexed URLs drop out of the index, Google can focus on other URLs, so noindex may free some budget over time. Use noindex for index control and robots.txt for crawl control.
Does Google support the Crawl-delay rule?
No. Google’s Myths page states that the non standard Crawl-delay rule in robots.txt is not processed by Google’s crawlers. To slow Googlebot quickly, Google says to return 500, 503 or 429 status codes for no more than one to two days, or to file a special request. Anthropic documents support for Crawl-delay for ClaudeBot, and Bing Webmaster Tools offers an hourly Crawl Control setting instead.
Do subdomains share one crawl budget?
No. Google’s crawl budget guide defines a site as a unique hostname, so www.example.com and shop.example.com each get their own budget. The Crawl Stats report for a Domain property can show the parent and up to 20 child hosts that received traffic in the past 90 days. Moving a section to a subdomain does not create extra capacity for the main site’s pages.
Can you ask Google for a higher crawl budget?
No. Google’s Crawl Stats documentation says you cannot tell Google to increase your crawl rate, and its special request form only handles reductions. Google’s guide lists two levers that can raise budget: more server resources when URL Inspection reports Hostload exceeded, and higher content value through popularity, uniqueness and user value.
The short answer
Crawl budget is a real limit for a small group of sites: those with hundreds of thousands to millions of URLs, daily churn, or a pile of Discovered, currently not indexed pages. For everyone else in 2026, Google’s own documentation says a current sitemap and a glance at Page Indexing are enough. If you do qualify, the fix is less crawling of the wrong URLs, not more crawling overall: fewer duplicates, honest 404s, single hop redirects, a fast server and a robots.txt file that always loads. AI crawlers add server load, not Google crawl budget, and the bots worth keeping are the ones that put your pages in answers.
Want to know which crawlers reach your key pages and which AI answers name you? Ask for a free crawl and citation check.
Tagged