Crawl budget is the number of URLs Googlebot can and wants to crawl on your site within a given period. Google defines it as the combination of two things: the crawl capacity limit, which is how much crawling your server can handle without slowing down, and crawl demand, which is how much Google actually wants to crawl based on how popular and fresh your pages are. For most websites, crawl budget is not a problem worth worrying about. Google's own guidance says it mainly matters for very large sites, sites whose content changes very quickly, and sites with a large share of pages stuck as "Discovered, currently not indexed" in Search Console.

That last point is where a lot of smaller sites get confused. A new blog with 200 posts that are not indexed rarely has a crawl budget problem. It usually has a quality, internal linking, or discovery problem that looks like one.

How Google Decides What to Crawl

Crawl capacity is the technical side. Googlebot raises or lowers how many simultaneous connections it opens to your site depending on how quickly and reliably your server responds. If pages load fast and return clean 200 status codes, the limit goes up. If the server slows down or throws 5xx errors, Googlebot backs off to avoid overloading it.

Crawl demand is the interest side. Google crawls URLs more often when they are popular, when they change frequently, and when it has reason to think there is something new to find. A site migration, for example, temporarily increases demand because Google needs to reprocess the new URLs.

Your actual crawl budget is whichever of the two is lower. A fast server does not help if Google sees little reason to crawl, and high demand does not help if your server buckles under load.

Which Sites Actually Need to Care

Google's documentation for large site owners names three groups: sites with over a million unique pages whose content changes around once a week, sites with more than 10,000 pages whose content changes daily, and sites with a large portion of URLs marked as discovered but not indexed. Ecommerce stores with faceted navigation, classifieds, news publishers, and large forums tend to fall into these groups.

If your site has a few hundred or a few thousand pages, Google can almost certainly crawl all of it without trouble. Spending a week optimizing crawl budget on a 300-page blog is time better spent on content quality and links. The foundations covered in XML sitemaps and robots.txt will do more for a small site than any crawl budget tactic.

What Wastes Crawl Budget

On larger sites, the biggest waste comes from URLs that should never have been crawlable in the first place. Faceted navigation is the classic example: filters for size, color, price, and brand can combine into millions of near-identical URLs. Session IDs and tracking parameters in URLs create the same problem.

Other common drains include long redirect chains, soft 404 pages that return a 200 status but show no real content, infinite spaces like calendars with endless "next month" links, and large volumes of thin or duplicate pages. Each of these makes Googlebot spend requests on pages that add nothing to your index.

Fixes That Actually Work

Blocking low-value URL patterns in robots.txt is the most direct fix, because a disallowed URL is not crawled at all. This is the right tool for faceted filters and parameter combinations you never want in search.

A common mistake is relying on noindex to save crawl budget. It does not. Google has to crawl a page to see the noindex tag, so noindexed pages still use crawl requests. Noindex controls indexing, not crawling. Canonical tags have a similar limitation, since Google still needs to crawl a duplicate to read its canonical.

Removing or consolidating thin pages helps on two fronts: fewer URLs to crawl, and a stronger signal of overall site quality. A planned approach to content pruning often improves crawl efficiency as a side effect, especially on older sites with years of low-traffic tag pages, archives, and outdated posts.

Beyond that, keep your XML sitemap clean and limited to indexable, canonical URLs, return proper 404 or 410 codes for removed pages, collapse redirect chains to a single hop, and improve server response times. Faster responses directly raise your crawl capacity limit.

How to Check Your Crawl Activity

The Crawl Stats report in Google Search Console, found under Settings, shows total crawl requests, average response time, and a breakdown by response code, file type, and Googlebot type. A sudden spike in crawling of parameter URLs, or rising response times, is worth investigating.

For larger sites, server log file analysis gives the clearest picture. Logs show exactly which URLs Googlebot requested, how often, and what status codes it received. Comparing crawled URLs against the URLs you actually want indexed quickly reveals where requests are being wasted.

Does JavaScript Affect Crawl Budget?

It can. Pages that rely heavily on client-side rendering need extra resources to process, and every JavaScript and CSS file Googlebot fetches counts toward crawl activity. Large sites built on heavy frameworks may see slower discovery of new content as a result. Getting the rendering setup right is part of good javascript seo, and it has a direct knock-on effect on how efficiently Google can crawl a big site.

The Bottom Line

Crawl budget is real, but it is a large-site problem far more often than a small-site one. If your site is big, fast-changing, or full of parameter URLs, controlling what Googlebot can reach is one of the highest-leverage technical fixes available. If your site is small, focus on quality, internal links, and a clean sitemap, and let Google handle the rest.