Crawl Budget Optimization

Do you actually have a crawl budget problem? Diagnosis before optimization

You have a crawl budget problem when Google knows about important URLs but doesn't fetch them, or fetches them late, while spending requests on URLs with no value. If new pages reach the index within a day or two, crawl budget work won't move anything, and your time is better spent on content or internal linking.

Google's guide for large sites (updated in July 2026) gives rough thresholds: more than a million unique pages changing about once a week, more than 10,000 pages changing daily, or a large number of URLs in the "Discovered – currently not indexed" status. That last condition matters most, because it also hits smaller sites, such as stores with extensive filters.

Two statuses that are easy to confuse

The Page Indexing report in Search Console separates two situations that need completely different fixes. Search Console Help describes "Discovered – currently not indexed" as a page Google found but hasn't crawled yet, typically because crawling it was expected to overload the site, so the crawl was rescheduled. "Crawled – currently not indexed" means the page was fetched but not added to the index.

The Crawl Stats report in Search Console: what I read from it

The Crawl Stats report shows how many requests Google sent to your site, with what result and for what purpose. It's the fastest data source when there are no logs, but it only exists for root-level properties. A property added for a subfolder such as /blog/ doesn't have it.

According to Search Console Help, the report breaks requests down by response, by file type (HTML, image, JavaScript, CSS, JSON and others), by purpose (discovery of new URLs or refresh of known ones) and by Googlebot type (smartphone, desktop, image, video, page resource load, AdsBot, StoreBot and others). A separate Host status section shows problems fetching robots.txt, resolving DNS and connecting to the server over the past 90 days.

Patterns I look for

• A low share of Discovery while you add lots of new pages. If you publish hundreds of products a day and almost every request is a refresh of old URLs, Google isn't keeping up with discovery.
• A high share of redirects and 404s. Every one of those is a fetch with no value. They usually come from stale internal links or from the sitemap.
• A large share of JSON or "Other file type". This often means Googlebot is fetching the API endpoints your JavaScript app calls during rendering.
• Page resource load as a big slice of requests. Pages need many JS and CSS files, or the file URLs change with every deploy.
• Average response time rising along with request volume. The server is becoming the bottleneck; once it's over its limit, Google will slow down on its own.
• robots.txt problems in Host status. Search Console Help explains that a failed robots.txt fetch can make Google slow or stop crawling, so this file has to respond reliably.

The report shows example URLs, not a full list, and it can't split requests by page template. So I treat it as a starting point and base decisions on the logs.

Server log analysis, step by step

Server logs are the only source that shows every Googlebot request to every URL. They tell you how much crawling goes to the pages that sell and how much to filters, parameters and errors. Crawler tools simulate a bot; logs show what the bot really does.

1. Getting the logs. I pull access logs from the server (Nginx, Apache), the CDN (e.g. Cloudflare, Fastly) or the hosting platform. With a CDN I take edge logs, because some requests never reach the origin. At minimum I need URL, status, response time, user agent, IP and timestamp.
2. Verifying Googlebot. User agents are easy to fake. Google documents two verification methods: a reverse DNS lookup (googlebot.com, google.com or googleusercontent.com) confirmed by a forward lookup, or matching the IP against the published range files such as common-crawlers.json. I automate this, because I regularly find bots in logs that only pretend to be Googlebot.
3. Classifying URLs by template. Every URL goes into a group: product page, category, filter, pagination, internal search, article, static asset, API, other. I use rules on URL patterns, and for unusual structures a script that matches URLs against the product and category database.
4. Joining with the list of canonical URLs. I combine the logs with a site crawl and the sitemap. That shows canonical URLs Googlebot didn't visit in the period, and visited URLs that aren't part of any structure (old links, campaign parameters, URLs generated by scripts).
5. Conclusions per template. For each group I calculate its share of requests, status mix, visit frequency and response time. Only then do I decide what to close off, what to speed up and what to strengthen with links.

I match the analysis window to how fast the site changes, usually a few weeks, to smooth out daily swings and cover a full refresh cycle of the key templates. For smaller datasets I use Screaming Frog Log File Analyser; with millions of rows, my own scripts and a database (BigQuery or DuckDB). The same logs are useful for measuring traffic from AI assistants, which I cover in How to Measure Traffic from ChatGPT, Perplexity and AI Mode.

robots.txt, noindex, canonical or 404: which tool for which URL

Each tool does something different: robots.txt stops fetching, noindex removes a page from the index but requires fetching it, canonical consolidates signals, and 404 or 410 ends a URL's life. The most crawl damage I see comes from using noindex to save crawl, or from blocking pages in robots.txt that should actually drop out of the index.

Google's guide also says not to use robots.txt to temporarily shift crawling to other pages. A block makes sense for URLs that should stay closed for good. I explain how robots.txt behaves, including with AI crawlers, in AI Crawlers – Who Scans Your Site and How to Manage It.

Faceted navigation: which filters to index and which to cut off

A filter deserves an indexable URL only when people search for that exact combination and the result has enough products to be useful. Every other filter works for users but shouldn't create URLs a bot can reach.

How I choose filters for indexing

1. I collect queries that combine a category with an attribute (brand, material, use case) from keyword tools and Search Console.
2. For each combination I check how many products the store has. A page with one or two products rarely beats the parent category.
3. I check the results page: if Google ranks filtered category pages for that query, the intent is confirmed.
4. Chosen combinations get a static, readable URL, their own title, heading, short description and a link from the category. The rest stay as filters without indexable URLs.

Technical rules from Google's documentation

In December 2024 Google published a dedicated document on faceted navigation (updated in December 2025). The recommendations I implement:

• If filtered URLs aren't needed in search, I disallow the filter parameters in robots.txt and leave only product pages and unfiltered listings crawlable.
• Filters stored in the URL fragment (after #) have no effect on crawling, because Google generally doesn't support fragments in crawling and indexing. That's a good fit for purely functional filters.
• rel=canonical and rel=nofollow can reduce crawling, but Google calls them less effective long term than robots.txt. Nofollow only works if every link to that URL carries it.
• If filters should be crawled: the standard & parameter separator, a consistent filter order in the URL and no duplicate filters.
• An empty filter combination, a nonsensical combination and a nonexistent pagination page return 404, not a redirect to a generic error page.

Stores have other noise sources too: out-of-stock products, variants on separate URLs and deep pagination. I cover the e-commerce strategy in E-commerce SEO & GEO.

Crawl capacity: server, HTTP caching and page resources

Crawl capacity goes up when the server responds quickly and reliably, and goes down with 5xx errors, 429 responses and growing response times. This is the part of crawl budget I can influence directly in code and infrastructure.

HTTP caching and 304 responses

Google's large-site guide recommends supporting the 304 (Not Modified) status. In its "Crawling December" series from December 2024, Google added that its crawlers support conditional requests with ETag and If-None-Match as well as Last-Modified and If-Modified-Since, and recommended ETag because it's less error-prone. On product pages and articles that rarely change, I implement an ETag computed from the page content, not from the render time, because otherwise every request looks like a change.

Page resources use crawl too

In the same series Google explained that fetching the resources needed to render a page (JavaScript, CSS, API responses) counts against the crawl budget of the host serving them, and that the rendering service caches the JS and CSS it fetches for up to 30 days regardless of cache headers. What that means in practice: fewer files per page, care with cache-busting parameters that change a file's URL without changing its content, and content available in the HTML without extra API calls. I cover rendering in detail in JavaScript SEO.

File size limit

Googlebot's documentation now says that when crawling for Search it fetches the first 2 MB of a supported file type and the first 64 MB of a PDF. Google described the February 2026 change as a documentation clarification rather than a change in behaviour. Most pages are far below the limit, but I check it on category pages with hundreds of products and on pages with a large application-state JSON inlined into the HTML. Content and links beyond 2 MB won't be read.

Slowing crawling during an incident

If Googlebot is overloading the server, Google recommends temporarily returning 500, 503 or 429 (with a Retry-After header) instead of 200, but only for a few hours, a day or two at most. Longer than that and URLs may drop out of the index. Google retired the crawl rate limiter tool in Search Console on 8 January 2024, so the lasting fix is a faster server and caching, not putting the brakes on the bot.

Crawl demand: sitemaps, lastmod and internal linking

Crawl demand depends on how important and how fresh Google thinks your URLs are. A sitemap with a trustworthy lastmod and internal links to key pages are two signals you fully control.

Sitemaps split by template

Google limits a single sitemap to 50,000 URLs or 50 MB uncompressed; larger sets are split into several sitemaps tied together by an index. I split them not just by the limit but by template: products, categories, articles, location pages. Search Console then shows the indexed share for each group, and you can see straight away which template has the problem.

• The sitemap holds only canonical, indexable URLs returning 200. No redirects, 404s, noindex pages or URLs blocked in robots.txt.
• lastmod changes only on a significant change to content, structured data or links. Google uses the value when it's consistently and verifiably accurate, and ignores it when every page carries today's date.
• I leave out priority and changefreq, because Google ignores them.
• The sitemap is generated from the same database as the site, so a discontinued product leaves it at the same moment it starts returning 404 or a redirect.

Click depth and links to new pages

In the site crawl I check how many clicks separate key categories and products from the homepage, and which canonical URLs have no internal link at all. New products get linked from their category (on the first page of the listing, not page 40) and from "new in" blocks; articles from related posts and service pages. Internal linking works on crawl demand and on rankings at the same time, which is why it gives the most for the least risk.

Beyond Google, don't forget Bing, whose index feeds Copilot and ChatGPT Search among others. Bing supports IndexNow, which notifies the search engine about a changed URL the moment you publish. I write about it in Bing, Microsoft Copilot and IndexNow.

AI crawlers and server performance

AI crawlers don't take Googlebot's budget, but they share the same server. If GPTBot, ClaudeBot and other bots generate heavy traffic, the server slows down, and a slower server means lower crawl capacity for Google.

In every log analysis I separate out AI bot traffic and check three things: how many requests they send, to which templates, and how they affect response times at peak hours. I decide with the client which bots to allow, separately for training bots and for AI search bots, since the latter can bring citations and traffic. This is where technical SEO meets GEO, optimization for AI search.

AI bots usually don't execute JavaScript, so content loaded by scripts is empty to them. That's one more argument for server-side rendering, which helps both Google's crawl budget and visibility in AI answers.

How I measure the results of crawl budget optimization

I measure the result by the change in the share of crawling spent on valuable URLs, the time from publication to indexing, and the drop in "Discovered – currently not indexed" URLs. Organic traffic is the goal, but it reacts later and depends on other factors too.

The first changes in Crawl Stats usually show after a few weeks, which is why my process includes 4–8 weeks of monitoring after deployment. I compare against a baseline period of the same length, and for stores I note seasonal peaks, which change Googlebot's behaviour on their own.

Special cases: migrations, programmatic SEO, language versions

Crawl budget usually becomes a problem not because of a site's steady size but because of a sudden change: a migration, thousands of newly generated pages or another language. In those situations I plan for crawling before the release, not after the drop.

• Migrations and URL changes. After a move, Google has to fetch old URLs to see the redirects and new ones to index them. Single-hop redirects, updated internal links and new sitemaps from day one shorten that period. I describe this under SEO migration.
• Programmatic SEO. Thousands of pages generated from a database quickly outrun crawl capacity when most of them look alike. I publish them in batches, starting with combinations that have real demand, and watch the indexed share in each batch's sitemap. More in Programmatic SEO with AI.
• Language versions. Every language multiplies the URL count, and broken hreflang or canonicals can make Google crawl versions it treats as duplicates. I cover the rules in Hreflang and International SEO.
• JavaScript apps. A site rendered only in the browser spends crawl on resources and API calls. When it's rebuilt with server-side rendering, for example as part of a Next.js migration, the problem often disappears along with the old architecture.
• Content cleanup. Thousands of thin, outdated articles also dilute crawl demand. How I decide what to update, merge or remove is in Content Decay.

Crawl budget optimization is one part of broader technical work on a site. You'll find the full context on the technical SEO page, and the structured data I often implement on the same templates under Schema.org implementation.