---
title: "Manage crawl budget on large sites"
description: "Crawl budget matters most for large or fast-changing sites. Segment server logs by Googlebot, measure requests per template, and remove the causes of waste: parameter URLs, soft 404s and duplicate paths. Improving response speed and internal linking also raises how much Google can crawl."
url: "https://seo-mcp.io/t/manage-crawl-budget-on-large-sites"
---

# Manage crawl budget on large sites

Crawl budget matters most for large or fast-changing sites. Segment server logs by Googlebot, measure requests per template, and remove the causes of waste: parameter URLs, soft 404s and duplicate paths. Improving response speed and internal linking also raises how much Google can crawl.

## Prerequisites

- Server log access
- Search Console Crawl stats

## Steps

1. **Segment logs by template** *(general best practice)* — Parse Googlebot requests and group them by path pattern and status. Compute the share of requests landing on thin, duplicate or error pages.
   Check: The report shows request counts per template and the percentage spent on non-indexable URLs.
2. **Fix the top waste source** — Address the template producing the most wasted requests: remove broken links, return correct status codes, or canonicalize duplicate paths. Change code, not only robots.txt.
   Check: The top waste pattern's request count falls in the next monthly log sample.
3. **Improve capacity signals** — Reduce server response time, remove redirect chains from internal links, and link important new pages from high-authority templates.
   Check: Median Googlebot response time is under 500 ms in the log sample.

## Source

@seo-mcp on 2026-10-19: https://developers.google.com/search/docs/crawling-indexing/large-site-managing-crawl-budget

Canonical: https://seo-mcp.io/t/manage-crawl-budget-on-large-sites