Crawl budget
The number of pages Googlebot will crawl on your site in a given time period.
What is Crawl budget?
Crawl budget is the combination of crawl rate (how many simultaneous connections Googlebot makes) and crawl demand (how much Google wants to crawl your site). It determines how many pages Google will crawl during each visit.
For most small to medium sites, crawl budget isn't a concern. It becomes important for large sites (millions of pages), sites with dynamically generated URLs, or sites with significant duplicate content.
Why Crawl budget matters
Limited crawl budget means Google may not discover all your important pages. Wasting crawl budget on low-value pages can delay indexing of important content. Large sites must optimize which pages get crawled.
How to implement
Block low-value pages in robots.txt
Use canonical tags to consolidate duplicates
Fix or remove broken pages (404s, 500s)
Improve server response time
Update sitemaps with important pages
Minimize URL parameters
Common mistakes to avoid
- Worrying about crawl budget on small sites
- Blocking important pages from crawling
- Generating infinite URLs through filters
- Slow server responses wasting crawl capacity
Examples
Ecommerce site with 1M products needs crawl optimization
Blocking /admin/, /cart/ from crawling
Using parameter handling in Search Console
Frequently asked questions
Only if your site has millions of pages, generates many dynamic URLs, or you notice indexing delays. Most sites under 10,000 pages don't need to optimize crawl budget.
Google Search Console's Crawl Stats report shows pages crawled per day, time spent downloading, and crawl requests over time.