𝗖𝗿𝗮𝘄𝗹 𝗕𝘂𝗱𝗴𝗲𝘁 𝗥𝗲𝗰𝗹𝗮𝗺𝗮𝘁𝗶𝗼𝗻
Google decides how many URLs it crawls on your site. This is your crawl budget. It consists of two parts: crawl capacity and crawl demand.
Crawl capacity depends on your server speed. If your server responds fast, Google crawls more.
Crawl demand depends on your content. If your content stays fresh and popular, Google wants to see it more often.
Most small sites do not need to worry about this. If you have fewer than 10,000 pages, focus on a good sitemap instead.
You only need to act if you have:
- Over 1 million unique pages.
- Over 10,000 pages that change daily.
- Many URLs stuck as "Discovered – currently not indexed" in Search Console.
Wasted crawl budget usually comes from structural issues. Common problems include:
- Faceted navigation and URL parameters.
- Infinite filter chains or calendars.
- Soft 404 errors.
- Duplicate or thin content.
- Long redirect chains.
- Slow server response times.
Follow these steps to fix your crawl budget:
- Merge duplicate pages and use canonical tags.
- Use robots.txt to block unimportant URLs. Do not use noindex for this. Google must crawl a page to see a noindex tag, which still wastes budget.
- Return 404 or 410 status codes for removed pages.
- Update your sitemaps with correct lastmod dates.
- Improve your server speed to increase capacity.
Some experts suggest deleting low-quality content to boost traffic. One study showed a 67% lift after deleting old, zero-traffic posts. Be careful. Deleting content improves site quality and focus, but it is not a guaranteed growth trick.
Crawl budget reclamation is about efficiency. It ensures Google finds your best pages quickly. It does not directly change your rankings.
Audit your site before you delete anything. Prune for quality, not just to hit a number.
Source: https://dev.to/mrtd/crawl-budget-reclamation-what-it-is-who-needs-it-and-the-pruning-playbook-139g