Add to Preferred Sources
What is crawl budget? It is the number of pages Googlebot will crawl on your site within a given period — determined by how much server capacity you provide and how much Google actually wants to explore. For most South African websites, the concept sits quietly in the background and causes no problems at all. For ecommerce stores running thousands of product URLs, WooCommerce filter combinations, or sites hosted on servers that go down during load shedding, it becomes the hidden reason important pages never make it into the index.
Understanding your SEO strategy for South Africa starts with knowing which technical constraints are real and which are theoretical — crawl budget is one of the real ones, for the right site profile.
The good news is that diagnosing a crawl budget problem takes just a few minutes in Google Search Console, and the most common fix — blocking low-value URLs from Googlebot — is a robots.txt change, not a site rebuild. This guide explains what crawl budget is, who actually needs to worry about it, what specifically wastes it on South African websites, and the mechanical steps to recover it.
Quick Answer
What is crawl budget? It is Google's operational limit on how many pages it will crawl on your site in a given timeframe, set by two factors: how much your server can handle (crawl capacity limit) and how much Google wants to crawl based on your content's popularity and freshness (crawl demand). Most small and stable websites do not face meaningful crawl budget constraints — if your pages are indexed the same day you publish them, this is not your issue. If you have thousands of product filter URLs, session IDs, or recurring server errors, crawl budget is worth auditing now.
In This Guide
How Google Calculates Crawl Budget
Does Your SA Website Actually Have a Crawl Budget Problem?
What Wastes Crawl Budget on SA Websites
How to Fix Crawl Budget Waste: Practical Checklist
Load Shedding and Crawl Budget: The SA-Specific Risk
Why South African Businesses Choose Growth Pulse Media
Not Sure if Crawl Budget Is Holding Your Site Back?
Send us your domain and we will pull the indexation data and flag exactly which URL patterns are eating Googlebot's time on your site.
Request a Crawl ReviewWhat Is Crawl Budget and How Does Google Measure Yours?
Google's official crawl budget guidance defines the concept as the combination of two independent signals: your crawl capacity limit and your crawl demand.
Crawl capacity limit is the maximum number of connections Googlebot can open to your server without overloading it. Google adjusts this automatically — if your server responds quickly and reliably, the limit rises over time. If your server starts returning errors (5xx codes, HTTP 429 responses) or slows down, Google pulls back to protect your infrastructure. You cannot manually override this ceiling, but you can influence it by making your server faster and more reliable.
Crawl demand is how much Google wants to crawl. Pages that attract links, generate traffic, and update frequently get revisited more often. Static, link-poor, or thin pages get deprioritised. Your sitemap's <lastmod> tags signal freshness and influence demand for individual URLs.
The July 2026 Update You Need to Know
In July 2026, SEO analysts tracking Google's documentation confirmed a significant change: the crawl capacity limit is now shared across all of Google's crawlers — including the bots that feed AI Overviews and Gemini. If AI crawlers are consuming a large slice of your server bandwidth, that directly reduces the capacity available for Googlebot to index your pages. Every URL you let AI bots crawl unnecessarily is a URL Googlebot does not get to see.
The practical implication: crawl budget is not just about search indexation anymore. It is a finite server resource competed for by multiple Google systems simultaneously. Managing it well is now part of a complete technical SEO approach for South African websites.
Does Your South African Website Actually Have a Crawl Budget Problem?
Knowing what is crawl budget is a different question from knowing whether it applies to your site — and for most South African websites, Google's own documentation confirms it does not. If your pages are indexed the same day you publish them, crawl budget is not your constraint. Most small, stable sites never hit their allocation.
The sites that do need to pay attention fall into three broad categories — though Google is explicit that these size thresholds are rough guidelines, not exact cutoffs:
| Site Type | Crawl Budget Priority | Common SA Example |
|---|---|---|
| 1M+ pages, updated weekly | High — essential to manage | Classifieds sites, large aggregators |
| 10,000+ pages, updated daily | High — essential to manage | Large ecommerce catalogues, news sites |
| Under 10,000 pages, stable content | Low — monitor, do not obsess | Most SA service and professional sites |
| Any size with faceted navigation | Medium to High — audit URL volume | WooCommerce stores with product filters |
| Any size with recurring 5xx errors | High — fix server first | Sites on unreliable shared hosting |
The fastest diagnosis: open Google Search Console, go to Pages, and look at the "Discovered — currently not indexed" report. A large and growing list there — particularly for URLs you actually want ranked — is one of the clearest indicators that Googlebot is finding your pages but not getting around to crawling them. That is crawl budget in action.
Key Takeaway
A growing "Discovered — currently not indexed" list in Google Search Console is the most reliable early warning sign that Googlebot is deprioritising your pages. Check this report before running any crawl budget audit — it tells you whether the problem is real before you spend time on a solution.
What Wastes Crawl Budget on South African Websites
The same URL patterns cause problems across most South African websites that have outgrown their default crawl allocation. Recognising them is the first step to removing them.
Faceted Navigation and Product Filter URLs
This is the dominant crawl budget problem for South African ecommerce stores. A typical WooCommerce catalogue with layered filter options — colour, size, price band, brand — does not generate one URL per product. It generates a unique URL for every filter combination applied to every category page. A catalogue with 50,000 products and 20 active filter types can mathematically produce more than 2 million unique URLs, most of which serve no searcher and contain near-identical content.
Googlebot will crawl a portion of those URLs, learning nothing useful from each one, while your actual category pages and product listings wait in the queue. Google's own guidance on this is blunt: most filter URLs should be blocked in robots.txt, not simply noindexed, because noindex still consumes crawl capacity — Googlebot has to fetch the page before it can read the directive. Learn more about this in our guide to improving crawlability for South African sites.
Duplicate Content Variants
HTTP and HTTPS versions of the same page, www and non-www variants, trailing slash differences, and printer-friendly copies all create multiple URLs for identical content. Each one splits your crawl budget. A proper redirect and canonical strategy consolidates these into a single crawlable URL.
Internal Search and Session Parameters
Internal site search (e.g., /search?q=blue+shoes) and session IDs appended to URLs (?sessionid=abc123) generate infinite unique URLs from a handful of real pages. These are almost never worth indexing and should be blocked via robots.txt or stripped with URL parameter handling in Google Search Console. If parameter URLs are eating into your crawl budget, our technical SEO service in SA starts with a free audit and agrees a flat monthly retainer before any fixes begin.
Thin and Low-Quality Pages
Tag archives, date archives, author pages with one post, and automatically generated stub pages are common on WordPress and WooCommerce sites. Google deprioritises these because they signal low overall site quality — and deprioritised crawling means your good pages get fewer visits too.
Orphan Pages and Redirect Chains
Pages with no internal links are difficult to find; pages behind long redirect chains waste crawl budget on every hop. Both are worth auditing. Our guide to finding orphan pages on South African websites explains the diagnostic process in detail.
What this looks like in practice: A South African fashion retailer runs WooCommerce with filters for colour (12 options), size (8 options), fabric (6 options), and price range (5 bands). That combination alone produces up to 2,880 filter URL variants per category page. With 30 categories, the site has potentially 86,000+ filter URLs Googlebot is attempting to process — none of which rank for meaningful queries — while hundreds of product pages sit in the "Discovered — currently not indexed" backlog.
How to Fix Crawl Budget Waste: Practical Checklist
Crawl budget recovery is methodical work, not guesswork. These are the fixes that move the needle, in order of impact for most South African sites.
| Fix | Method | Impact |
|---|---|---|
| Block faceted navigation URLs | robots.txt Disallow for filter parameter patterns | High — immediate reduction in low-value crawls |
| Fix duplicate URL variants | 301 redirects + canonical tags | High — consolidates crawl signals |
| Remove internal search from crawl | robots.txt Disallow /search/ | Medium — closes infinite URL loop |
| Clean your XML sitemap | Remove noindex, redirected, and non-200 URLs from sitemap | Medium — improves signal quality |
| Strengthen internal linking | Add links from high-authority pages to priority targets | Medium — increases crawl demand for key pages |
| Improve server response time | CDN with SA edge nodes, caching, hosting upgrade | High — raises crawl capacity limit over time |
| Return correct status codes | 404/410 for removed pages (not 200 soft 404s) | Medium — stops Googlebot cycling back to dead content |
The internal linking point deserves emphasis: improving your website's indexation rate is not just about sitemaps — it is about building link paths that guide Googlebot from your most crawled pages to the pages you most want indexed. A flat architecture where every important product is reachable within two clicks from the homepage gets crawled faster and more completely than a hierarchy where products are buried five levels deep.
Key Takeaway
Start with robots.txt before anything else. Blocking low-value URL patterns from Googlebot does not harm your indexation — it focuses Googlebot's time on the pages that actually matter. Every URL you remove from Googlebot's queue is a URL it can use for your real content instead.
Running a WooCommerce Store With 1,000+ Products?
Tell us your filter structure and we will map out exactly which URL patterns to block before your next crawl cycle.
Get a Filter AuditLoad Shedding and Crawl Budget: The SA-Specific Risk
South African websites face a crawl budget risk that most international guides never mention: server downtime from load shedding. Though South Africa's grid has seen meaningful stability improvements since 2024, load shedding remains a real possibility — and the SEO consequences of server downtime are worth understanding before the next outage cycle.
When your server goes offline during a Googlebot crawl attempt, the page returns a 5xx error. A single 5xx is logged and noted. Consistent 5xx errors — the kind that can occur when Eskom cuts power to your hosting provider's data centre during a load shedding period — cause Google to automatically reduce your crawl capacity limit to protect what it assumes is an unstable server.
The result: after a sustained period of outages, your crawl rate may be noticeably lower than it was before, and recovering it typically requires weeks of consistent uptime rather than days.
Practical mitigations for SA businesses:
- Hosted platforms (Shopify): No SA-side hosting exposure — Shopify's infrastructure absorbs the risk.
- CDN with SA edge nodes: Cloudflare and similar services cache responses and can serve cached pages even during brief server outages, preventing 5xx errors from reaching Googlebot's logs.
- Dedicated hosting with generator backup: Higher cost but eliminates the crawl capacity erosion problem entirely for high-traffic ecommerce sites.
- Uptime monitoring: Knowing exactly when and how long your site goes down lets you correlate crawl rate changes with outage history and make the business case for infrastructure investment.
Server reliability directly affects your Core Web Vitals performance and your crawl capacity ceiling simultaneously — fixing one improves both.
Why South African Businesses Choose Growth Pulse Media
Technical SEO problems like crawl budget waste are rarely the whole story. They are usually downstream of architecture decisions made when a site was first built — a product catalogue that scaled without URL structure planning, a WooCommerce theme installed with filters enabled by default, a hosting plan that was never upgraded as traffic grew.
Growth Pulse Media works with a deliberately limited number of South African clients so that every engagement gets senior attention, not a junior account manager working from a checklist. Dirk built and scaled a South African ecommerce operation before founding GPM — which means the crawl budget conversation starts with "what does your catalogue structure look like and how does it generate URLs?" rather than a generic audit template.
We audit the full crawl picture: server logs, GSC indexation data, robots.txt configuration, sitemap health, and internal link architecture. For WooCommerce stores on local hosting, we factor in load-shedding exposure directly. For Shopify stores, we look at where URL bloat typically originates on that platform specifically. Every recommendation is actionable and prioritised — our guide to prioritising SEO fixes for South African websites reflects the same logic we apply to client work.
If your site is generating organic traffic but important pages keep sitting in the "Discovered — currently not indexed" queue, that is a solvable problem. The SEO services we offer South African businesses include the full technical layer — not just content and links.
Who Crawl Budget Optimisation Is NOT For
Crawl budget optimisation is a real lever when the problem is real — and a costly distraction when it is not. These are the situations where it should not be on your priority list.
Sites under 500 pages with stable content. If you run a service business website with 20–30 service pages, a blog, and a contact form, Googlebot almost certainly indexes everything you publish on the same day. Spending time on crawl budget optimisation is a distraction from the actual SEO work that would move you — content, links, and local signals.
Businesses whose real problem is content quality. If pages are in the "Crawled — currently not indexed" bucket (as opposed to "Discovered — currently not indexed"), crawl budget is not the constraint. Google has visited those pages and decided not to index them — which means content quality, relevance, or duplicate signals are the issue. Blocking more URLs in robots.txt will not fix that.
Sites that have not addressed on-page fundamentals. Robots.txt optimisation and sitemap housekeeping are meaningless if your priority pages have thin content, no internal links pointing to them, and missing title tags. Fix the fundamentals before optimising the crawl layer — the crawl layer serves the content, not the other way around.
Anyone who has been told crawl budget is their primary SEO problem without evidence. "Discovered — currently not indexed" is a genuine diagnostic. "I think your crawl budget is the issue" without showing you the GSC data is speculation. Ask for the report before any fix is implemented.
Is Crawl Budget Actually Your SEO Bottleneck?
Share your Search Console access and we will tell you within 48 hours whether crawl budget is the constraint — or whether the fix lies elsewhere.
Book a Technical SEO AssessmentFrequently Asked Questions
What is crawl budget in SEO?
Crawl budget is the number of pages Googlebot will crawl on your website within a given timeframe. It is determined by two factors: your crawl capacity limit (how much your server can handle before Googlebot backs off) and crawl demand (how much Google wants to crawl based on your content's quality, freshness, and popularity). Crawl budget matters most for large sites with many pages or sites generating large volumes of low-value URLs through filters and parameters.
Does crawl budget matter for small South African websites?
Generally no. Google's own documentation states that if your pages are crawled the same day you publish them, crawl budget is not your constraint. Most South African service businesses, professional practices, and small ecommerce stores with under a few thousand pages are unlikely to face meaningful crawl budget constraints. Focus your technical SEO attention on site speed, indexation health, and internal linking instead.
How do I know if I have a crawl budget problem?
Open Google Search Console, navigate to the Pages report, and check the "Discovered — currently not indexed" section. A large and growing list of URLs you actually want ranked — particularly product pages or category pages — is the clearest signal. You can also check your server logs to see which URLs Googlebot is visiting most frequently; if it is spending the majority of its time on filter combinations or parameter URLs rather than your priority pages, that confirms a crawl budget waste problem.
How does load shedding affect crawl budget?
When load shedding takes your server offline during a Googlebot crawl attempt, your pages return 5xx server errors. A small number of these is not catastrophic, but consistent 5xx errors cause Google to automatically lower your crawl capacity limit to protect what it interprets as an unstable server. Over a sustained load-shedding period, your crawl rate can decline noticeably and take weeks of consistent uptime to recover. Using a CDN with South African edge nodes, or hosting on a platform like Shopify that is not exposed to local infrastructure risk, eliminates this problem.
Should I noindex or block faceted navigation URLs in robots.txt?
For URLs that generate no unique search value — filter combinations like colour + size + brand that no one searches for as a query — robots.txt blocking is more efficient than noindex. A noindex tag still requires Googlebot to fetch the page before it can read the directive, consuming crawl capacity in the process. Blocking in robots.txt prevents the fetch entirely. For filter combinations that do attract search volume in their own right (a specific brand category, for example), evaluate whether indexing is worth the crawl cost before blocking.
Get a Crawl Budget Audit From a Team That Knows SA Infrastructure
We review your robots.txt, sitemap health, GSC indexation data, and server response patterns — and tell you exactly what is wasting Googlebot's time on your site. Senior attention on every engagement. Named platform expertise across WooCommerce and Shopify. No obligation — we'll get back to you within 24 hours.
Request Your Crawl Audit