How Google search works comes down to three stages: Google finds your pages (crawling), stores what it understood about them (indexing), and picks which stored pages to show for each query (serving). A page can fail at any one of the three, and each failure needs a different fix, as our SEO in South Africa pillar explains.
This guide is written for owners who want to know which stage is costing them visibility before they spend on the wrong one.
Google held a 92.89% search engine market share in South Africa in October 2026, with Bing at 6.28%, according to Statcounter. That makes Google’s pipeline the one most SA owners need to understand first. This post sticks to the mechanics; for the channel view, see how SEO works in South Africa.
Quick Answer
How Google search works: automated crawlers discover pages by following links, Google analyses and stores eligible pages in its index, and ranking systems then choose which indexed pages to show for a query. Google says it does not accept payment to crawl a site more often or rank it higher, and it says a processed page may still not be indexed. Diagnose a visibility problem by stage: not found, not stored, or stored but not chosen.
Pages published but not showing up?
Send us the URL and we will tell you which of the three stages looks like the problem.
Book a Free ConsultationHow Google Search Works: Three Stages, Three Different Failure Points
The pipeline behind a Google result has three stages, and Google’s own documentation names them: crawling, indexing and serving search results. Google search crawling and indexing are the first two stages, and they run in order, so a problem at stage one hides everything after it.
| Stage | What Google does | What you control | Typical owner failure |
|---|---|---|---|
| Crawling | Crawlers download text, images and video from pages they found | Links to the page, robots.txt, server availability | Page is orphaned or blocked from crawling |
| Indexing | Analyses the content and stores it in the index | Content quality, noindex rules, canonical signals, rendering | Page is crawled but not stored |
| Serving | Ranking systems pick indexed pages for a query | Relevance to the query, quality, usability | Page is indexed but outranked or mismatched to intent |
The table follows Google’s own description of how Search works. The useful point for an owner is the third column: you influence each stage through different levers, so one generic “do more SEO” fix can target the wrong stage.
Crawling: the stage where a page gets found
Crawling is the discovery stage, where automated programs called crawlers download text, images and video from pages Google has found on the internet. Google states that it “primarily finds pages through links from other pages it already crawled,” so a page nothing links to is hard to discover.
A sitemap is a second route in. Google describes submitting one as optional, a file listing the URLs you care about. It helps discovery but is not required. Google also says that you usually only need to publish your site on the web, as most sites in its results are found and added automatically.
Two things sit outside your control. First, Googlebot uses an algorithmic process to decide which sites to crawl, how often, and how many pages to fetch from each.
Second, Google does not accept payment to crawl a site more frequently or rank it higher. A new Johannesburg service business cannot buy a faster crawl; it can make pages easy to reach through internal links and keep the server responsive. Our guide to how to improve crawlability covers those levers, and finding orphan pages covers the most common discovery gap.
Takeaway: Crawling
How Google search works begins with crawling, which is how Google discovers a page, mainly by following links from pages it has already crawled. A sitemap is optional help, not a requirement, and no payment buys a faster crawl. If a page has no internal links pointing to it, fix that before anything else.
Indexing: what Google stores and what it leaves out
Indexing is the stage where Google analyses a crawled page’s text, images and video and stores the result in its index. Google states plainly that “not every page that Google processes will be indexed.”
During the crawl, Google also renders the page and runs the JavaScript it finds using a recent version of Chrome. Google’s JavaScript documentation adds that a page can wait in the rendering queue for a few seconds or longer. If your site builds its content with JavaScript, that gap is worth knowing about, and our guide to SEO for JavaScript websites goes deeper.
Duplicates and canonical pages
How Google indexes a website includes grouping similar pages and selecting the one it judges most representative. That is why Google’s Page indexing report says you should not expect all URLs on your site to be indexed, only the canonical pages. A shop with colour and size variants, or a site reachable on several URL forms, will legitimately show pages as not indexed. See how canonical tags work for the signals you can send.
Robots.txt does not keep a page out of Google
Robots.txt controls crawling, not indexing. Google’s documentation says robots.txt is not a mechanism for keeping a web page out of Google, and that you should use noindex or password protection instead. A page blocked in robots.txt can still show up in results without a description if other sites link to it.
Do not combine the two on one page: Google says that if a page is disallowed from crawling in robots.txt, any indexing rules on it will not be found and will be ignored. A noindex only works on a page Google is allowed to crawl. The trade-offs are laid out in noindex vs robots.txt for SA businesses.
Reading two Search Console statuses. “Discovered – currently not indexed” means Google found the page but has not crawled it yet. “Crawled – currently not indexed” means Google crawled it and did not index it; per Google, it may or may not be indexed later, and you need not resubmit it. The first points to a discovery or crawl-scheduling issue; the second points to how Google assessed the page.
Takeaway: Indexing
Indexing is Google storing a page it has analysed, and Google says not every processed page will make it in. Duplicates are grouped under a canonical, so some non-indexed URLs are normal. Use noindex on a crawlable page to keep it out of results, because robots.txt only controls crawling, and Google will not read a noindex tag on a page that robots.txt blocks.
Not sure which URLs should be indexed?
Send us your Search Console page indexing export and we will talk you through which exclusions look normal and which need action.
Talk to Us About Your ExportServing: how Google search ranking works
Serving is the stage where Google returns indexed pages it judges relevant to a user’s query.
How Google search ranking works is a set of systems, not a single formula. Google’s ranking systems guide says its automated systems look at many factors and signals about hundreds of billions of web pages.
Google says relevance depends on hundreds of factors, including the searcher’s location, language and device. That is a plausible reason a “plumber” query in Durban and the same query in Pretoria can return different results. For the local version of this, see how local SEO works in South Africa.
Three points from Google’s ranking systems guide shape what an owner should expect of Google’s ranking systems:
- Page-level first. The systems are designed to work at page level, though site-wide signals and classifiers also contribute to how Google understands pages.
- Helpful content is core. The helpful content system became part of Google’s core ranking systems in 2024, so it is not a separate switch you can check.
- Spam systems run alongside. Google’s ranking systems guide says it employs a range of spam detection systems, including SpamBrain, to deal with content and behaviours that violate its spam policies.
Google does not publish a weighted list of ranking factors, and anyone quoting exact weights is offering a model, not documentation. Rankings also move when Google changes its systems; our guide to Google updates covers how to respond. Matching the query’s purpose matters here too, which is the subject of how search intent affects SEO.
Takeaway: Serving and ranking
Ranking is a set of automated systems that look at many factors and signals, with relevance depending on the searcher’s location, language and device. Google says it does not accept payment to rank a site higher, and Google search ranking factors are not published as a weighted list. Treat any exact “ranking factor percentage” as someone’s model.
Which stage is failing? A symptom-to-stage table
A visibility symptom usually maps to several possible stages, so the right first move is to run the check that separates them. This table lists hypotheses and the check that tells them apart; it is a starting map, not a verdict.
| Symptom | Possible stages and causes | Check that tells them apart |
|---|---|---|
| New page not in results | Crawling: no internal links, blocked in robots.txt. Indexing: not yet processed, or marked noindex. | Page indexing report status (“Discovered” vs “Crawled”); check for internal links to the URL; read robots.txt and the page’s robots meta tag. |
| Page marked “Crawled – currently not indexed” | Indexing: Google crawled the page and did not store it. Common hypotheses, which Google does not state for any given URL: thin, duplicate or low-value content. | Compare against pages that are indexed; check for a near-duplicate canonical; review whether the page adds something the site lacks. |
| Page indexed but traffic fell | Serving: lost position, intent shift, a system change. Also seasonality or a change in query mix. | Compare query-level clicks and average position over aligned periods; check the same period a year earlier for seasonality. |
| Page indexed, ranks, few clicks | Serving: result is outranked, or the title and snippet do not match the query. | Review the queries the page appears for against its title; look at the live results page for that query. |
| JavaScript-built content missing from results | Indexing: content depends on rendering; or crawling blocked a needed resource. | Compare the raw HTML to the rendered page; confirm needed resources are not blocked. |
For the step-by-step version of these checks, our Search Console guide and the DIY SEO audit walk through the reports. The website indexation guide covers what to change once you know indexing is the weak stage.
What a South African owner should do with this
The practical use of this model is to check the stages in order and stop at the first one that fails. Start with crawling, since nothing downstream works without it.
- Confirm the page is reachable. It should return a normal page, have at least one internal link pointing at it, and not be blocked in robots.txt.
- Confirm the page is eligible to be indexed. Check there is no noindex rule (and that the page is not also blocked in robots.txt, which would hide that rule) and that the canonical points to itself or to the page you intend.
- Confirm the page deserves to be chosen. Ask whether it answers the query better or differently than the pages already in the results.
Steps one and two are mechanical and checkable, so fix them first. Step three is a judgement about your page against the competing results, and no check in Search Console answers it for you.
Why South African Businesses Choose Growth Pulse Media
Growth Pulse Media’s SEO work starts by checking which of the three stages is limiting a South African site, then fixing that stage first. Our SEO service for South African businesses starts where this explanation of how Google search works ends. It is run by Dirk van Greuning, who built and scaled a large South African ecommerce business before founding the agency, and all work is executed in-house with a limited client load for senior attention.
Who This Is NOT For
This explanation is not the right fit for three kinds of reader, because Google’s own documentation rules out what they are looking for.
You want a promised position on page one. Google says it does not accept payment to rank a site higher, and that not every processed page will be indexed. Nobody outside Google can honestly promise you a position.
You want the exact ranking formula. Google describes many systems and does not publish factor weights. This post explains the pipeline, not a scoring sheet.
You expect every page to be indexed. Google’s own guidance says only canonical pages should be expected in the index. Non-canonical duplicate URLs being excluded is normal, so a target of indexing every URL is the wrong target.
Ready to find the weak stage?
Send us your site and we will return a plain-language audit of where it stalls.
Get Your Free AuditFrequently Asked Questions
How does Google search work in simple terms?
Google search works in three steps: crawlers find pages by following links, Google analyses and stores eligible pages in its index, and ranking systems choose which indexed pages to show for a query. A page can stall at any step.
How does Google find a new website?
Google primarily finds pages through links from pages it has already crawled. According to Google’s SEO starter guide, you can also submit a sitemap, though that is not required, and the vast majority of sites in its results are found and added automatically as it crawls the web.
Why is my page crawled but not indexed?
Search Console’s “Crawled – currently not indexed” status means Google crawled the page and did not index it, and per Google it may or may not be indexed later. Google does not give a cause for each URL; common hypotheses are duplicate or thin content, so compare the page with similar indexed pages before changing anything.
Can I pay Google to rank higher in search results?
No. Google says it does not accept payment to crawl a site more frequently or rank it higher.
Does robots.txt stop a page appearing in Google?
Not reliably. Robots.txt controls crawling, and Google says it is not a mechanism for keeping a page out of Google. A blocked page can still appear without a description if other sites link to it. Use noindex on a page Google can crawl, or password protection, because a noindex on a robots.txt-blocked page will not be read.
Want Expert Help Finding Where Your Site Stalls?
Growth Pulse Media works with Search Console and GA4 data to show South African businesses which stage is limiting visibility. No obligation — we’ll get back to you within 24 hours.
Get Your Free Audit

