noindex vs robots txt comes down to one distinction: noindex tells Google not to show a page it has fetched in search results, while robots.txt only governs which URLs crawlers may request. This post sits under our SEO in South Africa guide, and it settles which rule belongs on which page of a South African business site.
The two rules are often treated as interchangeable, and that is how a staging site, a thank-you page or a price-list PDF ends up in Google anyway. Google's documentation says plainly that a Disallow line is not a way to keep a page out of search. It also says the page must stay crawlable for a noindex rule to work. Every decision below follows from those two statements.
Quick Answer
Use noindex when a page must stay out of Google's results, and make sure that page is not blocked in robots.txt, because Google has to fetch the page to read the rule. Use robots.txt when you want to stop crawlers requesting sections of a site, such as filtered or internal search URLs. In the noindex vs robots txt choice, the one combination to avoid is Disallow plus noindex on the same URL: the Disallow hides the noindex from Google.
Jump to
Not sure which pages Google can see?
Send us your site address and ask for a visibility check of which pages Google can and cannot reach.
Request a visibility checkNoindex vs Robots Txt: What Each Rule Actually Controls
Noindex is an indexing rule and Disallow is a crawl rule, so one decides whether a fetched page may appear in results and the other decides whether Google requests the URL at all. Google describes a robots.txt file as telling crawlers "which URLs the crawler can access on your site", used mainly to avoid overloading a site with requests.
Noindex works one step later. Google's robots meta tag documentation defines it as "Do not show this page, media, or resource in search results." You set it with a noindex meta tag, which is a robots meta tag in the page's HTML, or with an X-Robots-Tag HTTP response header, which is the route for files such as PDFs that have no HTML to put a tag in.
| Question | Disallow (crawl rule) | Noindex (indexing rule) |
|---|---|---|
| What it controls | Whether a crawler requests the URL | Whether the page may appear in search results |
| Where it lives | One file at the root of each host | On each page (meta tag) or in its HTTP headers |
| Can the page still show in Google? | Yes, as a bare URL, if other sites link to it | No, once Google has fetched the page and read the rule |
| Does Google need to fetch the page? | No, that is the point | Yes, or it never sees the rule |
| Best for | Saving crawl effort on low-value URL patterns | Keeping a specific page out of search |
Two scope limits matter in practice. A robots.txt file "must be located at the root of the site host to which it applies" and applies only to the protocol, host and port where it is posted, so a rule on your main domain does nothing for a staging subdomain.
The second limit is the vocabulary. Google lists four supported fields for a robots.txt file: user-agent, allow, disallow and sitemap. A "Noindex:" line is not on that list, so it cannot stand in for the tag or header.
Why Using Both Rules on One URL Backfires
A page that carries a noindex tag but also sits under a Disallow line keeps that tag unread, because Google cannot fetch the page to see it. Google's block-indexing documentation states the requirement directly: "For the noindex rule to be effective, the page or resource must not be blocked by a robots.txt file, and it has to be otherwise accessible to the crawler."
The Google Search Central page on blocking indexing spells out the consequence: if the page is blocked, the noindex rule goes unseen and "the page can still appear in search results, for example if other pages link to it." Google's robots.txt introduction adds that for a blocked page the URL, and potentially anchor text from links to it, can still appear in results.
Why a Blocked Page Can Still Be Listed
Disallowing a URL stops Google reading the page, not knowing the URL exists. If other pages link to it, Google can list the address without a description, and a noindex tag on the blocked page cannot help because Google never fetches the page to read it. To remove such a page from results, lift the Disallow and let Google see the noindex.
Search Console reports this state as "Indexed, though blocked by robots.txt", which Google's help page describes as "The page was indexed despite being blocked by your website's robots.txt file." Google's recommended fix there is to lift the Disallow line and add a noindex tag instead. Our guide to how website indexing works covers the wider picture of what Google keeps and what it drops.
Which Page Gets Which Rule on a South African Site
Choosing noindex or robots txt for a page depends on one question: must this page stay out of search results, or do you only want crawlers to spend less time on it? If it must stay out, use noindex and keep the page crawlable. If you only want crawl effort saved, use robots.txt, and accept that the URL can still be listed.
The table below applies that question to pages common on South African business sites. It is a decision aid, not a measured result: each row follows from the Google documentation above.
| Page type | Goal | Use | Best for / watch-out |
|---|---|---|---|
| Thank-you page after a PayFast, Yoco or Ozow payment, or after an enquiry form | Keep it out of search results | noindex, page left crawlable | Aims to keep a confirmation page out of Google results (our judgement); do not also Disallow it |
| Internal site search results | Stop crawlers wandering through endless query URLs | Disallow rule on the search path | Saves crawl effort; if some of those URLs are already listed, switch to a noindex tag |
| Filtered or sorted category URLs on a store | Limit near-duplicate crawling | Disallow rules; canonical tags are another option where the filtered page is still useful (our judgement, not a Google rule) | See our guides on faceted navigation and duplicate content |
| Staging or test copy of the site | Stay unfindable by search and by visitors | Password protection first; noindex only as a backstop | A Disallow rule alone leaves the URLs listable; a login stops crawlers and visitors. Behind a login Google cannot fetch the page, so the noindex is only read if the login fails or is removed. Remove the noindex when the site goes live: a leftover one is a classic rebuild failure (our inference) |
| Price-list or brochure PDF you do not want in results | Keep it out of search results | X-Robots-Tag: noindex header | PDFs cannot carry a meta tag, so the header is the route |
| Thin or expired pages you want out of results but still reachable by link | Remove from results, keep for visitors | noindex | If the page has no purpose, see content pruning instead |
| Pages holding personal or client data | Restrict access | Authentication, not either rule | Neither rule restricts who can open the URL |
The noindex vs robots txt pattern behind the table is simple. Noindex vs disallow is really a choice between "hide from results" and "do not fetch", and only the first of those is something you can rely on for keeping a page out of search. Disallow earns its place on large URL patterns where fetching is the cost, which is where crawl budget matters.
The conflict check in three steps
1. List every Disallow line in the site's rules file. 2. Open a few URLs that match each line and look for a noindex meta tag or X-Robots-Tag header. 3. Where you find both, decide the outcome you want: for "out of search results", delete the Disallow and keep the noindex; for "save crawling", delete the noindex, since it is doing nothing.
Want a second pair of eyes on your crawl rules?
Send us the file and the three page types you most want kept out of search, and ask for a visibility check of your rules.
Send us your robots.txtSetting Each Rule Up Without Breaking the Site
Setting up either rule is a small code or settings change, and the risk is in the scope: a wrong Disallow pattern or a site-wide noindex tag can remove the pages that earn you enquiries. Make the change narrow, then check it in Search Console before moving on.
On Shopify
Shopify stores ship with a preconfigured robots.txt, and Shopify's help centre says you do not need to change it "unless you have a specific reason to do so". Its default file disallows admin pages, cart links, checkout pages and filtered or sorted collections.
For hiding a single page, one route Shopify documents is a robots meta tag with noindex placed in the theme.liquid layout file, wrapped in a condition on the page handle, so it fires only on the page you name. Its help page also lists the seo.hidden metafield and the unlisted product status as alternatives for products.
Two Shopify cautions are worth knowing. Shopify says a page hidden this way "is still visible to your storefront search", so noindex does not remove a product from on-site search. And on editing robots.txt.liquid, Shopify states that this "is an unsupported customization" and warns that "Incorrect use of the feature can result in loss of all traffic." Our Shopify SEO guide covers the rest of the store-level setup.
On other platforms
Whatever the platform, view the live page source or response headers after each change to confirm the tag or header is really being served.
Verify in Search Console
Search Console is the check that tells you what Google actually did. Its page indexing report lists a status named "Indexed, though blocked by robots.txt" for the conflict case.
A separate status, "URL marked 'noindex'", covers pages where Google "encountered a 'noindex' directive and therefore did not index it".
Use the URL Inspection tool on a sample page after each change. Our walkthrough of using Google Search Console covers the reports, and why pages sit in "crawled, currently not indexed" covers a different state that is often mistaken for a noindex problem.
Why Neither Rule Is a Security Control
Neither rule is a lock, so neither protects a page from anyone who has the URL. For robots.txt, the Robots Exclusion Protocol standard, RFC 9309, says so directly: "These rules are not a form of access authorization." A robots.txt file is also public, so a Disallow line announces the path it is trying to hide.
Noindex is a separate case, because RFC 9309 does not define it; it is a directive Google documents. Compliant search engines honour it, but anyone with the URL can still open the page.
If a page holds personal information, such as client lists, invoices or application forms, the relevant duty is POPIA section 19(1), under which a responsible party must secure the integrity and confidentiality of personal information in its possession by taking "appropriate, reasonable technical and organisational measures" to prevent, among other things, unlawful access. The Act does not name a login, so that step is our inference: a login or access control is the kind of measure that restricts who can open a page, and noindex and Disallow are not.
Neither Rule Locks a Page
Both rules rely on a crawler choosing to respect them, and a Disallow line is publicly readable. Anything confidential belongs behind a login, with these two rules used only to manage what search engines list and fetch.
The same logic applies to AI crawlers. Whether you let them fetch your content is a robots.txt decision, covered in our guide to controlling AI crawlers, and it is a request that compliant crawlers honour, not an enforcement mechanism.
Why South African Businesses Choose Growth Pulse Media
A noindex/robots.txt mix-up is a visibility problem that a technical SEO audit can find. Growth Pulse Media is a South African agency founded by Dirk van Greuning, who writes about SEO from an operator's perspective. Our SEO service starts with what Google can and cannot see on your site, using Search Console, before any content or link work is planned.
A noindex vs robots txt mix-up is cheap to fix and expensive to leave, so a visibility check is a sensible first step on a Shopify store or a site that has recently been rebuilt.
Who This Is Not For
A robots txt vs noindex explainer is the wrong resource for three kinds of reader, even though it suits owners and marketers choosing between the two rules.
Rebuilt your site and traffic dropped?
A leftover noindex tag or Disallow rule is a common suspect after a rebuild. Ask for a visibility check of your site.
Request a visibility checkFrequently Asked Questions
The short answer: noindex keeps a crawlable page out of results, and Disallow only stops crawling. The answers below cover the common follow-ups.
What is the difference between noindex and a Disallow rule?
Noindex controls whether a fetched page may appear in search results, while a Disallow rule controls whether a crawler may request a URL. A blocked page can still be listed as a bare URL if other sites link to it. A page marked noindex can be crawled but is kept out of results once Google reads the rule.
Can I use noindex and a Disallow rule together?
Not on the same URL. The Disallow stops Google reading the noindex, because Google must fetch a page to read the rule, so the noindex is never seen and the page can still appear in results. Choose one rule per URL: noindex to keep it out of results, Disallow to save crawling.
Does a Disallow rule stop a page appearing in Google?
No. Google's documentation says robots.txt "is not a mechanism for keeping a web page out of Google". To keep a page out, use password protection, or leave the page crawlable and add a noindex rule.
When to use noindex instead of a Disallow rule?
Use noindex whenever the goal is keeping a specific page out of search results, such as thank-you pages, internal-only pages or PDFs you do not want listed. Use robots.txt for crawl control on large patterns of low-value URLs, accepting that the URLs can still be listed.
Does noindex remove a page from my own site search?
Not necessarily. Shopify's help centre confirms that a product or page hidden with a robots meta tag is still visible to the store's own search, so remove it from the storefront separately if visitors should not find it.
Get your indexing rules checked properly
Send us your site and ask for a review of your Disallow rules, your noindex tags and what Search Console reports for them.
Contact Growth Pulse Media

