Hide Web Pages from Search Bots (Robots.txt Tags)
To keep a page out of search results, let search crawlers fetch it and serve a noindex directive in its HTML or HTTP header. Use robots.txt to guide crawling, not to remove a URL from results or protect private information. First decide your goal, then apply the matching rule and verify the live response.
Want to avoid changing the wrong URLs or waiting for a page to disappear when a crawler cannot see your instruction? Start by separating crawl control from search removal. The right choice depends on whether you want to reduce crawling, keep a page out of search results, or restrict access to confidential content.
Planning: Define Crawl Control Versus Search Removal
Crawl control tells a compliant crawler which URLs it may fetch. Search removal asks a search engine not to show a page in results. These are different outcomes, handled by different signals, so decide which one you need before editing a file.
A robots.txt file follows the Robots Exclusion Protocol, defined in RFC 9309. It can tell crawlers not to fetch parts of a site, but it does not reliably remove those URLs from search results. A URL may still appear if a search engine learns of it from links or other signals.
To request that a crawlable page stay out of search results, use a noindex directive. Google documents this behavior in its noindex guidance. The crawler must be able to fetch the page to read the directive.
If the content is confidential, neither method is enough. Use authentication and authorization controls, such as a sign-in requirement and server-side access checks. A robots rule is public, and it does not stop a person or an unauthorized crawler from requesting a URL.
I often see the same planning mistake in site migrations: teams block a page in robots.txt while expecting its noindex tag to remove it from search. That blocks the fetch needed to read the tag. Treat crawling, indexing, and access as separate controls.
Next step: List the URLs and state the goal for each: reduce crawling, remove from results, or restrict access.
Access and Scope: Match Rules to the Correct Origin and URLs
A robots file governs a specific origin: the combination of scheme, host, and port. Place it at that origin’s root, such as https://example.com/robots.txt. A file on one host or protocol does not automatically govern another.
Make an inventory of relevant versions before deploying rules. For example, http://example.com, https://example.com, and https://www.example.com are distinct origins for robots rules. A separate subdomain, such as shop.example.com, needs its own file if you want to give crawlers instructions for it.
| Need | Appropriate control | Important limit |
|---|---|---|
| Reduce crawling of a URL path | Disallow in robots.txt | Does not reliably remove a URL from results |
| Keep a crawlable HTML page out of results | HTML noindex meta tag | Crawler must fetch the page |
| Keep a PDF or other response out of results | X-Robots-Tag: noindex header | Crawler must fetch the response |
| Restrict confidential content | Authentication and authorization | Robots rules are not access control |
Write down exact URL patterns before choosing a rule. Check trailing slashes, capitalization, query strings, and alternate paths. A rule for /account/ should not be assumed to cover /accounts/ or a separate hostname.
RFC 9309 says parsers must support at least 500 KiB of robots file content. In practice, keep the file much smaller and easy to review. The RFC also specifies that the most specific matching rule, based on the longest matching path, takes precedence. When equally specific Allow and Disallow rules conflict, Allow should win.
Next step: Confirm the origin and URL patterns with live examples, not just the page names in a content system.
Execution: Deploy Robots Rules and Noindex Directives
Use a narrowly scoped robots.txt rule when your goal is crawl management. Use a page-level directive when your goal is search removal. Do not combine a crawl block with a page-level noindex expectation, because the crawler may not be able to read the directive.
A basic robots file can look like this:
User-agent: *
Disallow: /staging/
Allow: /staging/public-guide.html
User-agent: * applies the group to all crawlers that follow the protocol. Disallow names a path crawlers should not fetch, while Allow names a path that may be fetched. The example permits one file within a blocked directory because its path is more specific.
For an HTML page you want excluded from search, keep it crawlable and add this within the page’s <head>:
<meta name="robots" content="noindex">
For a non-HTML response, such as a PDF, send an HTTP header instead:
X-Robots-Tag: noindex
A deployment should follow this order:
- Survey: Record target URLs, their current search status, and the desired outcome. Include every relevant hostname and protocol.
- Choose: Use
robots.txtfor crawl management. Usenoindexfor removal from search results. Use access controls for private content. - Deploy: Publish a UTF-8
robots.txtat the origin root. Addnoindexto the intended HTML pages or response headers, and leave those URLs fetchable. - Review: Check for broader rules that may override or conflict with the intended behavior. Confirm the change on the live site, not only in a staging copy.
A robots.txt file is plain text, not an HTML page. Keep directives on separate lines and group them under the correct User-agent. Avoid copying a rule from another site without checking its origin and path scope.
Next step: Test the deployed response before treating the change as complete.
Verification: Test Fetchability, Directives, and Search Status
Verification checks what a crawler can actually retrieve, rather than what your editor or content system displays. Fetch the robots file and each target URL, inspect the response status and headers, then confirm that the HTML contains the intended directive.
You can use curl from a terminal:
curl -i https://example.com/robots.txt
curl -I https://example.com/private-report.pdf
curl -sL https://example.com/target-page/ | grep -i 'name="robots"'
The first command shows the robots file and its HTTP response. The second checks response headers, including a possible X-Robots-Tag. The third follows redirects and searches returned HTML for a robots meta tag. If your site needs JavaScript to add the tag, verify what the search engine can process; a tag absent from the delivered HTML may require extra testing.
Check that the URL returns a normal, fetchable response and is not blocked by a Disallow rule when you rely on noindex. Also inspect redirects: the directive needs to reach the page or response you intend to exclude. If the page redirects, test the final URL as well.
Use the search engine’s webmaster tools to inspect a URL and review its crawl and indexing status. A robots tester can help check rule matching, but it does not prove that a page has been removed from search. Search engines need to recrawl and process changes; timing can vary, so do not promise a fixed removal interval.
A temporary removal tool, where available, can hide a result sooner, but it is not a substitute for the lasting noindex directive or access control. Confirm that the permanent signal is in place and monitor the URL after recrawling.
Next step: Keep a record of the tested URL, response, rule, and date so you can trace unexpected results.
Deployment Examples: Match the Fix to the Problem
These examples show how to choose a control for common site situations. They are implementation scenarios, not claims about a particular company. The key is to test the response and the crawler’s ability to read the instruction.
A public campaign page should disappear from results. Leave it accessible to crawlers, add noindex in its HTML, and verify the rendered or delivered page. A robots block would prevent the crawler from reading that instruction.
A large filtered section is consuming crawl attention. Identify the exact URL patterns and decide which should not be fetched. Add a narrowly scoped robots rule, then test allowed and disallowed paths. Do not assume that blocking every variation also removes known URLs from search.
A downloadable report should not appear in results. Send X-Robots-Tag: noindex on the file response and confirm the header with curl -I. If the report is confidential, also restrict access; the header alone is not a privacy barrier.
A staging site contains unreleased content. Protect it with authentication. A robots file may guide compliant crawlers, but it does not prevent access, and a URL can become known through other sources. Keep access restrictions in place even if you also publish crawler instructions.
Common Errors and Safer Corrections
A small configuration error can produce the opposite of the intended result. Check the purpose of each directive and the live URL before making broad changes, especially when a rule covers an entire directory.
| Error | Why it fails | Safer correction |
|---|---|---|
Disallow a page expected to be de-indexed by its meta tag | Crawler may not fetch the page or see noindex | Allow crawling and serve noindex |
| Use robots rules for confidential files | Rules do not block access | Require authentication and authorization |
| Put rules on the wrong host or protocol | They do not govern the intended origin | Publish at each relevant origin’s root |
| Block a broad folder to target one page | Other paths may also be affected | Use the narrowest path that meets the goal |
Assume a Disallow guarantees no search listing | A URL may be known without its content being fetched | Use noindex for search removal |
When reviewing a robots file, test both what should be blocked and what must remain available. For example, a broad directory rule can unintentionally stop a crawler from reaching a page that carries the directive you need it to read.
FAQ
These answers distinguish crawling, indexing, and access, which are often mixed together when site owners edit robots rules. Use the answer that matches your goal, then verify the live URL and response rather than relying on the configuration alone.
Does robots.txt remove a page from Google?
No. A Disallow rule can prevent crawling, but it does not reliably prevent a URL from appearing in search results. Use a crawlable noindex directive to request removal.
How do I keep a page out of search results?
Allow crawlers to fetch the page and add <meta name="robots" content="noindex"> to its HTML. For non-HTML files, use an X-Robots-Tag: noindex response header.
Can I use noindex on a page blocked in robots.txt?
You can add the tag, but a crawler blocked from fetching the page may not see it. Allow crawling if you need the crawler to process the noindex directive.
Where does robots.txt go?
Put it at the root of the relevant origin, such as https://example.com/robots.txt. A different scheme, host, or port has a separate scope.
Does a robots file protect private pages?
No. It is not an access control system. Use authentication and server-side authorization to restrict private or confidential content.
Can I use noindex for a PDF?
Yes. Send X-Robots-Tag: noindex in the PDF’s HTTP response, then check the live header. The file must remain fetchable for a crawler to read it.
Which rule wins if Allow and Disallow conflict?
RFC 9309 gives priority to the most specific matching path. If the matching Allow and Disallow rules are equally specific, Allow should win.
How quickly will a noindex change take effect?
There is no fixed time. A search engine must recrawl and process the URL. Check webmaster tools and search results over time, and make sure the page remains fetchable.
Does the robots file cover every subdomain?
No. Each host has its own robots file and scope. Check each hostname and protocol that serves the URLs you want to manage.
How large can robots.txt be?
RFC 9309 requires parsers to support at least 500 KiB. Keep your file organized and test the live version so rules remain clear and scoped correctly.
(This article was written by one of our staff writers, Richard Montclair. Visit our Meet the Team page.)
