Choose crawl, index and snippet controls correctly — robots.txt cannot reliably keep a known URL out of search, and a crawler that cannot fetch a page may never see its noindex.
Next step
Use the free audit or a strategy call to map these frameworks to your current website, market, and revenue goals.
Start free auditRelated services
Related sample
Direct answer
Use robots.txt to express crawler access preferences, a robots meta tag for HTML page indexing/snippet directives, and X-Robots-Tag for supported directives on HTTP responses including non-HTML resources. If a crawler cannot fetch a page, it may not see the page-level noindex. Robots.txt is not a reliable way to keep a known URL out of search.
Key takeaways
| Desired outcome | Primary control | Critical caveat |
|---|---|---|
| Reduce compliant crawler access to paths | robots.txt | URL may remain known; not security |
| Remove an accessible HTML page from index consideration | meta robots noindex | Crawler must access the directive |
| Control non-HTML resource indexing | X-Robots-Tag | Verify header at every edge response |
| Limit snippets | documented snippet directives | May affect presentation, not access |
| Protect private content | Authentication/authorization | Robots directives are public preferences |
| Consolidate duplicates | Canonical/redirect decision | Owned separately from robots alone |
Robots.txt is not a reliable way to keep a known URL out of search; robots directives are public preferences, not access control.
Test the exact protocol, host, port and path because robots.txt scope follows them. Check 2xx/3xx/4xx/5xx behavior for the robots file, rule precedence, case, wildcards and subdomains. Google documents special handling when robots.txt is unavailable and limits processing to valid content.
Robots.txt scope follows the exact protocol, host, port and path, with documented special handling when the file is unavailable.
A team blocks /old/ in robots.txt and adds noindex to the same pages. Known URLs persist because Google cannot fetch the directive. The correct fix routes permanent removals to appropriate status/redirect behavior, not more robots rules.
Taskcover's Technical SEO service audits robots.txt, meta robots, and X-Robots-Tag together to remove contradictory directives.
No. Robots.txt expresses a crawl access preference. It is not a reliable way to keep a known URL out of search, and the URL can remain known even when disallowed.
If the crawler cannot fetch the page because robots.txt blocks it, it may never see the page-level noindex directive, so the block and the noindex effectively conflict.
Use X-Robots-Tag when the indexing directive needs to apply to a non-HTML resource, such as a PDF or image, since meta robots only works on HTML pages.
Taskcover can audit robots.txt, meta robots, and X-Robots-Tag rules and build a directive decision tree your team can maintain.
Google Search Central · Accessed: July 13, 2026
Google Search Central · Accessed: July 13, 2026
Google Search Central · Accessed: July 13, 2026
Build canonical-only XML sitemaps by template, keep lastmod values truthful, segment for diagnosability, and monitor processing so exceptions stay accountable.
A prioritized technical SEO audit checklist for crawlability, indexation, architecture, canonicalization, rendering, Core Web Vitals, schema, international setup, and migrations.
Map live, moved, removed, empty and temporarily unavailable URLs to honest HTTP responses — a friendly 200 error page can be read as a soft 404.
Use the free audit or a strategy call to map these frameworks to your current website, market, and revenue goals.
Taskcover uses necessary storage for security and forms. You can choose whether we use preferences, analytics, and marketing measurement.
