Skip to the content
searchboar.

Crawling and indexing

X-Robots-Tag: Meaning, examples and limitations

X-Robots-Tag is an HTTP response header for supported indexing and search-preview directives. It can apply rules to HTML pages and to files such as PDFs, where an HTML robots meta tag is not available.

At a glance

  • The directive travels in the response headers, separately from the document body.
  • It is useful for non-HTML resources such as PDFs.
  • It does not prevent unauthorized access to a confidential file.
On this page
A document envelope carries an instruction before the file is opened.
X-Robots-Tag is an HTTP response header for supported indexing and search-preview directives.

Why use a header?

A PDF download has no HTML head in which to insert a robots meta tag. A server can instead send X-Robots-Tag: noindex with the file response. The same mechanism can also be used for HTML. Supported directives and crawler-specific forms are described in the primary documentation.

This makes the delivery layer part of the indexing decision. An application, web server or CDN rule may add the header even when the page template contains no exclusion.

Example: a public downloadable checklist

Imagine a business offering a checklist through a public landing page. It wants the explanatory page discoverable but does not want the PDF itself listed as a separate search result. A file-specific header can express that distinction without placing noindex on the landing page.

The PDF remains publicly downloadable. If it contains confidential information, authentication and authorization are required. Noindex is not a password and does not stop a visitor sharing the URL.

How do overlapping rules behave?

Inspect every delivered rule. A permissive HTML tag does not guarantee that a restrictive response header is ignored. A broad deployment header can therefore unintentionally exclude many public pages. Conversely, removing a file-specific rule globally may change unrelated downloads.

Scope rules to the resources that need them, and verify the actual response after any delivery or cache change. Cached copies can preserve earlier headers until they are refreshed.

What is the crawling limitation?

The crawler must retrieve the response to observe its header. If robots.txt blocks retrieval, the header cannot be relied on as an observed removal instruction. Compare crawl access, delivered headers and index status as separate facts. A configuration file expressing the desired rule is not evidence that the public response already contains it.

Sources

  1. Google: X-Robots-Tag