Skip to the content
searchboar.

Crawling and indexing

Robots meta tag: Meaning, examples and limitations

A robots meta tag is an HTML element that gives supported crawlers page-level instructions about indexing and search presentation. It can carry directives such as noindex or snippet controls, but it does not authenticate visitors or block access to the page.

At a glance

  • Robots metadata applies to the page whose response contains it.
  • The crawler must be able to retrieve the instruction.
  • A permissive tag does not necessarily override a restrictive header or another tag.
On this page
A web document carries an instruction card in its head section.
A robots meta tag is an HTML element that gives supported crawlers page-level instructions about indexing and search presentation.

What does the tag control?

A tag such as <meta name="robots" content="noindex"> asks supporting search crawlers not to include that page in their index. Other supported values concern link handling or search-result previews. Noindex and nofollow express different instructions; they should not be treated as an inseparable pair.

The supported vocabulary and interpretation depend on the crawler. The primary reference describes Google's rules, not a universal agreement that every internet bot will obey them.

Example: a preview and a public guide

Imagine an unpublished preview and a finished guide sharing one layout. The preview needs an exclusion rule; the published guide should not inherit it accidentally. Inspect the HTML delivered by both URLs rather than relying only on a content-management switch.

A shared template can make a small metadata mistake affect many pages. The inverse is also true: removing all restrictions globally can expose pages that were deliberately excluded. The intended audience of each route matters.

How is it different from robots.txt?

Robots.txt controls crawler access to paths. If it prevents retrieval of the page, Google may not see the page's noindex instruction. Blocking a crawl and excluding an index entry are separate operations.

The X-Robots-Tag HTTP header is another place to carry indexing instructions. Check both the actual HTML and headers. Conflicting rules are not resolved simply by adding an index tag beside a remaining noindex rule.

What should be verified after a change?

Check the successful response on the intended public hostname and the deliberately excluded preview. Then use URL inspection to distinguish the current response from previously processed search data. Removing an accidental exclusion makes the page eligible for further processing; it does not guarantee indexing, immediate recrawling or a particular position.

Sources

  1. Google: Robots meta tag