robots.txt: what does this file control?
Robots.txt is a text file at a website’s root that tells cooperating crawlers which URLs they may fetch. It communicates crawling rules, not password protection, and does not guarantee that a URL disappears from Google results.
Also known as: robots file
At a glance
- Robots.txt governs crawler requests rather than visitor permissions.
- A blocked URL may still appear in search based on other links.
- Google reads noindex in page content or HTTP headers; it does not support it in robots.txt.
On this page

Where is it located, and what does it cover?
A conventional HTTPS location is https://example.com/robots.txt. The filename is lowercase and contents are plain text. Another subdomain can have its own file.
Google's specification ties scope to protocol, host and port. Checking the main domain therefore does not establish the blog subdomain's rules. /hu/robots.txt is not the normal site-wide location.
Always identify which crawler wants which exact URL. “We blocked bots” is too vague for diagnosis.
How do you read a simple file?
This fictional example illustrates syntax:
User-agent: *
Disallow: /print/
Sitemap: https://example.com/sitemap.xml
User-agent identifies a crawler or group; the asterisk supplies the general group. Disallow restricts matching URL paths for that group. Sitemap gives the absolute XML sitemap address.
The example restricts print versions, not the entire site, but can cover every matching path. Reducing the path to / changes the scope drastically. Check real paths and objectives before copying rules.
Why is it different from noindex?
Google's introduction distinguishes crawl control from search exclusion. A blocked crawler cannot retrieve page content, but the URL can remain known through other links.
Noindex instead requests index exclusion. Google documents that the crawler must access the response containing it. Google does not support a robots.txt noindex field.
Combining page noindex with a crawling block is not simply double protection: the latter can prevent reading the former.
What actually keeps a document private?
Robots.txt is public. It neither removes user access nor stops programs that disregard it. RFC 9309 explicitly distinguishes the protocol from access control.
Protect private quotes, personal information and internal documents with server-side authorization. Listing a restricted path may reveal its existence rather than conceal it.
Public accessibility and indexing are separate decisions. A public page can be excluded from search; a genuinely protected page is available only to authorized users.
Example: two accommodation pages
Imagine a guesthouse with a public room page and a guest's private booking document. The former should help prospective customers discover the offer; the latter should be accessible only to the appropriate guest.
Trying to solve both with robots.txt confuses crawler behavior with user permissions. Crawling may be desirable for the room page, while authorization is required for the document. This illustrates separate objectives, not a universal configuration recipe.
What should you check before changes?
State whether the goal is reducing unnecessary requests, excluding search appearances or protecting data. Inspect the actual file on the correct domain and representative matching URLs.
Review all crawler groups and avoid unintentionally affecting needed images, styles or public pages. Verify the served response rather than trusting only an editor setting.
Serving the file correctly belongs to technical SEO. HTTP status codes describe retrieval outcomes, while HTTPS protects transport. Assess these separately from crawl rules.