Bot verification: Meaning, examples and limitations
Bot verification is the process of checking whether an automated request actually comes from the provider it claims to represent. It uses evidence such as documented IP ranges or validated DNS records rather than relying only on the user-agent name.
At a glance
- A crawler name is easy to imitate.
- Use the specific provider’s current verification method.
- Verified identity does not remove the need for access and cost controls.
On this page

What is being verified?
A request can claim to be Googlebot by putting that name in its user-agent header. That label is not cryptographic proof. Verification compares the request's origin with independently documented provider information.
Google describes checking published IP ranges or performing reverse DNS followed by forward DNS confirmation. The hostname must match the documented domain rules, and the forward result must include the original address. A convincing-looking name alone is insufficient. Provider reference
Example: a sudden crawl spike
Imagine a publisher seeing many requests that identify themselves as a search crawler. The first question is whether they really belong to that provider. The next is what they request and what those requests cost. These are different investigations.
If the identity is genuine, the traffic can still expose a crawl trap or repeatedly request expensive dynamic pages. If it is false, an allow rule based only on the user-agent may have given an impersonator unnecessary access.
What are common verification mistakes?
Checking reverse DNS without checking the forward result is incomplete for the documented DNS method. Matching a hostname by an unbounded substring can also accept a deceptive name. Storing a provider's IP list forever ignores future updates.
Use an implementation that follows the provider's published procedure. A service's “verified bot” classification has its own scope; inspect its documentation before treating every automated client as covered.
What does verification not authorize?
It does not grant access to private account pages, waive application permissions or guarantee that every request is inexpensive. Public content delivery and authenticated product operations remain separate. Review request logs, response codes and rate limits using both identity and behavior. Verification is evidence for a routing or security decision, not a substitute for that decision.