Server log analysis: what crawlers actually requested
Server log analysis examines requests recorded by a website’s serving layers. Logs can show which URLs were requested, when and with what result, but their location and coverage determine what is visible. A request that claims to be a bot requires separate identity verification.
Also known as: Log file analysis · Crawler log analysis
At a glance
- The name Googlebot in a User-Agent is not proof of identity.
- Requests served by a CDN may never appear in application-server logs.
- A successful request does not prove indexing, ranking or a human visit.
On this page

What a request log can contain
A log may record request time, path, method, response code and other serving details. These can reveal whether a crawler reaches an important page regularly or repeatedly requests broken URLs.
Available fields depend on the provider and configuration. Before drawing conclusions, establish the period, time zone, sampling and serving layer involved. Not every log is complete, and not every request reaches the same system.
How logs differ from analytics
Browser analytics can observe user events and visits. Server logs describe serving requests. Many crawlers do not execute analytics like a person’s browser, so the totals should not be expected to match.
An analytics view of organic traffic is not a Googlebot request count. Fetching an image or stylesheet is not another visitor. Logs can, however, reveal a failure that prevented browser analytics from running at all.
Verifying Googlebot
The sender supplies the User-Agent header, so its contents can be forged. Google documents verification through published IP ranges or appropriate reverse and forward DNS checks. Googlebot verification
Use a client address from a trusted source. An arbitrarily supplied forwarding header may not identify the real requester. If a CDN or proxy sits in front of the website, understand its documented logging and forwarding behavior instead of automatically trusting a familiar header name.
Example: a crawler disappears from the origin log
Imagine a knowledge base enabling a new cache. Its application server records fewer requests attributed to Googlebot, although the public pages remain available. The team initially suspects less crawling.
The CDN logs show that edge servers now answer some crawler requests without contacting the origin. In this example, the decline reflects a changed serving path. Bot identity and actual responses still require verification; the application log simply no longer covers the whole request flow.
Patterns worth investigating
Compare URL families, HTTP status codes and response times. Repeated requests for broken parameter URLs, redirect chains or frequent server errors can provide specific investigation targets.
Before invoking crawl budget, establish whether the website is large enough and important pages are actually being missed. A few unnecessary requests are not automatically a serious problem. Consider the effect on access to useful content and operating cost. Crawl statistics
What the evidence does not prove
A recorded 200 response establishes what that serving layer logged. It does not show that Google indexed the page or displayed it in search. Google Search Console and logs provide complementary evidence. Page indexing
Logs may contain sensitive data in parameters or identifiers. Limit access and retention appropriately, and filter shared examples. An external support ticket generally needs a relevant anonymized sample, not an entire log containing user data.