Skip to the content
searchboar.

Crawling and indexing

Crawl Stats: understand Google’s requests to your site

Crawl Stats is Search Console’s aggregated reporting on Google’s crawling requests to a website. It helps identify serving problems and request patterns. It does not measure visitor traffic, provide a complete server log or count indexed pages.

Also known as: Crawl Stats · Googlebot crawl statistics

At a glance

  • A crawl request is not the same as a newly indexed page.
  • Resources and redirect steps contribute to request counts alongside HTML pages.
  • Investigate errors with URL types, dates and evidence from the serving system.
On this page
Crawl requests are grouped by period, response and file type.
Crawl statistics describe crawler requests, not visitor totals or the number of indexed pages.

What questions does the report answer?

Crawl Stats helps investigate how Google encountered the website's serving system. Did errors increase? Did responses become slower? Which resource types received requests? These concern access to the content.

Search Console's guidance describes this as an advanced tool. On a small website, accessible and useful main pages generally matter more than increasing the daily Googlebot count. The report is for diagnosis, not endlessly rising graphs.

What contributes to request counts?

During crawling, Google requests HTML and other resources. Redirects can create multiple requests because the original address and destination each return a response. Reporting concerns the requested addresses rather than merging every observation into the canonical page.

A hundred requests need not represent a hundred distinct articles. Images, stylesheets, repeated fetches and redirects can contribute. A template change can alter resource counts without adding content, so inspect breakdowns before interpreting totals.

What do statuses and response times reveal?

An HTTP status code distinguishes successful, redirected and failed responses. Response times and availability indicators offer additional clues. Compare timing with maintenance and releases when investigating a spike.

Not every 404 requires correction. A genuinely removed page can correctly report absence. A main service page unexpectedly failing is different. The business meaning depends on what should have been available at that URL.

Example: more requests without much new content

Imagine a shop adding ten product pages weekly while crawl requests triple over several days. The owner initially assumes Google values the site more. A breakdown instead reveals filter parameters generating many similar addresses.

The next task is investigating URL patterns and navigation. Do those variants provide distinct value, and are important products still accessible? This fictional example does not establish a universal crawl-budget problem, but shows why more requests are not automatically success.

How does this connect with other evidence?

Server logs can provide more detailed request records where the relevant layer's logs exist. Coverage can differ: a request served from a cache may never reach the application server.

Search Console indexing and performance reports concern other stages. A retrieved page can remain unindexed; an indexed page can receive no clicks. Investigate the relationship at URL level over a suitable period rather than treating adjacent charts as proof of a causal chain.

What should the resulting task look like?

Identify a concrete broken route, deteriorating response time or unnecessary URL family. Record evidence, affected pages, responsibility and a follow-up check. “Get more crawling” is too vague.

If serving and access are healthy, attention may shift to content and discovery. Uneven daily request counts alone do not justify unnecessary infrastructure replacement.

Sources

  1. Search Console: Crawl Stats report
  2. Google: Verifying Googlebot
  3. Search Console: Page indexing report