Crawling: when a search engine retrieves a webpage
Crawling is a search engine bot’s retrieval of web content: the bot requests a URL and downloads what is available there. Google’s search crawler is called Googlebot. Downloading a page does not establish that it is indexed or appears for any particular search.
Also known as: crawling · web crawling
At a glance
- A discovered URL may not yet have been downloaded.
- Crawling and indexing are separate stages, neither of which guarantees rankings.
- Discoverable links and functioning public delivery both matter.
On this page

How does a bot find a new page?
A search engine first learns about a URL through a link, an XML sitemap or another discovery route. Crawling occurs when it actually requests the address. Knowing a URL and downloading its content are separate states.
Google's explanation of Search distinguishes crawling, indexing and serving results. A bot visit establishes retrieval activity, not visibility for every relevant query.
Why do internal links matter?
A guide may work at its direct address while having no route from the rest of the website. Readers struggle to find it, and search systems have fewer discovery signals. A relevant internal link helps both.
Google's link guidance emphasizes HTML links containing a real URL. A label that responds to a custom click handler may not expose an extractable link. Descriptive wording such as “window-measuring guide” also identifies the destination more clearly than an isolated arrow.
The aim is not to link every page to every other page. Connect the reader's next relevant question.
What can prevent retrieval?
Lack of discovery differs from a known URL being inaccessible. Login requirements, server errors or an applicable robots.txt restriction can prevent access.
Google's technical requirements distinguish crawler access, a successful HTTP response and indexable content. A visually polished error page is not automatically a successful content response.
Inspect the actual response for the precise URL. A site owner's authenticated browser does not prove anonymous visitors or crawlers receive the same material.
Example: a new measuring guide
Imagine a curtain shop publishing a window-measuring guide without adding it to navigation or product pages. The owner can open its direct link, but that does not establish search discovery.
A sensible step is linking it from a relevant product explanation and checking the sitemap. Crawler access and indexing can then be investigated separately.
This fictional example does not promise discovery within a set time or a ranking increase after adding two signals.
What is crawl budget?
Search engines do not send unlimited requests to every site. Crawl budget relates to available capacity and demand for retrieval. Google's documentation discusses serving performance and content value.
On a small business site, one missing article should not immediately be blamed on exhausted budget. First establish where the process fails. Many unnecessary URL variants present a different problem from one accidentally blocked service page.
Where should an investigation begin?
Record the exact URL, observation date and evidence source, then check:
- Does an accessible page contain a normal link to it?
- Can an anonymous visitor retrieve the actual content?
- Do the relevant crawler rules permit access?
- Is there evidence of crawling, and when?
- What does the separate indexing status show?
This avoids rewriting content to fix an access failure or adding links to fix a server error.
JavaScript SEO concerns content produced by execution, and mobile-first indexing concerns the mobile version used for processing. AI web crawlers require provider-specific interpretation. Crawl Stats supplies aggregated Google observations rather than every bot's complete log.