Skip to the content
searchboar.

Topic

Crawling and indexing

How search engines access, process, and select pages for their indexes.

Browse A–Z

71 definitions available

Searchboar glossary

All terms

71 terms

B

  • Bot verification is the process of checking whether an automated request actually comes from the provider it claims to represent. It uses evidence such as documented IP ranges or validated DNS records rather than relying only on the user-agent name.

    Read definition
  • Understand broken links, error responses and incorrect destinations, and how to repair the customer paths that matter most.

    Read definition

C

  • How browser and shared caches differ, what no-cache and no-store mean, and how to verify that readers receive the correct content.

    Read definition
  • Cache invalidation is the process of making a stored response unusable or requiring it to be refreshed when it is no longer appropriate. It helps align delivered content with a change, but each cache layer and variant needs to be considered.

    Read definition
  • Cache-Control is an HTTP header containing instructions for storing and reusing responses in browsers and shared caches. Its directives describe freshness, validation and storage permissions; they do not decide whether a page belongs in a search index.

    Read definition
  • Understand canonical URL selection, self-referencing tags and duplicate versions, and learn how canonical differs from redirects and noindex.

    Read definition
  • How a content delivery network differs from caching, where it can improve delivery, and why freshness and private data need separate rules.

    Read definition
  • Click depth is the number of link-following steps needed to reach a page from a chosen starting page, often the homepage. It describes a navigation path, not simply the number of slashes in a URL.

    Read definition
  • Client-side rendering, or CSR, uses JavaScript in the browser to create or update page content. In a fully client-rendered page, the initial HTML may contain only a shell while scripts and data requests produce the main document later.

    Read definition
  • What Cumulative Layout Shift measures, how it differs from loading speed, and how reserved space can prevent disruptive jumps.

    Read definition
  • How LCP, INP and CLS describe different user experiences, what the thresholds mean, and why field data differs from a laboratory test.

    Read definition
  • When crawl budget matters, why more requests do not guarantee indexing, and how to investigate unnecessary URLs without blocking useful content.

    Read definition
  • What Search Console Crawl Stats includes, why request counts differ from pages and indexing, and how to connect errors with server evidence.

    Read definition
  • A crawl trap is a website structure that leads crawlers into very large or effectively endless sets of low-value URL variations. Calendars, filters and recursively generated links can create such paths even when the underlying useful content is limited.

    Read definition
  • How crawling differs from discovering and indexing a URL, what can prevent access, and how to investigate a missing page in the right order.

    Read definition

D

  • Why duplicate content can arise without copying, how canonical URLs differ from redirects and noindex, and what to check after changes.

    Read definition

F

  • How filter combinations expand a website’s URL space, which filtered pages deserve search visibility and why robots, noindex and canonical differ.

    Read definition
  • Understand FCP, its difference from LCP and how to interpret the first visible content without mistaking it for a complete or interactive page.

    Read definition

G

  • Googlebot is the common name for Google Search crawlers that retrieve web resources for processing. A Googlebot visit establishes that a request was made; it does not by itself prove indexing, a ranking position or inclusion in an AI answer.

    Read definition

H

  • Understand how hreflang connects equivalent language and regional pages, how it differs from canonical and lang, and which errors break the relationship.

    Read definition
  • HTTP 304 Not Modified tells a client making a conditional request that its stored representation can be reused. The server sends validation information instead of sending the resource body again.

    Read definition
  • HTTP 404 is a response status indicating that the server cannot find the requested resource. It describes the response to a particular URL, not the health or availability of the entire website.

    Read definition
  • HTTP 410 Gone indicates that a resource is no longer available at the requested address and that this condition is likely permanent. It is a deliberate statement about removal, stronger in meaning than simply not finding a page.

    Read definition
  • HTTP 429 Too Many Requests means that a client has exceeded a request rate accepted by a server or intermediary. It asks the requester to slow down; it does not mean the requested page has been permanently removed.

    Read definition
  • HTTP 503 Service Unavailable indicates that a server is temporarily unable to handle a request, for example during overload or maintenance. It distinguishes temporary unavailability from a resource that is missing or permanently gone.

    Read definition
  • Learn what common HTTP responses mean, why the code must match the content, and how to distinguish a correct 404 from a service failure.

    Read definition
  • Understand what HTTPS protects, what a certificate does not prove, and why redirects, resources and page responses still need attention.

    Read definition

I

  • Understand the difference between crawling, indexability and actual indexing, and learn how to interpret an excluded URL without assuming every exclusion is an error.

    Read definition
  • Learn how IndexNow notifies participating search engines about changed URLs, what a successful submission proves, and how it differs from Google’s Indexing API.

    Read definition
  • Infinite scroll is an interface pattern that loads more content as a visitor moves down a page. For search discoverability, the additional content needs an accessible loading and linking strategy; a crawler should not have to imitate a person scrolling indefinitely.

    Read definition
  • Understand how INP measures interaction responsiveness, what the 200 ms threshold means, and why visible feedback differs from completing a background task.

    Read definition

J

  • Learn how initial HTML, rendering and indexing differ, and how SSR, SSG and CSR affect reliable access to public content.

    Read definition

L

  • Learn which images can load later, why the main visible image should not wait, and how lazy loading differs from image compression.

    Read definition
  • When does the largest visible content element appear? A practical explanation, example and important distinctions.

    Read definition

M

  • Understand Google manual actions, affected pages, documented fixes and reconsideration requests, separately from algorithmic ranking changes.

    Read definition
  • Why does the mobile version provide the content Google processes? Practical explanation, example and important distinctions.

    Read definition

N

  • What does nofollow say about a link, and when are sponsored or ugc appropriate? Practical explanation, examples and distinctions.

    Read definition
  • Noindex and nofollow explained with HTML and HTTP examples, robots.txt conflicts and a homepage accidentally excluded from search.

    Read definition

O

  • What makes a page orphaned, how it differs from a missing or noindexed page, and how to reconnect it to relevant website content.

    Read definition

P

  • Search Console’s Page indexing report explained: discovered or crawled but not indexed, and when action is appropriate.

    Read definition
  • Page speed describes how quickly a page delivers useful content and responds during a visit. It is a broad performance concept rather than one universal score, and it varies with the page, device, connection and user interaction.

    Read definition
  • How should later pages of a long list remain accessible? A practical explanation, example and important distinctions.

    Read definition
  • Set measurable performance budgets for assets and experience metrics, define the test conditions and handle exceptions through deliberate trade-offs.

    Read definition
  • How 301 and 308 redirects communicate a lasting move, differ from canonical and noindex, and preserve useful routes to replacement content.

    Read definition
  • Understand when preload helps important images or fonts, how matching requests matters and why indiscriminate preloading can waste bandwidth.

    Read definition

R

  • How redirect chains form, why loops fail, and how to check the complete URL path and the relevance of its final destination.

    Read definition
  • Understand render-blocking resources, the difference from parser blocking and how to improve loading without breaking layout or functionality.

    Read definition
  • Rendering is the process of turning page resources into the document and visual interface that a user or crawler can process. HTML, CSS, images and sometimes JavaScript contribute to the result; downloading the initial HTML is not always the end of the process.

    Read definition
  • A robots meta tag is an HTML element that gives supported crawlers page-level instructions about indexing and search presentation. It can carry directives such as noindex or snippet controls, but it does not authenticate visitors or block access to the page.

    Read definition
  • What is robots.txt for, and what does it not protect? User-agent, Disallow and Sitemap examples, with the crucial noindex distinction.

    Read definition

S

  • Understand Domain and URL-prefix properties, which URLs their reports include and what ownership verification actually grants.

    Read definition
  • Understand temporary Search Console removals, permanent exclusion methods and why hiding a Google result does not remove the underlying content from the internet.

    Read definition
  • Understand request logs, verified Googlebot identity, CDN visibility and why a successful request does not prove indexing.

    Read definition
  • Server-side rendering, or SSR, creates a page’s HTML on the server before sending it to the browser. The response can already contain meaningful content, while optional client-side code adds interactive behavior afterward.

    Read definition
  • Understand nosnippet, max-snippet and data-nosnippet, how their scope differs from noindex, and why none of them protects confidential information.

    Read definition
  • Why Google can treat a URL returning 200 as missing, how this differs from a real 404 and what to check before fixing it.

    Read definition
  • Static site generation, or SSG, creates HTML documents before visitors request them. The finished files can be delivered directly, so reading a page does not inherently require running the application or querying a database for every visit.

    Read definition
  • A subdirectory is a path section beneath a website’s hostname, such as /glossary in example.com/glossary/keyword. On modern websites it may be a logical route prefix rather than a physical folder on a server.

    Read definition
  • A subdomain is a named subdivision of a domain, such as blog.example.com under example.com. It creates a distinct hostname, which can have its own hosting and configuration; it is different from a path such as example.com/blog.

    Read definition

T

  • Understand access, indexing controls, URL handling and rendering, and how to verify a technical fix without claiming unmeasured search results.

    Read definition
  • Learn when 302 and 307 redirects are appropriate, how they differ, and why a temporary destination needs a clear end date and useful content.

    Read definition
  • Understand TBT calculations, long main-thread tasks and the distinction between a lab responsiveness diagnostic and real interaction latency measured by INP.

    Read definition
  • Understand Time to First Byte, its relationship to LCP and why a fast initial response does not prove the page loads quickly.

    Read definition

U

  • A URL fragment is the part after the # character that identifies a location or state within a resource. In ordinary web navigation, the browser uses it locally; it is not sent to the server as part of the HTTP request target.

    Read definition
  • Understand indexed data, live tests, canonical information and indexing requests without confusing a successful test with a search appearance.

    Read definition
  • URL normalization is the deliberate conversion of equivalent address forms into a consistent representation. It can reduce accidental duplication, but equivalence must follow the website’s real behavior rather than assumptions about case, slashes or parameters.

    Read definition
  • A URL parameter is a named value in a URL’s query string, commonly written after a question mark. Parameters can select content, apply filters or record campaign information, so their meaning must be understood before they are removed, cached or canonicalized.

    Read definition

W

X

  • X-Robots-Tag is an HTTP response header for supported indexing and search-preview directives. It can apply rules to HTML pages and to files such as PDFs, where an HTML robots meta tag is not available.

    Read definition
  • Which URLs belong in a sitemap, what lastmod should represent and why successful processing does not guarantee indexing.

    Read definition