Browse Definitions :

web crawler

What is a web crawler?

A web crawler, crawler or web spider, is a computer program that's used to search and automatically index website content and other information over the internet. These programs, or bots, are most commonly used to create entries for a search engine index.

Web crawlers systematically browse webpages to learn what each page on the website is about, so this information can be indexed, updated and retrieved when a user makes a search query. Other websites use web crawling bots while updating their own web content.

Search engines like Google or Bing apply a search algorithm to the data collected by web crawlers to display relevant information and websites in response to user searches.

If an organization or website owner wants its website to rank in a search engine, it must first be indexed. If webpages aren't crawled and indexed, the search engine can't find them organically.

Web crawlers begin crawling a specific set of known pages, then follow hyperlinks from those pages to new pages. Websites that don't wish to be crawled or found by search engines can use tools like the robots.txt file to request bots not index a website or only index portions of it.

Performing site audits with a crawling tool can help website owners identify broken links, duplicate content and duplicate, missing or too long or short titles.

web crawling process diagram
This diagram illustrates the web crawling process and how a web crawler searches and indexes a webpage.

How do web crawlers work?

Web crawlers work by starting at a seed, or list of known URLs, reviewing and then categorizing the webpages. Before each page is reviewed, the web crawler looks at the webpage's robots.txt file, which specifies the rules for bots that access the website. These rules define which pages can be crawled and the links that can be followed.

To get to the next webpage, the crawler finds and follows hyperlinks that appear. Which hyperlink the crawler follows depends on defined policies that make it more selective about what order the crawler should follow. For example, defined policies may include the following:

  • how many pages link to that page;
  • the number of page views; and
  • brand authority.

These factors signify a page may have more important information for indexing.

While on a webpage, the crawler stores the copy and descriptive data called meta tags, and then indexes it for the search engine to scan for keywords. This process then decides if the page will show up in search results for a query, and if so, returns a list of indexed webpages in order of importance.

In the event a website owner doesn't submit its site map for search engines to crawl the site, a web crawler can still find the website by following links from indexed sites that are linked to it.

Examples of web crawlers

Most popular search engines have their own web crawlers that use a specific algorithm to gather information about webpages. Web crawler tools can be desktop- or cloud-based. Some examples of web crawlers used for search engine indexing include the following:

  • Amazonbot is the Amazon web crawler.
  • Bingbot is Microsoft's search engine crawler for Bing.
  • DuckDuckBot is the crawler for the search engine DuckDuckGo.
  • Googlebot is the crawler for Google's search engine.
  • Yahoo Slurp is the crawler for Yahoo's search engine.
  • Yandex Bot is the crawler for the Yandex search engine.

Why web crawlers are important for SEO

Search engine optimization (SEO) is the process of improving a website to increase its visibility when people search for products or services. If a website has errors that make it difficult to crawl, or it can't be crawled, its search engine results page (SERP) rankings will be lower or it won't show up in organic search results. This is why it's important to ensure webpages don't have broken links or other errors and to allow web crawler bots to access websites and not block them.

Likewise, pages that aren't crawled regularly won't reflect any updated changes that may otherwise increase SEO. Regular crawling and ensuring that pages are updated can help improve SEO, especially for time-sensitive content.

How crawling fits into SEO
Web crawling collects information on the content of online published works, so a bot can then index it for SEO.

Web crawling vs. web scraping

Web crawling and web scraping are two similar concepts that can be easily confused. The main difference between the two is that while web crawling is about finding and indexing webpages, web scraping is about extracting the data found on one or more webpages.

Web scraping involves creating a bot that can automatically collect data from various webpages without permission. While web crawlers follow links continuously based on hyperlinks, web scraping is usually a much more targeted process -- and may only be after specific pages.

While web crawlers follow the robots.txt file, limiting requests to avoid overtaxing web servers, web scrapers disregard any strain they may cause.

Web scraping may be used for analytics purposes -- collecting data, storing then analyzing it -- in order to create more targeted data sets.

Simple bots can be used in web scraping, but more sophisticated bots use artificial intelligence to find the appropriate data on a page and copy it to the correct data field to be processed by an analytics application. AI web scraping-based use cases include e-commerce, labor research, supply chain analytics, enterprise data capture and market research.

Commercial applications use web scraping to do sentiment analysis on new product launches, curate structured data sets about companies and products, simplify business process integration and predictively gather data.

Learn more about four other SEO strategies including creating better topic clusters and backlink strategies.

This was last updated in September 2022

Continue Reading About web crawler

  • cloud-native network function (CNF)

    A cloud-native network function (CNF) is a service that performs network duties in software, as opposed to purpose-built hardware.

  • microsegmentation

    Microsegmentation is a security technique that splits a network into definable zones and uses policies to dictate how data and ...

  • Wi-Fi 6E

    Wi-Fi 6E is one variant of the 802.11ax standard.

  • MICR (magnetic ink character recognition)

    MICR (magnetic ink character recognition) is a technology invented in the 1950s that's used to verify the legitimacy or ...

  • What is cybersecurity?

    Cybersecurity is the protection of internet-connected systems such as hardware, software and data from cyberthreats.

  • Android System WebView

    Android System WebView is a system component for the Android operating system (OS) that allows Android apps to display web ...

  • privacy compliance

    Privacy compliance is a company's accordance with established personal information protection guidelines, specifications or ...

  • contingent workforce

    A contingent workforce is a labor pool whose members are hired by an organization on an on-demand basis.

  • product development (new product development -- NPD)

    Product development, also called new product management, is a series of steps that includes the conceptualization, design, ...

  • talent acquisition

    Talent acquisition is the strategic process employers use to analyze their long-term talent needs in the context of business ...

  • employee retention

    Employee retention is the organizational goal of keeping productive and talented workers and reducing turnover by fostering a ...

  • hybrid work model

    A hybrid work model is a workforce structure that includes employees who work remotely and those who work on site, in a company's...

  • Salesforce Trailhead

    Salesforce Trailhead is a series of online tutorials that coach beginner and intermediate developers who need to learn how to ...

  • Salesforce

    Salesforce, Inc. is a cloud computing and social enterprise software-as-a-service (SaaS) provider based in San Francisco.

  • data clean room

    A data clean room is a technology service that helps content platforms keep first person user data private when interacting with ...