Browse Definitions :
Definition

spider

A spider is a program that visits Web sites and reads their pages and other information in order to create entries for a search engine index. The major search engines on the Web all have such a program, which is also known as a "crawler" or a "bot." Spiders are typically programmed to visit sites that have been submitted by their owners as new or updated. Entire sites or specific pages can be selectively visited and indexed. Spiders are called spiders because they usually visit many sites in parallel at the same time, their "legs" spanning a large area of the "web." Spiders can crawl through a site's pages in several ways. One way is to follow all the hypertext links in each page until all the pages have been read.

The spider for the AltaVista search engine and its Web site is called Scooter . Scooter adheres to the rules of politeness for Web spiders that are specified in the Standard for Robot Exclusion (SRE). It asks each server which files should be excluded from being indexed. It does not (or can not) go through firewall . And it uses a special algorithm for waiting between successive server requests so that it doesn't affect response time for other users.

This was last updated in April 2005
SearchNetworking
SearchSecurity
  • man in the browser (MitB)

    Man in the browser (MitB) is a security attack where the perpetrator installs a Trojan horse on the victim's computer that is ...

  • Patch Tuesday

    Patch Tuesday is the unofficial name of Microsoft's monthly scheduled release of security fixes for the Windows operating system ...

  • parameter tampering

    Parameter tampering is a type of web-based cyber attack in which certain parameters in a URL are changed without a user's ...

SearchCIO
  • chief procurement officer (CPO)

    The chief procurement officer, or CPO, leads an organization's procurement department and oversees the acquisitions of goods and ...

  • Lean Six Sigma

    Lean Six Sigma is a data-driven approach to improving efficiency, customer satisfaction and profits.

  • change management

    Change management is a systematic approach to dealing with the transition or transformation of an organization's goals, processes...

SearchHRSoftware
SearchCustomerExperience
  • clickstream data (clickstream analytics)

    Clickstream data and clickstream analytics are the processes involved in collecting, analyzing and reporting aggregate data about...

  • neuromarketing

    Neuromarketing is the study of how people's brains respond to advertising and other brand-related messages by scientifically ...

  • contextual marketing

    Contextual marketing is an online marketing strategy model in which people are served with targeted advertising based on their ...

Close