You are presented with multiple paper abstracts, alongside the associated literature review and chapter titles. Given the provided information, you are going to do a literature review. Please generate a coherent literature review that synthesizes the content, draws connections between different works, highlights major findings, and identifies any gaps in the existing research.

Literature Review Title: Change Detection and Notification of Webpages: A Survey 
Chapter Title:  DETECTING CHANGES OF WEBPAGES Web Crawlers and Crawling Techniques 
1. Abstract of Cited Paper (BIB001):  Information Retrieval deals with searching and retrieving information within the documents and it also searches the online databases and internet. Web crawler is defined as a program or software which traverses the Web and downloads web documents in a methodical, automated manner. Based on the type of knowledge, web crawler is usually divided in three types of crawling techniques: General Purpose Crawling, Focused crawling and Distributed Crawling. In this paper, the applicability of Web Crawler in the field of web search and a review on Web Crawler to different problem domains in web search is discussed. 
2. Abstract of Cited Paper (BIB002):  Although web crawlers have been around for twenty years by now, there is virtually no freely available, open-source crawling software that guarantees high throughput, overcomes the limits of single-machine systems, and, at the same time, scales linearly with the amount of resources available. This article aims at filling this gap, through the description of BUbiNG, our next-generation web crawler built upon the authors’ experience with UbiCrawler [9] and on the last ten years of research on the topic. BUbiNG is an open-source Java fully distributed crawler; a single BUbiNG agent, using sizeable hardware, can crawl several thousand pages per second respecting strict politeness constraints, both host- and IP-based. Unlike existing open-source distributed crawlers that rely on batch techniques (like MapReduce), BUbiNG job distribution is based on modern high-speed protocols to achieve very high throughput.

Literature Review:
"A web crawler (also known as a spider) is a software or a programmed script that browses the WWW in a systematic, automated manner" BIB002 ] and systematically downloads numerous webpages starting from a seed BIB003 ]. Web crawlers date back to the 1990s, where they were introduced when the WWW was invented. The "World Wide Web Worm"  ] and "MOMspider"  ] were among the early web crawlers. Moreover, commercial search engines developed their own web crawlers as well such as "Google crawler" ], "AltaVista" BIB001 ], etc. Later on, web crawlers which could efficiently download millions of webpages were built. We can consider the Internet to be a "directed graph" where each node represents a webpage and the edges represent hyperlinks connecting these webpages BIB002 ]. Web crawlers traverse over this graph like structure of the Internet, go to webpages and download their content for indexing purposes. It is crucial to identify which crawling technique should be used according to the purpose of the application. Web crawling can be considered as the main process behind web caching, search engines and web information retrieval. A web crawler begins crawling from a seed URL and visit pages. Then it downloads the page and retrieves the URLs in it. The discovered URLs will be kept in a queue and this process repeats as the crawler travels from page to page. Figure 7 shows an overview of the web crawling process. Many crawling techniques are being used by web crawlers at present BIB002 ] such as (1) "General Purpose Crawling", (2) "Focused Crawling" and (3) "Distributed Crawling" BIB002 ]. 4.1.1. General Purpose Crawling. In the general-purpose crawling technique, the web crawlers collect all the possible webpages from a set of URLs and their links to a central location. It can fetch numerous pages from many different locations. However, this technique can slow down the crawling process and reduce the network bandwidth as all the pages are being fetched.