Britain has embraced the web. With over sixty percent of the UK population online and three quarters of those connected by high speed broadband, it’s becoming difficult for many of us to imagine a time before the Internet touched our lives.
Search engine statistics show that we use the web for many reasons. We satisfy our thirst for celebrity, play games and look for movie reviews. We find directions, look up the latest news and investigate our health. We talk to people around the globe who share our interests. We connect with each other.
Search engines take a giant haystack full of needles and index each and every one. When you need to find the right answers from the web at any time of the day or night, all you have to do is ask.
This book celebrates search engines. It traces their history back to the beginning of the web and before, showing how they’ve grown. Along the way we’ll uncover the human side of the web, the people who use it and make it what it is. We’ll round things off with a look at the present and the future of the search engine. It’s a future that’s assured. Search engines are here to stay.
In the Beginning…
Before the web, the Internet was a stuffy and self-contained place. Started by the U.S. military in the late 60s and used mainly by academic boffins, the key applications for many years were email, file transfer (FTP) and a terminal connection protocol called “telnet”. The latter allowed you to connect to remote systems and, given the right credentials, access their contents. Some servers ran bulletin board systems, some ran rudimentary games, others simple databases. Like all secret worlds, there were interesting enclaves even then. An early search engine called “Archie” could be used to find files on FTP servers, but the only way to find other resources was to be told about them.
Fast forward to 1991. For most people the story of Internet search starts with the World Wide Web, but in the early 90s there was a competing system already in place. Gopher was a distributed directory protocol that allowed you to browse from one site to another using embedded links. Although that sounds similar to the way the web works, Gopher lacked the “page” metaphor. It could only index static documents. Still, the jokily named system spawned the first true search engine, Veronica (Very Easy Rodent-Oriented Net-wide Index to Computer Archives).
Although Gopher and the WWW were near contemporaries, the web attracted few users until 1993 when the first graphical web browser NCSA Mosaic was released. The new software transformed the way people were able to access information online and Gopher disappeared back into its burrow.
During those early days, from the web’s launch in 1991, there were no search engines. Instead, folks passed site addresses around by email or they built rudimentary directories. The original “home page” was a simple site you created yourself, full of links to other pages.
Then, in 1993, a dedicated search engine was launched. ALIWEB (Archie Like Indexing for the Web ) was still quite different from current models. The only way to get a page listed was to submit an address to the engine using an online form. Modern search engines actively “spider” the web as well, following links and adding new pages to their databases automatically. WebCrawler, in 1994, was the first engine of this kind and it was quickly followed by Lycos, Excite and InfoSeek – sites with similar technology. In a parallel move, Yahoo! launched its web directory service. These and other search engines changed the way users access the web forever.
The advent of search engines began the mainstreaming of the Internet in earnest. In 1994 the first Internet radio stations began, broadcasting to American universities with fixed line connections. Few people were able to access the Net from home Desptite that, Pizza Hut still launched its first online ordering service…
1995 to the year 2000 were the web’s boom years; the dotcom bubble. It was a time of rapid development, change and excess. As the web moved from schools and universities to the homes of early adopters, modem speeds went from 14 to 28 to 56 kilobits per second. The fastest equipment was still 10 times slower than the first broadband package.
Only the strongest and best search engines survived this turbulent period. Askjeeves.com (now known as Ask.com), launched in 1996, rode the storm like a stalwart sea captain, carving a niche with its innovative use of question based queries. The approach was ideal for this simpler time when everyone was a newbie. HotBot was briefly the most popular engine around, with a comprehensive boolean interface designed with computer savvy users in mind. It’s no surprise that Google, with a much plainer design, had taken the crown by the end of the decade.
Few statistics survive from the bubble years. We know that in 1998 people were already searching for adult content, the Simpsons and Star Trek. They were interested in the same mix of celebrities, news, music and movies that Internet surfers are today. The main changes have been driven by evolving technology. From a simple network of text documents in 1992, the web became a vibrant place with Flash animation and full colour images by the year 2000. As the 21st century progressed, video, blogs and sound would be added to the mix – and the search engines grew and evolved to meet these new challenges.