History of Search Engines
Search engines have become such an integral framework for how we learn, cross-check facts, and process information, that is difficult to remember a time when they were not an available resource. Instead of logging hours poring over books in a library, we can access seemingly limitless databases at our fingertips in a matter of seconds. Today the world of search engines is not only a vast source of information, but has become one of the most influential platforms for marketing and growing businesses--this is where SEO comes into play.
What search engines are and how they work
Search engines are usually "crawler based," meaning that they that crawl the internet searching for web pages by scanning for key words. They scan the content and information, and follow links that lead to other pages. Search engines operate using algorithms, which find information on websites, and store them in a large index, or catalogue. As sites are updated or changes, the index is updated as well.
As users search the web for keywords, they will provide answers and find results from what the engine has crawled and stored on the index, by calculating the relevancy. The engines sort through pages, and bring up the matches that are closest to the keywords searched. The pages are ranked according to the greatest relevance of content, and the order of ranking greatly contributes to the site's popularity and success.
How Search Engine Development Began
The need for search engines was first noted in 1945 when American engineer and scientist Vannevar Bush published an article in The Atlantic Monthly, emphasizing the necessity for an expansive index for all knowledge: "[Information] has been extended far beyond our present ability to make real use of the record. A record, if it is to be useful to science, must be continuously extended, it must be stored...Our ineptitude in getting at the record is largely caused by the artificiality of the systems of indexing. The human mind does not work this way. It operates by association."
Decades later, college students and electrical engineers attempted to make this kind of index a reality. The development of Archie (named for "archives"), the very first search engine created in 1990, was designed to search and store directory listings on file transfer protocol sites. Similar engines began to follow suit, when Gopher (a site that indexed text files) came on the scene, it was followed by Veronica and Jughead, both of which corresponded to Gopher. These early versions paved the way for the technologies utilized by main line search engines today.
For example, JumpStation was the first world wide web resource-discovery tool which employed crawling and indexing. This was followed by the first full-text search engine, WebCrawler created in 1994 by Brian Pinkerton. Prior to this, only webpage names/domains were indexed in catalogues. As the internet expanded, search engines and web directory technologies increased, creating competitive platforms for the strongest emerging technologies to arise.
Search Engines Development Timeline
Archie (1990) was the first tool created by Alan Emtage and L. Peter Deutsch for indexing, and is considered the first basic search engine. What began as school project at McGill University in Montreal, was an index that predated the world wide web. Gopher, released in 1991 by students from the University of Minnesota, was a protocol used to index and search for documents online as a form of anonymous FTP. Archie, Gopher and similar counterparts lost traction in the late 90's.
Lycos (1993) was created as a university project, but was the first to attain commercial search engine success. In 1999 Lycos was the most visited search engine in the world and was available in 40 countries. Now it is currently comprised of a social network with email, webhosting and media entertainment pages.
Yahoo! (1994) started at Stanford University by Jerry Yang and David Filo (both electrical engineering grad students) that became a web portal and search engine.
WebCrawler (1994) created by Brian Pinkerton WebCrawler was the first crawler which indexed complete pages online. AOL purchased WebCrawler, using the technology for their network, and when Excite purchased WebCrawler, AOL used Excite to run their program NetFind. WebCrawler was one of the foundational search engines.
AltaVista (1995) an industry leader, was once the most popular search engines of its time. It differed from its contemporaries because of two factors: Alta Vista used a multi-threaded crawler (Scooter) that covered more webpages than people knew existed at the time. It also had a well-organized search-running back-end advanced hardware. By 1996 AltaVista had become the sole search results provider for Yahoo. In 2003 Alta Vista was bought by Overture Services, Inc., which only months later was acquired by Yahoo.
Looksmart (1995) competed with Yahoo! Directory and had an initial a goal to create a substantial directory of websites. When it went public in 1999, it lost a fair amount of customers. By 2002 Looksmart became a pay-per-click provider, and after being dropped by Microsoft, bought a search engine called WiseNut. Sadly it never gained serious traction, and Looksmart lost its momentum.
WiseNut (2001) was a crawler-based search engine that was introduced as a beta, and was owned by Looksmart. Initially the site was well reputed with an unsullied database, and an automatic clustering of search results by using a technology called WiseGuide. Looksmart bought WiseNut in 2002, and was eventually closed in 2007.
Excite (1995) Founded originally as "Architext" by Stanford University students, Excite was launched officially having purchased two search engines (Magellan and WebCrawler), and signed exclusive agreements with Microsoft and Apple. Fellow Stanford students Larry Page and Sergey Brin once offered to sell their startup "Google" to the Excite team for $1 million, which was refused. Google became a $180 billion company, and Excite is now used as a personal portal called My Excite.
Hotbot (1996) a search engine also popular in the 90's was launched by Wired Magazine, and is now owned by Lycos.
Dogpile (1996) was a search engine developed by Aaron Flin and shortly thereafter sold to Go2net. Now Dogpile fetches results from Google, Yahoo, and Yandex.
Google (1996) Started for a research project by Stanford students Larry Page and Sergey Brin. They created a search engine that would rank websites based on the number of other websites that linked to that page. Prior to this, other engines have ranked sites based on the number of times the search term appeared on the webpage. This strategy developed the world's most successful search engine today.
MSN Search (1998) was the engine used by Microsoft, sourcing search results from Inktomi, and later Looksmart. By 2006 Micosoft started performing their own image searches, and MSN became branded as Windows Live Search, then Live Search, and finally to Bing (2009) which was set to replace the Yahoo search engine.
ASK (1996) was originally titled "AskJeeves.com" and was designed by Garret Gruener and David Warthen in Berkeley, CA. The goal was to provide users with answers to queries typed with normal every-day language and colloquialisms. It was acquired in 2005 by IAC and continues to grow with over 100 million users.
Teoma (2000) meaning "expert" in Gaelic, was a search engine created by professor Apostolos Gerasoulis and Tao Yang at Rutgers University. Teoma's subject-specific technology centered on a link popularity algorithm which allowed pages to rank higher if other pages with a similar content and subject matter linked back to the page. Teoma was acquired by Ask Jeeves in 2001, and rebranded as Ask.com.
Infoseek (1994) was a search engine begun by Steve Kirsch, and was bought by The Walt Disney Company in 1998, merging with Starwave to become go.com. Eventually it was replaced by Yahoo, and no longer exists.
Overture (1998) was originally named "GoTo," where top listings were sold on a cost-per-click or pay-per-click basis. In 2000, they began driving traffic by placing its paid listings on other search engines, and was eventually bought by Yahoo in 2003.
Alltheweb (1999) began in 1994 out of FTP Search, from Norwegian University of Science and Technology, when then turned into Fast Search & Transfer, or FAST. Alltheweb (1999) was said to have once rivaled Google, but the number of users declined when Overture bought the company in 2003.
AOL Search (1999) bought Web Crawler (one of the major crawler-based engines of it's time) in 1995, and after a number of deals, purchases and exchanges, AOL relaunched their search engine, calling it AOL Search. Teaming with Google, the search engine relaunched in 2006 with newer features including video, search marketplace, etc.
Newer Search Engines
Cuil (2008) was a search engine that arranged pages by content, showing large entries with pictures, and thumbnails for results etc. The search engine claimed to have over 120 billion web pages indexed, and would not store user's search activity or their IP number. By 2010, Cuil suddenly shut down after an acquisition agreement failed to go through, and their patents were sold to Google when the company dissolved.
Secure Search Engines
Ixquick.com (1998) is a metasearch engine that offers a proxy service for Ixquick and an email service that offer privacy protection, called StartMail. It was relaunched in 2005 and included a re-engineered metsasearch algorithm. Devoted to privacy, Ixquick entirely ended recording IP addresses, and only used one cookie, which is set to remember the user's search preferences for future searches and is removed once a user does not return to the search engine homepage after 3 months. Ixquick was recertified with Startpage in 2013.
StartPage (2009) is a secure search engine, meaning it pulls all the same results as Google, but uses the privacy protection of Ixquick, which allows users to search with privacy.
DuckDuckGo (2006) is a search engine that does not store or share any information about the user, and is unique to other search engines by providing all users the same results for a given search term, as well as providing search results from what they describe as the "best sources" rather than from the most sources. DuckDuckGo's results are sourced from places like Yahoo, Search Boss, Wikipedia, Wolfram Alpha, and their own web crawler the DuckDuckBot, etc.
Specialized Search Engines
Wolfram Alpha (2009) is a "computational knowledge engine" that answers factual queries by computing the answer from externally sourced "curated data" instead of listing relevant websites which could lead to the answer.
Major Non-US Search Engines
Baidu (2000) is one of the main search engines in China, based on a special identification technology that classifies and groups articles. Baidu locates information, products, and services through Chinese language search terms (via phonetic Chinese), advanced searches, snapshots, spell checker, stock quotes, news, images, video, space information, weather, train and flight schedules and other local information. Baidu's greatest competitors are Google Hong Kong and Yahoo! China.
Yandex (1997), originally standing for "yet another indexer," is the largest search engine in Russia, and ranked as the 4th largest search engine in the world, serving over 150 millions searches per day.
Local Engines
Yelp (2004), named for the concept "Yellow Pages" began as an email service exchange recommending local business. Yelp now is connected to social networking sites and functions as a search engine, where users can access reviews for companies/restaurants/businesses under a specific search/product. Yelp recently announced that it is now powering the Microsoft Bing local search engine results.
Foursquare (2009) is a location-based social networking search engine for mobile devices, utilizing a GPS hardware system where users can search for restaurants/entertainment, etc in their immediate locale and connect with others in the area.