Search Engine

Imagine you are in a library and are looking for a particular book. Now if you have to go through every book in each category, it will be a tedious and difficult task. Moreover, if the library has more than a million books then this task seems next to impossible. You are definitely going to need a librarian who can bring the relevant books for you without any delay. Well, that’s where a search engine comes in.

Search engine spamming refers to the practice of creating Web pages, or sets of Web pages, designed to get a high relevance rank for some queries, even though the sites are not popular sites. Popularity ranking schemes such as PageRank make the job of search engine spamming more difficult, since just repeating words to get a high TF– IDF score was no longer sufficient. However, even these techniques can be spammed, by creating a collection of Web pages that point to each other, increasing their popularity rank. Techniques such as using sites instead of pages as the unit of ranking (with appropriately normalized jump probabilities) have been proposed to avoid some spamming techniques, but are not fully effective against other spamming techniques. The war between search engine spammers and search engines continues even today.

The hubs and authorities approach of the HITS algorithm is more susceptible to spamming. A spammer can create a Web page containing links to good authorities on a topic, and gains a high hub score as a result. In addition, the spammer’s Web page includes links to pages that they wish to popularize, which may not have any relevance to the topic. Because these linked pages are pointed to by a page with high hub score, they get a high but undeserved authority score.

What is a Search Engine?

A search engine is a software that brings to user relevant information(which they search) from the vast library of data available on World Wide Web. Users can search for multiple things including queries, documents, images, videos, webpages, and other content on a search engine. Search engines are build in such a way that they effectively generate the required information by crawling across the web and searching from the available databases on internet.

We all use search engine in our day to day life or should I say daily in our lives! I guess we all use Google a number of times in a day even to search basic things. Well Google is one of the most widely used search engine all around the world due to its variety of services like web search, image and video search, etc.

History of search engines

• Archie was the first well developed search engine in the year 1990. It used to search files by matching their names and indexing on FTP server.
• In 1992 Veronica search engine was developed for Gopher based websites.
• Later in 1993 W3Catalog and Aliweb were formed which were web search engines. WebCrawler search engine was the first to allow users search keywords.
• Finally in 1994, the search engine Yahoo! was developed which gained immense popularity. Earlier it was just a directory but in 1995 search feature was also added.
• In 1998 when Google was founded as till now is the most used and preferred browser all across the globe. However, there have been other frequently used search engines formed after google like, Baidu, Bing, Yandex, etc.

Working of a search engine

Lets understand how a search engine works through this chart:

If we look back to earlier example the search engine acts as a librarian that gathers relevant books which is required information from the library of data available on the internet.

To summarize, when user searches for a particular data the web crawlers scan or crawl through the data available on web and gather all the relevant information (Crawling). After this, the gathered information is organized in the form of catalog or database so that the relevant web pages can be selected quickly. The search engine then picks up the most relevant results according to the ranking and finally displays it in the results page or SERP. It is quite a technical process, but all this happens so quickly that user gets the results as soon as they search something on the search engine.

Architecture Of Search Engine

If we talk about the architecture or the framework of a search engine, it can be described in three main components –

• Web crawlers – As the name suggests these acts as spiders which crawl all over the web to collect required information. These are special bots that search throughout the internet and accumulates data using various links.

• Database – It is a collection of data which is gathered by the web crawlers after searching throughout the world wide web.

• Search Interface – It provides a medium or interface for users so that they can access and search on the database for required information.

How queries are processed in search engine?

Whenever we search anything on the search engine, it only takes a second or two for the output generation. However, a lot goes on in the backend. Indexing and Querying are two essential components behind the processing of a search engine. They are like the building blocks of search engines. Let’s take a look at these processes –

Indexing

• The indexing process begins with web crawling where the so-called spiders crawl across the world wide web and collect data.

• The data collected is stored in the form of a database for the process of indexing. This is also termed as text acquisition.

• Then the collected data is broken down into tokens or keywords. These tokens are used by the search engine in creating indexes. Each keyword is associated to a particular document and through indexing the data becomes organized and it helps the search engine to quickly retrieve a particular information.

Querying

• When a user searches something on the search engine a query input is generated.

• Then the search engine parses the generated query and searches at the indexes for the matching documents.

• Using a ranking algorithm, the search engine ranks the documents based on their relevance. Finally, the generated list is presented to the user with most relevant results on the top.

Search Engine Advantages:

• Search engines have made it possible to navigate through the internet. Even a person without any technical knowledge is able to use a search engine for solving a query.

• The quick and efficient responses of search engine make it easier for a user to immediately get result for their search.

• A search engine not only supports text results but also results like images, videos, maps, documents and various other formats, hence offering users a one stop solution.

• In today’s time people are using search engines not only for technical purposes but also for researches, educational purposes and even in day-to-day life because of its diversified result generation.

• The user-friendly interface, organized results, customization features and diversity of search results makes search engines one the most essential tool for surfing on internet.

Examples Of Popularly Used Search Engines

In today’s time Google is the most widely used search engine, however there are several other popular search engines being used. Some of the popular search engines are listed below:

• Google – Founded in 1998 by Larry Page and Sergey Brin, Google is the most popularly and widely used search engine. It has an attractive an user-friendly interface with a versatile library of features which makes it first choice for maximum users when it comes to search engines.

• Bing – Founded in 2009 by Microsoft, Bing is quite similar to other search engines. It also allows users to search through images.

• Yahoo – Founded in 1994 by Jerry Yang and David Filo, Yahoo was among the earliest used search engines. However, its popularity has declined over time. Earlier Yahoo offered a platform called “Yahoo answers” where users could ask or answer various questions.

• Baidu, DuckDuckGo and Yandex are some other popular emerging browsers in today’s time.