One search engine for the whole internet is not a good idea anymore!

Image copyright Wikimedia Commons

We are living in an era where information overload is beyond comprehension. If you use one search engine for the whole internet it won’t work most of the time. Be it information, news or product, job. We want to build one search engine for every single country to overcome this ordeal.

The enormous scale of the today’s data is beyond comprehension. And it’s growing exponentially every day. And it’s getting worse everyday. Now a days every country alone generates a large volume of data everyday. The World Wide Web has passed a long journey. It’s not only about the global information we seek. It has become a part and parcel of our everyday life. We do use it in most of modern day services.

Search is not a Q/A

Information we seek in internet are of mainly two types:

• We know what we are looking for
• We don’t know what we are looking for

Either way traditional search engines behave to get you a list of possible items with almost no way to drill it down.

One search engine for the whole internet will make you to search multiple times to get what you want. The idea behind it is you ask a question and it will show you a possible list of items. That’s it!

There are others (amazon, linkedin, netflix) who would get you filters but those are focused on one type of information. also called vertical search engine

• Search within results
• Facet
• Tag based taxonomy
• Range queries

These features could become very handy in this case. Search is more like a journey to seek what you are looking for. It’s an experience. We believe the search engine should guide you through the journey. It might ask you questions while doing it and help you act accordingly. It should also recommend you items based on your interaction.

Systems who restrict this information overflow are becoming popular

If we look into the recent popular systems it’s prevalent that people are tired of this information overload. That’s why systems like twitter, vine, periscope, snapchat became so popular. It’s the time to remove the clutters in search engine.

Not everyone speaks English

If we use one search engine for the whole internet it gotta be in english. There is no doubt about that. But there are at least 500 languages in the world. and Everyone adore theirs’. There is one country who has even fought a bloody war to speak their minds in their own language. Bangladesh it is! Almost everything of black and white is in Bangla for them. We know there are others as well who does the same.

There are new technologies which could play a vital role in Information Retrieval systems

Since we decided to come up with the idea of SEO instead of decentralising it there has been a lot of new algorithms, design patterns immersed. Big players seems not interested in those. We decided to embrace it. We integrated CLIR Cross Language Information Retrieval. It means when you search for an english word it would also get you items with it’s meaning and vice versa. If you want to see it in action please visit

One search engine can never be realtime

Every country is generating a lot of data everyday. Using one search engine could never be realtime. You will notice that the search results are horribly outdated. I won’t blame anybody. It really takes time to crawl a whole website and get back to it for a recrawl.

Important information also served in local outlets

The world has become smaller in terms of communication. We know the important stuff around the world instantly as it happens. This made it obvious that the same information is also published in native languages. There are many people who would understand better if it were in their mother tongue.

There was a time when Yahoo was the most important company in the internet

Sooner or later the internet will have to be decentralised. It already started happening. We started with khujun.com a vertical search engine, specifically made for Bangladesh. We crawl only a specific list of websites relevant to this country. People are loving it. We are working to release it in Saudi Arabia, Nigeria and Mexico.

User should be in control and see everything in result

For instance the website crawl metadata and result scoring should be displayed to the user. Also we think if it has an image to represent the content it should be shown as a thumbnail. Which we do. We are also let the user sort the results based on the data.

There are so many valid results for a query

Let’s say we are looking for the price offerings of iPhone among the local ecommerce websites. As of now we can use Google’s country specific search. and this is what I got (notice url country specific tld in screenshot):

Every country has a mammoth collection of valid results for your query. The idea of one search engine will end up with reverse engineering the search algorithm behind it. While we could have decentralised it in the very first time. It is a bad idea to use one search engine for the whole internet.

PS: I have tremendous respect to Google for what they did and doing. Information on this post is only my opinion. No pun intended. If you have any thoughts about this post I would love to converse with you about it. You can reach me at twitter. My handle is @ferdous