Skip to main content

Search engine and indexing techniques

A search engine is an information retrieval system designed to help find information stored on a on the World Wide Web.

Search engine indexing collects, parses, and stores data to facilitate fast and accurate information retrieval. Alternatively web indexing is the process of search engines that are designed to find web pages on the Internet.

Index design integrates interdisciplinary concepts from linguistics, Cognitive Psychology, Mathematics, Informatics, and Computer Science. Popular engines focus on the full-text indexing of online, natural language documents. The searchable media types are text, video, audio, and graphics etc. Search engines helps to minimize the time required to find information and the amount of information which must be consulted to other techniques for managing information overload.

Information retrieval (IR) is the science of searching for relevant information, searching for documents themselves, and also searching for metadata that describe data, and for databases of texts, images or sounds from a huge collection of information resources

An IR system is a software that provide access to books, journals and other documents, stores them and manages the document. Searches can be based on full-text or other content-based indexing. Automated information retrieval systems are used to reduce what has been called information overload.  A web search engine is a software system that is designed to search for information on the World Wide Web. The search results are generally presented in a lines of results, often referred to as search engine result page (SERPs) commonly called hits The information may be a mix of webpages, images and other types of files. Search engines also maintain a real time information by running an algorithm on a web crawler.

Web crawler, sometimes called a spiderbot or simply spider and often shortened to crawler, is an internet bot that systematically browses the World Wide Web, typically for the purpose of Web indexing. Web crawlers copy pages for processing by a search engine which indexes the downloaded pages so users can search more efficiently. Crawlers can validate hyperlinks and HTML code.

Internet content that is not capable of being searched by a web search engine is generally described as the deep web.

One of Computer scientist, Michael K. Bergman coined the term deep web in 2001 as a search indexing term. He defined deep web as invisible web, or hidden web are parts of the World Wide Web whose contents are not  indexed by standard web search engines. The content of the deep web is hidden behind HTTP forms and includes many very common uses such as web mail, online banking, and services that users must pay for, and which is protected by a paywall. So deep web content can be located and accessed by a direct URL or IP address, and may require password or other security access.

List of Top 10 Most Popular Search Engines In the World (Updated 2018)
1. Google: The search engine giant holds the first place in search with a stunning difference of 66% from second in place Bing. According to the latest netmarketshare report (January 2018) 74.52% of searches were powered by Google and only 7.98% by Bing. Google is also dominating the mobile/tablet search engine market share with 93%!
2. Bing: Bing is Microsoft’s attempt to challenge Google in the area of search, but despite their efforts they still did not manage to convince users that their search engine can produce better results than Google
3. Yahoo:Since October 2011 Yahoo search is powered by Bing. Yahoo is still the most popular email provider and according to reports holds the fourth place in search.
4. Ask.com:Formerly known as Ask Jeeves, Ask.com receives approximately 0.05% of the search share. ASK is based on a question/answer format where most questions are answered by other users or are in the form of polls. It also has the general search functionality but the results returned lack quality compared to Google or even Bing and Yahoo.
5. AOL.com:According to netmarketshare the old time famous AOL is still in the top 10 search engines with a market share that is close to 0.04%. The AOL network includes many popular web sites like engadget.com, techchrunch.com and the huffingtonpost.com.
6. Baidu: Baidu was founded in 2000 and it is the most popular search engine in China. It’s market share is increasing steadily and according to Wikipedia, Baidu is serving billion of search queries per month. It is currently ranked at position 4, in the Alexa Rankings.
7. Wolframalpha:Wolframalpha is different that all the other search engines. They market it as a Computational Knowledge Engine which can give you facts and data for a number of topics. It can do all sorts of calculations, for example if you enter “mortgage 2000” as input it will calculate your loan amount, interest paid etc. based on a number of assumptions.
8. DuckDuckGo: Has a number of advantages over the other search engines. It has a clean interface, it does not track users, it is not fully loaded with ads and has a number of very nice features (only one page of results, you can search directly other web sites etc). According to duckduckgo traffic stats, as of October 2018, duckduckgo is serving more than 30 million searches per day.
9. Internet Archive:archive.org is the internet archive search engine. You can use it to find out how a web site looked since 1996. It is very useful tool if you want to trace the history of a domain and examine how it has changed over the years.
10. Yandex.ru:According to Alexa, Yandex.ru is among the 30 most popular websites on the Internet with a ranking position of 4 in Russian.
Yandex present themselves as a technology company that builds intelligent products and services powered by machine learning. According to Wikipedia, Yandex operates the largest search engine in Russia with about 65% market share in that country.
Finally, if you search What is the best search engine? in any of the search engines listed above, you will get an answer that Google is the best and most popular search engine and Bing is in the second place (on a Global level).

David

David

My name is Dawit Uta (David). I am Computer Science Professional. I am interested on research & project work, website designing and development, teaching of any programming languages at any level, helping and encouraging the students, tutoring, consulting about tech. My research areas are Artificial Intelligence (AI), Machine Learning (ML), Deep Learning (DL), Internet of Things (IoT), Data Science and BigData Technologies.

241 thoughts to “Search engine and indexing techniques”

  1. Pingback: canadian viagra
  2. Pingback: viagra by mail
  3. Pingback: cialis for sale
  4. Pingback: cialis and alcohol
  5. Hello I am soo happy I found your webpage, I really found you by error, while I was researching on Aol for something else, Anyhow I am
    here now and would just like to say thanks for a incredible post andd a all round entertaining blog
    (I also love tthe theme/design), I don’t have time to read it all
    at the minute but I have saved it and also added inn your RSS feeds, so when I have ime I will
    bee back to read a great deal more, Please do keewp up the superb job.

    Visit my web paage … Taxi Leiden

  6. Do you mind if I quote a couple of your articles as long as I provide credit and sources back to your webpage?
    My blog site is in tthe exact same niche as yours aand my visitors
    would treuly benefit from a lot of the information you present here.
    Please let me know if this ok wiith you. Thanks a lot!

    Check ouut my website – Romeo

  7. I just like the helpful information you supply in your articles.
    I’ll bookmark your blog and check once more right here frequently.
    I’m somewhat sure I will learn a lot of new stuff right
    here! Good luck for the next!

  8. You actually make it appear really easy with your presentation however
    I in finding this matter to be actually one thing which I believe I might never understand.
    It seems too complicated and extremely extensive for me.
    I’m having a look forward on your next submit, I will try to get the cling of
    it!

  9. Howdy terrific blog! Does running a blog like this take a great
    deal of work? I have noo knowledge oof computer programming but I had
    been hoping to start my own blog iin the near future.
    Anyhow, should you have any recommendations or techniques for new
    blog owners please share. I know this is off topoic
    nevertheless I just needed to ask. Thanks!
    https://cenniesist.medium.com/act-essay-321852981d35
    best paper writers
    best paper writers https://medium.com/@whizztrantio/best-college-application-essay-ever-af070c159a8c

  10. I have been exploring for a little for any high-quality articles or weblog
    posts in this kind of area . Exploring in Yahoo I at last stumbled upon this web site.
    Studying this info So i am glad to convey that I have a very just right uncanny feeling I came
    upon exactly what I needed. I such a lot undoubtedly will make certain to do not overlook this web
    site and provides it a look on a relentless basis.

  11. Unquestionably consider that which you said. Your favourite justification appeared
    to be at the net the simplest factor too bee aware of.
    I say to you, I certainly get annpyed while people consider worries that
    they just do not understand about. Yoou managed to hit thee nail upon the
    hghest as smartly as outlined out the whole thing without having side effect
    , people can take a signal. Will likely be again to get more.
    Thank you

    Feel freee to surf to my pave … Kbcjiolotterywinner.com

  12. I actually wanted to type a simple comment to appreciate you for all the remarkable tricks
    you are sharing at this website. My extensive internet look
    up has at the end been paid with beneficial strategies to
    share with my close friends. I ‘d assert that most of us visitors are undoubtedly endowed
    to exist in a fine network with very many awesome
    people with very beneficial hints. I feel somewhat fortunate to have
    encountered the website page and look forward to really more
    enjoyable times reading here. Thanks again for a lot of things.

    my page Peak Summit Keto Review

  13. Great goods from you, man. I have understand your stuff previous to
    and you are just too magnificent. I actually like what you have
    acquired here, certainly like what you’re
    saying and the way in which you say it. You make it entertaining and you still care for to keep it smart.
    I cant wait to read far more from you. This is actually a terrific web site.

    Here is my web site BioPure Derma Serum

  14. I was just searching for this info for some time. After 6 hours of
    continuous Googleing, finally I got it in your site. I wonder what’s the lack of Google strategy that do
    not rank this kind of informative sites in top of the list.
    Usually the top sites are full of garbage.

    Here is my web-site; Simple Keto Burns

  15. Hey! I know this is kinda off topic however , I’d figured I’d ask.

    Would you be interested in exchanging links or maybe guest writing a blog article or vice-versa?
    My website discusses a lot of the same topics as yours and
    I think we could greatly benefit from each other. If you happen to be interested feel free to shoot me
    an e-mail. I look forward to hearing from you! Superb blog by
    the way!

  16. Good – I should certainly pronounce, impressed with your site.
    I had no trouble navigating through all the tabs
    and related info ended up being truly simple to do to access.
    I recently found what I hoped for before you know it in the least.

    Reasonably unusual. Is likely to appreciate it for those who add forums or anything,
    web site theme . a tones way for your customer to communicate.
    Excellent task.

    my blog post … Peace CBD Oil Price

  17. Great – I should definitely pronounce, impressed with your site.
    I had no trouble navigating through all the tabs and related
    info ended up being truly simple to do to access. I recently
    found what I hoped for before you know it in the least.
    Quite unusual. Is likely to appreciate it for those who add forums or something, web
    site theme . a tones way for your customer to communicate.
    Nice task.

    My site: Azure Keto

  18. I know this if off topic but I’m looking into starting my own weblog
    and was wondering what all is needed to get setup? I’m assuming having a blog like yours would
    cost a pretty penny? I’m not very internet smart so
    I’m not 100% certain. Any tips or advice would be greatly appreciated.
    Many thanks

    Here is my blog post: https://geegram.net/blog/63109/smoking-marijuana-and-using-positive-affirmations/

  19. Thank you for sharing superb informations. Your site is
    so cool. I’m impressed by the details that you have on this blog.
    It reveals how nicely you understand this subject.
    Bookmarked this web page, will come back for more articles.
    You, my pal, ROCK! I found just the information I already searched everywhere
    and just couldn’t come across. What a great website.

    Here is my page; https://jalandhar.indiaolx.com/user/profile/29398

  20. Hi would you mind stating which blog platform you’re working with?
    I’m planning to start my own blog in the near future
    but I’m having a hard time selecting between BlogEngine/Wordpress/B2evolution and Drupal.
    The reason I ask is because your design seems
    different then most blogs and I’m looking for something completely unique.
    P.S Apologies for getting off-topic but I had to ask!

    Have a look at my website http://joellwilliams.qhub.com/member/2022387

  21. My brother recommended I may like this blog. He used to be totally right.
    This publish actually made my day. You cann’t believe just how much time I had spent for this info!
    Thank you!

  22. I blog frequently and I truly appreciate your information. Your article has really
    peaked my interest. I will book mark your blog and keep
    checking for new details about once a week. I subscribed
    to your Feed too.

  23. I still can not quite believe I could always be one of those reading the important
    points found on your web blog. My family and I are really thankful for
    the generosity and for giving me the potential to pursue the chosen profession path.
    Many thanks for the important information I managed to get from your web page.

    Also visit my web-site … https://blog.mylighthost.com/2020/11/08/are-raise-by-itself-diets-really-the-best/

  24. Great – I should certainly pronounce, impressed with your website.

    I had no trouble navigating through all the tabs and related info ended up being truly easy to do to access.
    I recently found what I hoped for before you know it in the least.

    Quite unusual. Is likely to appreciate it for those who add forums or anything, web site theme .

    a tones way for your client to communicate. Nice task.

    Also visit my blog … http://infiniterestoreskincare.com/

  25. I tend not to leave many responses, but
    I read a few of the responses on this page Search engine and indexing techniques » Welcome to davidtechnotips.com.
    I do have 2 questions for you if it’s okay. Is it just me or do a few of
    the remarks appear like they are coming from brain dead
    folks? 😛 And, if you are writing at other online social sites, I’d like to follow you.
    Would you make a list of the complete urls of your social
    networking sites like your Facebook page, twitter feed, or linkedin profile?

    Also visit my blog post: Keto VIP Reviews

  26. I would like to thank you for the efforts you have put in writing this blog.
    I’m hoping the same high-grade website post from you in the upcoming as well.
    In fact your creative writing skills has inspired me to get my own website now.
    Actually the blogging is spreading its wings quickly.
    Your write up is a good example of it.

    My webpage; http://flowgenix.net/

  27. It’s in reality a nice and useful piece of information. I am happy that you just
    shared this useful info with us. Please keep us informed
    like this. Thanks for sharing.

    Feel free to visit my homepage; Back

  28. We are a bunch of volunteers and starting a brand new scheme in our community.

    Your website offered us with useful info to work on. You have performed a formidable activity and our
    whole neighborhood will probably be thankful to
    you.

    Stop by my blog – xml

  29. I would like to voice my gratitude for your generosity in support
    of visitors who must have guidance on this one concern. Your personal commitment to passing the solution up and down had become astonishingly informative
    and have continually permitted ladies much like me to reach their desired goals.
    Your entire useful recommendations indicates a great deal to me and somewhat more
    to my fellow workers. Thanks a lot; from everyone of us.

    Here is my blog post :: Enhanced Keto X

  30. I have been browsing online more than 4 hours today, yet I never found any interesting article like yours.
    It’s pretty worth enough for me. In my view, if all webmasters and bloggers
    made good content as you did, the net will be a lot more useful than ever before.

    my homepage [ПЕРЕЙТИ]

  31. Hiya, I am really glad I have found this info. Nowadays bloggers
    publish only about gossips and web and this is actually frustrating.

    A good site with exciting content, that is what I need.
    Thank you for keeping this website, I will be visiting
    it. Do you do newsletters? Can’t find it.

    Also visit my page http://Trainingteachers.org.za/groups/how-accomplish-weight-and-make-mass-obtaining-fat-1674549298/

  32. Great – I should definitely pronounce, impressed with your site.
    I had no trouble navigating through all the tabs and related info ended up being truly simple to do
    to access. I recently found what I hoped for before you know it in the
    least. Quite unusual. Is likely to appreciate
    it for those who add forums or anything, web site theme
    . a tones way for your customer to communicate. Nice task.

    Feel free to visit my homepage :: Infinite Restore Skin Care

  33. I know this if off topic but I’m looking into starting my own blog and was curious what all is required to get setup?
    I’m assuming having a blog like yours would cost a pretty penny?
    I’m not very internet smart so I’m not 100% certain. Any recommendations or advice would
    be greatly appreciated. Thank you

    Review my website – Natural Vibration CBD Oil

  34. My spouse and i felt peaceful when Peter could finish up his reports by way of the ideas he obtained using
    your web page. It is now and again perplexing to just possibly be giving
    for free instructions which often people today might have
    been selling. And we all discover we need you to thank because of that.
    The main illustrations you have made, the simple website navigation, the relationships you can make it
    possible to instill – it’s mostly terrific, and it’s
    really facilitating our son in addition to the family
    reckon that the matter is excellent, and that’s incredibly serious.
    Thanks for all!

    Also visit my webpage … [ПЕРЕЙТИ]

  35. I seldom comment, however i did a few searching and wound up here Search engine and
    indexing techniques » Welcome to davidtechnotips.com.
    And I actually do have a couple of questions for
    you if it’s allright. Is it simply me or does it look as if like a
    few of the responses appear like they are left by brain dead
    individuals? 😛 And, if you are posting on additional online sites, I would
    like to follow anything new you have to post.
    Could you make a list of every one of your shared pages like your twitter feed, Facebook page or linkedin profile?

    Have a look at my web blog; Fast Flow Male Enhancement Reviews

  36. I just like the valuable information you provide in your articles.
    I’ll bookmark your weblog and take a look at once more right here frequently.
    I am somewhat sure I’ll be informed lots of new stuff proper
    here! Good luck for the following!

    Check out my web site … Azure Keto

  37. I want to get across my admiration for your kind-heartedness
    giving support to those people who must have guidance on this important theme.
    Your real dedication to getting the message all-around
    had become remarkably informative and has constantly allowed others like me to achieve
    their goals. This insightful information indicates a whole
    lot to me and substantially more to my colleagues.
    Regards; from everyone of us.

    Visit my site: Organic Green CBD

  38. Needed to put you that bit of observation so as to give many thanks as before relating to the unique methods
    you have shared in this article. This has been simply generous with people
    like you to offer extensively precisely what numerous people would have made
    available for an e book to end up making some dough on their own, and in particular considering that you could possibly have done it in case
    you desired. The advice as well acted as a great way to recognize that
    many people have similar eagerness really like my personal own to know somewhat
    more in regard to this matter. I am certain there are millions of more fun times up
    front for many who read through your blog.

    Here is my webpage :: http://www.quickregister.us/classifieds/user/profile/221911

  39. I usually do not drop many remarks, but i did a few searching and wound up here Search engine and indexing
    techniques » Welcome to davidtechnotips.com.
    And I do have 2 questions for you if it’s allright. Is it simply me or does it give the impression like some of
    the remarks come across like they are left by brain dead visitors?
    😛 And, if you are posting at other places, I would like
    to follow everything fresh you have to post. Would you list of the
    complete urls of your community sites like your twitter feed,
    Facebook page or linkedin profile?

    Here is my homepage :: https://oathbreakermtg.org/forums/users/mvrsaul419903111/

  40. I was honored to obtain a call coming from a friend as soon as he found out the important ideas shared
    on the site. Examining your blog post is a real amazing
    experience. Thank you for thinking about readers
    much like me, and I desire for you the best of achievements like a professional
    in this field.

    Here is my homepage; https://www.groovelineentertainment.com/blog/6648/why-did-i-received-this-room-again-10-signs-your-memory-is-failing/

  41. hey there and thank you for your information ? I’ve certainly picked up anything new from right here.

    I did however expertise some technical issues using this web site, since I experienced to reload the website
    a lot of times previous to I could get it to load correctly.
    I had been wondering if your hosting is OK? Not that I am complaining,
    but sluggish loading instances times will sometimes affect your placement in google and could damage your high-quality score if ads
    and marketing with Adwords. Anyway I’m adding this RSS to my email and can look out for much
    more of your respective intriguing content. Ensure that you update
    this again very soon.

    My web site … Dabin Earbuds Review

  42. First of all I would like to say awesome blog! I had a quick question in which I’d like to ask
    if you do not mind. I was interested to find out
    how you center yourself and clear your mind prior to writing.
    I’ve had trouble clearing my mind in getting my thoughts out there.
    I truly do enjoy writing but it just seems like the first 10
    to 15 minutes tend to be lost just trying to figure out
    how to begin. Any suggestions or tips? Thanks!

    Also visit my web page – https://blog.mylighthost.com/2020/11/11/6-methods-to-accelerate-pounds-reduction-and-drop-pounds/

  43. It’s a pity you don’t have a donate button! I’d most certainly donate to this brilliant blog!
    I guess for now i’ll settle for bookmarking and
    adding your RSS feed to my Google account. I look forward to fresh
    updates and will share this blog with my Facebook group.
    Chat soon!

    My web blog – Her

  44. To follow up on the update of this matter on your website and would
    wish to let you know simply how much I appreciated the time you took to write this beneficial post.
    Inside the post, you spoke of how to truly
    handle this issue with all convenience. It would be my own pleasure to collect some more ideas from your blog and come as much as offer others what I learned from you.
    I appreciate your usual fantastic effort.

    my blog post … http://agilityfitketo.net/

  45. Its like you learn my mind! You seem to grasp a lot about this, such as you wrote the ebook in it or something.
    I feel that you can do with a few % to force the message house a little bit,
    but other than that, this is magnificent blog. A fantastic read.
    I will certainly be back.

    Also visit my site :: 直接访问

  46. Excellent pieces. Keep posting such kind of information on your blog.
    Im really impressed by it.[X-N-E-W-L-I-N-S-P-I-N-X]Hello there, You have performed an incredible job.

    I will definitely digg it and individually recommend to my friends.

    I am confident they’ll be benefited from this site.

    Feel free to surf to my homepage … http://www.usafreeclassifieds.org/classifieds/user/profile/253087

  47. Just to follow up on the update of this issue on your web page and would like to let you know just how much I
    loved the time you took to generate this helpful post. Within the post, you
    spoke regarding how to actually handle this thing with all comfort.
    It would be my own pleasure to collect some more ideas from your blog and come up to offer
    other folks what I discovered from you. Many thanks for your usual terrific effort.

    my web page aqui

  48. I would like to use the opportunity of saying thanks to you for that
    professional suggestions I have enjoyed viewing your site.
    I am looking forward to the actual commencement
    of my university research and the overall preparation would never have been complete without consulting this site.
    If I can be of any assistance to others, I would be pleased to help via what I have discovered from here.

    Here is my blog post: https://medsbiotechcbd.org/

  49. Good day! I could have sworn I’ve visited this site before but after looking at
    some of the posts I realized it’s new to me. Anyhow, I’m certainly pleased I found it and I’ll be book-marking it and checking back often!

    Here is my blog … DominXT Testo

  50. I have learn some good stuff here. Certainly price bookmarking for revisiting.

    I wonder how so much effort you set to create this type of wonderful informative web site.

  51. Hello there! This post couldn’t be written much better!
    Looking through this article reminds me of my previous roommate!
    He always kept preaching about this. I am going to send this information to him.
    Fairly certain he’ll have a good read. Thank you for sharing!

  52. Hi there!

    I’m Dan, Managing Director of https://www.track-r.net which is the fastest growing Google rank tracker. It shows exactly where you’re placed for your target keywords in Google.

    By knowing and understanding where your site is placed, you can figure out a marketing plan to get higher up the rankings. We have a decade of experience in the online marketing industry, and can help advise (for free) and you can have our free SEO template email series which will definitely help your business grow.

    Use https://www.track-r.net and get 2 free weeks to try it out. If you like it, keep going for the lowest package or if you don’t please let me know what you think about it.

    I also run a competition to get a free 12 month license, which will be explained to you once you register.

    Thanks,
    Dan

  53. Attractive section of content. I just stumbled upon your
    website and in accession capital to assert that I acquire actually enjoyed account your blog posts.
    Anyway I will be subscribing to your feeds and even I achievement you
    access consistently quickly.

  54. I do believe all of the concepts you have presented on your post.
    They’re really convincing and will definitely work.
    Still, the posts are too brief for novices. May you please extend them a bit from next
    time? Thanks for the post.

  55. I do not even know the way I ended up here, but I thought this post was good.
    I do not recognise who you’re but definitely you’re going to a famous blogger if
    you happen to aren’t already. Cheers!

Leave a Reply

Your email address will not be published. Required fields are marked *