About Google Web Crawler

hi guys.......
google downloads the whole of the web thru crawlers.... if i want to download a website to my pc wat software do u think i can use........
do reply.......๐Ÿ˜€

Replies

  • zia.sepsis
    zia.sepsis
    please explain your query clearly. so that people out here can help you.
  • Kaustubh Katdare
    Kaustubh Katdare
    bharath muthuku
    hi guys.......
    google downloads the whole of the web thru crawlers.... if i want to download a website to my pc wat software do u think i can use........
    do reply.......๐Ÿ˜€
    I think I got what the question is.

    First of all, Google does not download the whole web. It simply lists all the information on it's servers through its complicated Page Rank algorithm.

    What Google does is this :

    Google sends it web crawler (web spider) to crawl the website. While crawling, the crawler picks up relevant keywords and information and sends them to Google's servers. Google's servers then analyze the keywords and decide the rank of the page from which they were picked up. As people search for the keywords on Google's search engine, Google lists them on its search results pages according to their ranks.

    If you want to do the same; you will have to write your own web crawler. If you know PERL, you may do that. But indexing the whole of web will be too big for your computer ๐Ÿ˜‰ . Google owns server frams that are twice as big as football ground! ๐Ÿ˜
  • MaRo
    MaRo
    - Crawlers don't download the web, they just make indexes for them..like index of a book the index doesn't contain the whole book

    - You want to view websites while no connection here you are HTTrack Website Copier - Free Software Offline Browser (GNU GPL)

    - You want to make a crawler go learn how to ask first.
  • ms_cs
    ms_cs
    Very Interesting information about Web Crawler...

You are reading an archived discussion.

Related Posts

Want to do internship abroad.. I want to do internship abroad. Please suggest me how can I get that opportunity. I have already done some project work on various topics...
Can someone tell me what is the difference between a pen drive and a flash drive? Also is USB drive the same thing?
What exactly is code rendering? Can anyone explain in detail? Thanks
I got this mail in my inbox. If someone can answer - hiii i am biswaranjan panda doing b-tech in N I S T berhampur i want to do summer...
A two-storey building on a highly reactive site is founded on a grid of piles. The soil profile shows a 4m deep reactive clay horizon overlying a deep profile of...