CrazyEngineers Logo

I wrote a bot and started collecting no. google of search results for Modi, AK, RaGa etc since May 4

[​IMG]​

site: #-Link-Snipped-#
source code: GitHub - avinassh/polistats: Google Search Trends of Indian Politics

What it is and How it works:

These two following graphs show the number of search results appeared on Google for certain keywords. If you search for 'Narendra Modi ' today you will see approximately 15 million search results. The graphs here show exactly that from May 4, 2014. I just wanted to compare the number search results for Narendra Modi, Arvind Kejriwal and Rahul Gandhi. These graphs also include search results for Sonia Gandhi, BJP, AAP and Congress India. One graph shows number of search results on main Google search site and other on Google News.

I wrote a bot which visits Google four times a day at 10am, 2pm, 6pm and 10pm. It searches for the keywords on both sites and stores the results in database. The bot is active since May 4, 2014. Bot is written in Python and MongoDB is used to store the data. An API end point is available to get all the data in JSON and it is powered by Tornado.

If you want to run on your own, installation instructions here:

1. Install required libraries from requirement.txt:

pip -r install requirements.txt
2. Run bot.py to start collecting data. If you want run the bot multiple times a day, use crontab:

$crontab -e
00 10,14,18,22 * * * /home/johnappleseed/polistats/bot.py
3. Run server.py to provide API end point

Replies

  • avii
    avii
    thats some good work. you can use google's search API (which has no official support and may be deprecated later, but that will take another year or so) instead of plain scraping.

    and also you can include paging so that only latest data is thrown and graph is drawn and then you keep receiving data in reverse chronological dates and keep drawing the graph behind. Why? because when user visits the page, he will see instantly see the graph.
  • avi2
    avi2
    thank you... yes, I am learning about paging already ;-)
  • Deepika Bansal
    Deepika Bansal
    Please elaborate the term bot for me.
  • Kaustubh Katdare
    Kaustubh Katdare
    Request: please add your project to CE Projects. It sure will inspire a few engineers. 😀
  • avii
    avii
    Deepika Bansal
    Please elaborate the term bot for me.
    #-Link-Snipped-#
  • avi2
    avi2
    Kaustubh Katdare
    Request: please add your project to CE Projects. It sure will inspire a few engineers. 😀
    Does it support version control?
  • Kaustubh Katdare
    Kaustubh Katdare
    avi2
    Does it support version control?
    You bet? 😀 It's been disabled as of now, though. I think we'll have to keep the project here itself for the time being.
  • avi2
    avi2
    Oh nice! I hope it's git server, can you give me credentials and server details?

You are reading an archived discussion.

Related Posts

Is it possible to track the mobile phone in Off mode?

Is there is any possible to track the mobile phone when it switched off.....?

Admissions for 2014 August Batch (CDAC, C-CAT)

I got 4240 rank in C_CAT. What i should expect? - PG-DAC - PG-DWIMC - PG-DSSD - PG-DITISS Also which college should i expect? Please tell me the probable colllege...

India To Have Its Own EMF Portal

To address the public outcry over Electromagnetic Field Pollution due to mobile towers, the government is considering to set up a national database which will contain the location of a...

CCAT August 2014 Batch

hi My rank is 1602 in DESD.... n around 1200+ ppl have already applied for those course Do i stand a chance of getting seat for the Above said course...

can any one please say guide for desd at cdac

Hello my name is vivek I have secured score like this as below ..Can anyone say can i get seat in cdac hyderabad / banglore / pune , if not...