CrazyEngineers
CrazyEngineers
  • [​IMG]​

    site: #-Link-Snipped-#
    source code: GitHub - avinassh/polistats: Google Search Trends of Indian Politics

    What it is and How it works:

    These two following graphs show the number of search results appeared on Google for certain keywords. If you search for 'Narendra Modi ' today you will see approximately 15 million search results. The graphs here show exactly that from May 4, 2014. I just wanted to compare the number search results for Narendra Modi, Arvind Kejriwal and Rahul Gandhi. These graphs also include search results for Sonia Gandhi, BJP, AAP and Congress India. One graph shows number of search results on main Google search site and other on Google News.

    I wrote a bot which visits Google four times a day at 10am, 2pm, 6pm and 10pm. It searches for the keywords on both sites and stores the results in database. The bot is active since May 4, 2014. Bot is written in Python and MongoDB is used to store the data. An API end point is available to get all the data in JSON and it is powered by Tornado.

    If you want to run on your own, installation instructions here:

    1. Install required libraries from requirement.txt:

    pip -r install requirements.txt
    2. Run bot.py to start collecting data. If you want run the bot multiple times a day, use crontab:

    $crontab -e
00 10,14,18,22 * * * /home/johnappleseed/polistats/bot.py
    3. Run server.py to provide API end point
     Replies
Howdy guest!
Dear guest, you must be logged-in to participate on CrazyEngineers. We would love to have you as a member of our community. Consider creating an account or login.
Replies

  • avii

    Member • Jun 25, 2014

    thats some good work. you can use google's search API (which has no official support and may be deprecated later, but that will take another year or so) instead of plain scraping.

    and also you can include paging so that only latest data is thrown and graph is drawn and then you keep receiving data in reverse chronological dates and keep drawing the graph behind. Why? because when user visits the page, he will see instantly see the graph.
    Are you sure? This action cannot be undone.
    Cancel

  • avi2

    Member • Jun 25, 2014

    thank you... yes, I am learning about paging already ;-)
    Are you sure? This action cannot be undone.
    Cancel

  • Deepika Bansal

    Member • Jul 1, 2014

    Please elaborate the term bot for me.
    Are you sure? This action cannot be undone.
    Cancel

  • Kaustubh Katdare

    Administrator • Jul 1, 2014

    Request: please add your project to CE Projects. It sure will inspire a few engineers. 😀
    Are you sure? This action cannot be undone.
    Cancel

  • avii

    Member • Jul 2, 2014

    Deepika Bansal
    Please elaborate the term bot for me.
    #-Link-Snipped-#
    Are you sure? This action cannot be undone.
    Cancel

  • avi2

    Member • Sep 9, 2014

    Kaustubh Katdare
    Request: please add your project to CE Projects. It sure will inspire a few engineers. 😀
    Does it support version control?
    Are you sure? This action cannot be undone.
    Cancel

  • Kaustubh Katdare

    Administrator • Sep 9, 2014

    avi2
    Does it support version control?
    You bet? 😀 It's been disabled as of now, though. I think we'll have to keep the project here itself for the time being.
    Are you sure? This action cannot be undone.
    Cancel

  • avi2

    Member • Sep 10, 2014

    Oh nice! I hope it's git server, can you give me credentials and server details?
    Are you sure? This action cannot be undone.
    Cancel
Home Channels Search Login Register