1. Home >
  2. Apps >
  3. Groups >

I wrote a bot and started collecting no. google of search results for Modi, AK, RaGa etc since May 4

Question asked by avi2 in #Coffee Room on Jun 25, 2014
avi2
avi2 · Jun 25, 2014
Rank D2 - MASTER
[​IMG]

site: https://avinassh.github.io/polistats
source code: https://github.com/avinassh/polistats

What it is and How it works:

These two following graphs show the number of search results appeared on Google for certain keywords. If you search for 'Narendra Modi ' today you will see approximately 15 million search results. The graphs here show exactly that from May 4, 2014. I just wanted to compare the number search results for Narendra Modi, Arvind Kejriwal and Rahul Gandhi. These graphs also include search results for Sonia Gandhi, BJP, AAP and Congress India. One graph shows number of search results on main Google search site and other on Google News.

I wrote a bot which visits Google four times a day at 10am, 2pm, 6pm and 10pm. It searches for the keywords on both sites and stores the results in database. The bot is active since May 4, 2014. Bot is written in Python and MongoDB is used to store the data. An API end point is available to get all the data in JSON and it is powered by Tornado.

If you want to run on your own, installation instructions here:

1. Install required libraries from requirement.txt:

pip -r install requirements.txt
2. Run bot.py to start collecting data. If you want run the bot multiple times a day, use crontab:

$crontab -e
00 10,14,18,22 * * * /home/johnappleseed/polistats/bot.py
3. Run server.py to provide API end point Posted in: #Coffee Room
avii
avii · Jun 25, 2014
Rank B2 - LEADER
thats some good work. you can use google's search API (which has no official support and may be deprecated later, but that will take another year or so) instead of plain scraping.

and also you can include paging so that only latest data is thrown and graph is drawn and then you keep receiving data in reverse chronological dates and keep drawing the graph behind. Why? because when user visits the page, he will see instantly see the graph.
avi2
avi2 · Jun 26, 2014
Rank D2 - MASTER
thank you... yes, I am learning about paging already ;-)
Deepika Bansal
Deepika Bansal · Jul 1, 2014
Rank B2 - LEADER
Please elaborate the term bot for me.
Kaustubh Katdare
Kaustubh Katdare · Jul 1, 2014
Rank A1 - PRO
Request: please add your project to CE Projects. It sure will inspire a few engineers. 😀
avii
avii · Jul 2, 2014
Rank B2 - LEADER
Deepika Bansal
Please elaborate the term bot for me.
https://bit.ly/Ii3WkY
avi2
avi2 · Sep 10, 2014
Rank D2 - MASTER
Kaustubh Katdare
Request: please add your project to CE Projects. It sure will inspire a few engineers. 😀
Does it support version control?
Kaustubh Katdare
Kaustubh Katdare · Sep 10, 2014
Rank A1 - PRO
avi2
Does it support version control?
You bet? 😀 It's been disabled as of now, though. I think we'll have to keep the project here itself for the time being.
avi2
avi2 · Sep 10, 2014
Rank D2 - MASTER
Oh nice! I hope it's git server, can you give me credentials and server details?

You must log-in or sign-up to reply to this post.

Click to Log-In or Sign-Up