I wrote a bot and started collecting no. google of search results for Modi, AK, RaGa etc since May 4

@avi2-gV81vT • Oct 26, 2024

Oct 26, 2024

1.8K

site: #-Link-Snipped-#
source code: <a href="https://github.com/avinassh/polistats" target="_blank" rel="nofollow noopener noreferrer">GitHub - avinassh/polistats: Google Search Trends of Indian Politics</a>

What it is and How it works:

These two following graphs show the number of search results appeared on Google for certain keywords. If you search for 'Narendra Modi ' today you will see approximately 15 million search results. The graphs here show exactly that from May 4, 2014. I just wanted to compare the number search results for Narendra Modi, Arvind Kejriwal and Rahul Gandhi. These graphs also include search results for Sonia Gandhi, BJP, AAP and Congress India. One graph shows number of search results on main Google search site and other on Google News.

I wrote a bot which visits Google four times a day at 10am, 2pm, 6pm and 10pm. It searches for the keywords on both sites and stores the results in database. The bot is active since May 4, 2014. Bot is written in Python and MongoDB is used to store the data. An API end point is available to get all the data in JSON and it is powered by Tornado.

If you want to run on your own, installation instructions here:

1. Install required libraries from requirement.txt:

pip -r install requirements.txt

2. Run bot.py to start collecting data. If you want run the bot multiple times a day, use crontab:

$crontab -e
00 10,14,18,22 * * * /home/johnappleseed/polistats/bot.py

3. Run server.py to provide API end point

Like 0 Replies 8

Replies

Welcome, guest

Join CrazyEngineers to reply, ask questions, and participate in conversations.

CrazyEngineers powered by Jatra Community Platform

avii

@avii-TGGs8o • Jun 25, 2014

thats some good work. you can use google's search API (which has no official support and may be deprecated later, but that will take another year or so) instead of plain scraping.

and also you can include paging so that only latest data is thrown and graph is drawn and then you keep receiving data in reverse chronological dates and keep drawing the graph behind. Why? because when user visits the page, he will see instantly see the graph.
0
avi2

@avi2-gV81vT • Jun 25, 2014

thank you... yes, I am learning about paging already ;-)
0
Deepika Bansal

@deepika-jf1ysv • Jul 1, 2014

Please elaborate the term bot for me.
0
Kaustubh Katdare

@thebigk • Jul 1, 2014

Request: please add your project to CE Projects. It sure will inspire a few engineers. 😀
0
avii

@avii-TGGs8o • Jul 2, 2014

Deepika Bansal
Please elaborate the term bot for me.
#-Link-Snipped-#

0
avi2

@avi2-gV81vT • Sep 9, 2014

Kaustubh Katdare
Request: please add your project to CE Projects. It sure will inspire a few engineers. 😀
Does it support version control?
0
Kaustubh Katdare

@thebigk • Sep 9, 2014

avi2
Does it support version control?
You bet? 😀 It's been disabled as of now, though. I think we'll have to keep the project here itself for the time being.
0
avi2

@avi2-gV81vT • Sep 10, 2014

Oh nice! I hope it's git server, can you give me credentials and server details?
0