Twitter-->Tweepy-->File (Tweets as Json)-->LogStash-->ElasticSearch-->Kibana
Logstash actually seems to have an input plugin for Twitter but I couldn't make it work in OSX as it uses some native libraries (which is no longer supported by JRuby).
(I am assuming OSX but should be the similar for other Unix variants )
-
I recommend creating an isolated virtual environment for python before installing Tweepy. Go through this and have it installed in your user area.
-
Install tweepy ( inside the virtualenv you just created )
pip install tweepy -
We will use Twitter's Streaming API to access the tweets.
-
In order to use it, we need to get the following information
- Consumer Key
- Consumer Secret
- Access Token
- Access Token Secret
Follow the instructions here to setup a Twitter 'Application' and get this information.
-
I have included the simple wrapper around Tweepy to get the tweets and write them into a file. Update tweepy/oauth.token with the 4 pieces of information.
-
Run Tests.py and redirect the output to a file.
python Tests.py | tee tweets -
Once we confirm that the tweets are being written to the file, all that needs to be done is to run Logstash with embedded elastic search server.
-
Go through Logstash documentation to learn how to setup logstash. Update "logstash/logstash.conf" with the full path of file where the tweets are being written and use it to start up logstash in standalone mode.
-
That's it. Open up your browser and point to http://localhost:9292 and play around with Kibana.