GitHub - cbeach/twitter_vamp: A server that continually collects and stores tweets from the streaming api

cbeach / twitter_vamp Public

Notifications You must be signed in to change notification settings
Fork 0
Star 1

A server that continually collects and stores tweets from the streaming api

Notifications

Name		Name	Last commit message	Last commit date
Latest commit History 42 Commits
core		core
data		data
data_collection		data_collection
nlp		nlp
util		util
.gitattributes		.gitattributes
.gitignore		.gitignore
README		README
config.py		config.py
dump_archive.py		dump_archive.py
ekg.py		ekg.py
example_config.py		example_config.py
file_streamer.sh		file_streamer.sh
manage.py		manage.py
replay.py		replay.py
sort_by_language.sh		sort_by_language.sh
start_archiver.sh		start_archiver.sh
todo		todo
tweet_structure		tweet_structure
user_list.py		user_list.py

Repository files navigation

Twitter_vamp - Twitter archiving tool
-------------------------------------------------------------------------------
Copyright © 2011 Casey Beach

Developed by Casey Beach to be used as a data collection tool for research 
into natural language processing, machine learning, and data analysis.

Special thanks to Derek Z Hurley for his work on clubot.

TODO:
-------------------------------------------------------------------------------
* Create IRC bot for control.
* Use a config file
* Write database transfer tool.


Release Notes
-------------------------------------------------------------------------------
0.01 - collect_data.py will connect with the twitter api useing a hardcoded
user name and password.  The twitter streaming api is dumped straight to the
terminal.  The data base initialization is written but not yet tested in any
way.


AMQP Structure
-------------------------------------------------------------------------------
Exchanges:
    *direct.raw
    *direct.text
    *direct.mentions
    *direct.hashtags
    *direct.urls
    *direct.user
    *direct.place
    *direct.delete

Structure:
                                     * ----------------*
                                     * ----> text      *
                                     * ----> user      *
*------------*      *----------*     * ----> mentions  *
*  Raw Feed  *----> *  Parser  *---->* ----> hashtags  *
*------------*      *----------*     * ----> urls      *
                                     * ----> place     *
                                     * ----> deletes   *
                                     *-----------------*


Database Description
-------------------------------------------------------------------------------
I'm changing the way the Database works, stay tuned...