Using Twitter's Streaming API, to compute trends.

Currently at a very early stage.

Authentication to Twitter's API is handled by Leah Culver's oauth module. Can be found at: http://oauth.googlecode.com/svn/code/python/oauth/oauth.py

• Main Thread: Initiates worker threads
  • Thread Listener: Connects to the streaming API, reads the stream, splits tweets into tokens, and updates the frequency table.
  • Thread Statistician: Reads the frequency table and computer the trends.
• Classes:

  • trends.api.stream.Stream: Handles authentication to Twitter API.

    • handler: File-like object that listens to Twitter API.
    • connect(): Opens the connection of the handler to the API.
    • get_tweet(): Transforms a received JSON dictionary to a Tweet instance.
    • close(): Closes the connection of the handler to the API

  • trends.api.stream.Tweet: Instances of this class represent tweet information.

    • text: Tweet text
    • get_tokens(): Analyzes the text and returns its tokens.

  • trends.analysis.listener.Listener: Facilitates the Stream class, to retrieve data from the Twitter API.

    • stream: A Stream instance

  • trends.analysis.statistician.Stats: Wrapper class around the data structure that will store token along with their statistics.

    • _scores: Dictionary that stores pairs of `token`:<TokenStats object>
    • add(token): If no entry exists in dictionary _scores for key token, an entry is created. Else, it's updated.
    • get_stats(): Initiates the computation of stats for the tokens in _scores.

  • trends.analysis.statistician.TokenStats: Instances of this class maintain the statistics that correspond to a single token.

    • observation: Observations of the token, within this round
    • rounds_observed: Amount of rounds that this token has been observed at least once
    • mean: Floating mean.
    • sqr_mean: Floating square mean
    • std: Floating Standard deviation
    • score: Floating average z-score
    • increase(): Increases the amount of observations for the corresponding token, by 1.
    • compute_scores(): Computes the score and all the statistical parameters.
    • zero(): Zeros parameters observation and score

  • trends.analysis.statistician.Statistician: Analyzes the Stats class and computes statistics.

:

--------------

+--------------- Main Thread --------------+ | --------------- | | | | | | |

---------------- -------------------- Listener Thread Statistician Thread ---------------- -------------------- Stream <-----------------+ Statistician | Stats | TokenStats | | Twitter API