deep-hashtagprediction

Deep Convolutional Neural Network for Twitter Hashtag Prediction

Preparation

To prepare the project for execution you need to do the following:

Put your tweets inside the tweets/ folder. Note that they should be gzipped. They should provide a test and a train file. Adapt the parse_tweets.py script.
If you already have word embeddings place them inside the embeddings/ folder and adapt the extract_embeddings.py. You need to change the wemb_ variables and define the name of the file, the delimiter used and the number of dimensions.

If you don't have word embeddings you may run the 'create_wordembeddings.py' script. In the main method declare the files with the tweets you are using. In current case only the tweets/hastag_tweets.gz are used. The output is stored in embeddings/hashtag_tweets_embedding.
Parse tweets: Run the 'parse_tweets.py' script. It uses as input the files with the training and test data. It creates 6 files. It creates the tweet-representation for both the training and test data. It creates the hashtag-representation for both the training and the hashtag. It creates 2 index, one for the twitter words and one for the hashtags.
Extract_embeddings: it creates 2 lookup tables with word-idx -> word-embedding. One lookup table for the twitter-words and one for the hashtags. Run the 'extract_embeddings.py' script.

Run the training.py script as follows: THEANO_FLAGS=mode=FAST_RUN,device=gpu,floatX=float32,optimizer_including=cudnn python training.py

You need to have CUDA and CuDNN installed.

Name		Name	Last commit message	Last commit date
Latest commit History 5 Commits
embeddings		embeddings
parsed_tweets		parsed_tweets
tweets		tweets
README.md		README.md
Stats.txt		Stats.txt
alphabet.py		alphabet.py
balanced_testset.py		balanced_testset.py
conv1d.py		conv1d.py
create_wordembeddings.py		create_wordembeddings.py
evalutaion_metrics.py		evalutaion_metrics.py
extract_embeddings.py		extract_embeddings.py
hashtag_frequency.py		hashtag_frequency.py
nn_layers.py		nn_layers.py
parse_tweets.py		parse_tweets.py
sgd_trainer.py		sgd_trainer.py
training.py		training.py
utils.py		utils.py