Redditor's Club

Redditor's Club is a data pipeline for analysis of Reddit trends. Reddit is an interest-based social media network popular among young males in North America.

Technology Stack

This project currently makes use of the following technologies:

Amazon DynamoDB
Apache Spark 1.6.1 with Hadoop 2.7
MySQL
AWS S3
AWS Redshift
Django 1.9.6 with the following frameworks: HighCharts, jQuery, Bootstrap

The figure below shows the flow of data through the pipeline:

The data used in this project is the set of Reddit comments published and compiled by reddit user /u/Stuck_In_the_Matrix. The total size of all comments from October 2007 to December 2015 is greater than 1TB.

Name		Name	Last commit message	Last commit date
Latest commit History 168 Commits
airflow		airflow
images		images
project		project
redshift		redshift
serving_optimization		serving_optimization
sqlite		sqlite
word_cloud		word_cloud
.gitignore		.gitignore
.gitmodules		.gitmodules
Dockerfile		Dockerfile
README.md		README.md
build.sh		build.sh
domain.py		domain.py
download.py		download.py
foreign.txt		foreign.txt
ngram-nltk.py		ngram-nltk.py
ngram.cql		ngram.cql
ngram.py		ngram.py
render.yaml		render.yaml
requirements-dev.txt		requirements-dev.txt
requirements.txt		requirements.txt
setup.sh		setup.sh
storage.py		storage.py
subreddits.txt		subreddits.txt
timeconverter.py		timeconverter.py
tokenizer.py		tokenizer.py
utils.py		utils.py

yuguang/reddit-comments

Folders and files

Latest commit

History

Repository files navigation

Redditor's Club

Technology Stack

About

Resources

Stars

Watchers

Forks

Languages