EECSE6895 Group 37 Final Project Submission.

License: All code is distributed under the Apache License V2.0. More information can be found here at http://www.apache.org/licenses/ and in the accompanying License file.

Group Members: Marshall Van Loon (mhv2109) Palash Matey (pm2824)

Dependencies: All code is written in Python 2.7 supplied with 64-bit Miniconda2 distribution (http://conda.pydata.org/miniconda.html) numpy (pip install numpy OR conda install numpy) scipy (pip install scipy OR conda install scipy) Spark-1.6.1 (http://spark.apache.org/downloads.html) sqlite-jdbc-3.8.11.2.jar (https://bitbucket.org/xerial/sqlite-jdbc/downloads) textstat-0.2 (pip install textstat) NLTK (pip install nltk) SQLAlchemy (pip install sqlalchemy) PRAW (pip install praw) boto3 (pip install boto3)

Note: All code written for demonstration during our presentation is located in respective src/ folders and is named *_demo.py. The tests/ folders contain scratch scripts. The utils/ folders contain useful scripts not necessary to run code in src/. READMEs for specific modules are within the root directory of their respective modules.

To run code on a cluster, type:

/path/to/spark/bin/spark-submit --master spark://localhost:7077 --driver-class-path /path/to/jar/sqlite-jdbc-3.8.11.2.jar pythonfile.py