##Step1. Data Wrangling

• Run "data_wrangling.py"
• Get API Key from NYTimes.
• Get articles from NYTimes and convert json to csv format to save.
• They don't provide full article.
• Challenges:

• simplejson library errors occurs and fixed it with upgrade 3.6.5.
• Sometimes server timeout.
• Article have utf-8 and unicode both of them.
• Sometimes JSON Data contains journalist name but does not have article.

##Step2. Data Exploratory

• Run "data_exploratory.py"
• Clean stemmer with SnowballStemmer.
• Change encoding utf-8 to unicode.
• step2_data_exploratory.py generates pkl file with all article from 2010 to 2014.

• It is to expensive to my MAC.

• step2_data_exploratory(2_people).py generates pkl file with two journalist's articles from 2010 to 2014.

• It is not that expensive.
• I choose two journalist, NEIL GENZLINGER and STEPHEN HOLDEN because they have written so many articles.
• During 2010~2014, NEIL GENZLINGER wrote 1211 articles and STEPHEN HOLDEN wrote 1317 articles.

##Step3. Feature Engineering

• Feature Extraction

• Applied TfidfVectorizer and deleted stopwords.

• Feature Selection

• SelectKBest, chi2 are faster than SelectPercentile in my case.
• f_classif for classification problem.
• f_regression for regression problems.
• chi2 for classification problems with sparse non-negative data (typically text data).
• My prefer : SelectPercentile(score_func=chi2), SelectKBest(k=100), SelectPercentile(f_classif, percentile=30).

• preprocess_4 method is for supervised learning.
• preprocess_2 method is for unsupervised learning.

##Step4. Apply Different Algorithm

• I applied various Supervised and Unsupervised Algorithm with Dimension Reduction.
• Supervised Learning

• Classification : GaussianNB, BernoulliNB, AdaBoostClassifier, DecisionTreeClassifier, KNeighborsClassifier, RandomForestClassifier, SVC
• Regression : LinearRegression, ElasticNet, Lasso, Ridge

• Unsupervised Learning

• KMeans, SpectralClustering, AgglomerativeClustering

• Dimension Reduction

• Linear Dimensionality Reduction : PCA(unsupervised), RandomizedPCA(unsupervised), LDA(supervised)
• Non Linear Dimensionality Reduction : Isomap(unsupervised), LocallyLinearEmbedding(unsupervised)

Feature Selection : SelectKBest

Feature Selection : SelectPercentile

Feature Selection : SelectKBest

Feature Selection : SelectPercentile

Dimension Reduction : PCA

Dimension Reduction : RandomizedPCA

Dimension Reduction : LDA

Dimension Reduction : Isomap

Dimension Reduction : LocallyLinearEmbedding

##Reference

• Book

• Building Machine Learning Systems with Python
• Machine Learning in Action

• Class

• Intro to Machine Learning

• Site

• http://sujitpal.blogspot.kr/2013/05/feature-selection-with-scikit-learn.html
• http://stackoverflow.com/questions/25250654/how-can-i-use-a-custom-feature-selection-function-in-scikit-learns-pipeline
• http://stackoverflow.com/questions/22294241/plotting-a-decision-boundary-separating-2-classes-using-matplotlibs-pyplot
• http://stackoverflow.com/questions/12118720/python-tf-idf-cosine-to-find-document-similarity
• http://stackoverflow.com/questions/12118720/python-tf-idf-cosine-to-find-document-similarity
• http://nbviewer.ipython.org/github/jakevdp/sklearn_scipy2013/blob/master/notebooks/05.2_application_to_text_mining.ipynb
• http://nbviewer.ipython.org/github/temporaer/tutorial_ml_gkbionics/tree/master/
• http://www.datasciencecentral.com/profiles/blogs/python-scikit-learn-to-simplify-machine-learning-bag-of-words-to
• http://www.naftaliharris.com/blog/visualizing-k-means-clustering/
• http://www.slideshare.net/SarahGuido/kmeans-clustering-with-scikitlearn
• http://blog.christianperone.com/?p=1589
• http://scikit-learn.org/stable/