In healthcare, large amounts of heterogeneous medical data have become available in various healthcare organizations (payers, providers, pharmaceuticals). This data could be an enabling resource for deriving insights for improving care delivery and reducing waste. The enormity and complexity of these data-sets present great challenges in analyses and subsequent applications to a practical clinical environment. This course introduced the characteristics of medical data and associated data mining challenges on dealing with such data. We covered various algorithms and systems for big data analytics in the context of concrete healthcare analytic applications such as predictive modeling, computational phenotyping and patient similarity. Used a big data analytics system such as Hadoop family (Hive, Pig, HBase), Spark and Graph DB. Used Docker container for code implementation. This course consisted of five Projects and a group project.
• Project 1: Used Python in predicting patient mortality analysis. Did preliminary descriptive statistics analysis, constructed features selection process, created machine learning models using logistic regression, support vector machine learning, and decision tree. Analyzed performance of the models, using accuracy, area under the curve (AUC), precision, recall, F-score, K-fold and randomized K-fold validation.
• Project 2: Used Hive in descriptive statistics related to patient’s hospital admission data, used Pig in converting the raw data to standardized SVMLight format which involved the feature selection process, used logistic regression classifier in Python and used AUC and ROC metrics to assess performance, trained multiple logistic regression classifiers using Hadoop’s map reduce program.
• Project 3: Used Spark with Scala to do K-Means clustering and Gaussian Mixture algorithms to discover groups of patients with similar characteristics.
• Project 4: Similarity among patients were analyzed using Spark GraphX using Scala.
• Project 5: Used PyTorch and Python to implement neural network algorithms on clinical data.
• Group Project: Used tools and techniques learnt from this course in analyzing big data with over 10 million records extracted from the MIMIC III database in predicting mortality of patients in ICU. The final report includes, motivation, literature survey, setting up the environment, data cleaning/pre-processing, algorithm implementation and performance metrics comparison and analysis.