The project code can be categorized into 3 groups:
- Data management and extraction
- Transformers for data pipeline
- Methods for queueing, building, and exceuting tests
Below I will group the file names by these categories, and all files not mentioned are miscelaneous, superfluous or unimportant.
(1) DATA MANAGEMENT, EXTRACTION AND VALIDATION
anonymizer.py -- anonymizes MRNs, SSNs and names from free text during initial data transformation
extract_data.py -- misc data extraction functions
free_text_jsonifyer.py -- Extracts meta data from free text files
language_processing.py -- used to clean extracted values
loader.py -- Loads patient data from disk
tables.py -- generates table statistics for paper
validate.py -- Generates statistics to validate results
generateTurkTasks.py -- Creates a csv file for localturk to efficiently do manual extraction
(2) TRANSFORMERS
baseline_transformer.py -- contains all structured data transformers
doc2vec_trainer.py -- Creates doc2vec models to be used by doc2vec transformer
doc2vec_transformer.py -- Transforms arbitrary length text into fixed dimensional semantic representation
icd_transformer.py -- Transforms ICD9 code into hierarchical numeric representation
value_extractor_transformer.py -- contains the Regex extractors
(3) QUEUING, BUILDING, AND EXECUTING TESTS
decision_model.py -- contains the hard-coded clincical guideline
experiment_runner.py -- Daemon process to continually run tests
model_builder.py -- Build ML/NLP models to test
model_tester.py -- Used by experiment runer to run a test on a given model
queue_test.py -- Helper function to queue up tests
run_test.py -- Standalone code to run an individual model and test