DDLite is intended to be a lightweight but powerful framework for prototyping & developing structured information extraction applications for domains in which large labeled training sets are not available or easy to obtain, using the data programming paradigm.

In the data programming approach to developing a machine learning system, the developer focuses on writing a set of labeling functions, which create a large but noisy training set. DDLite then learns a generative model of this noise—learning, essentially, which labeling functions are more accurate than others—and uses this to train a discriminative classifier.

At a high level, the idea is that developers can focus on writing labeling functions—which are just (Python) functions that provide a label for some subset of data points—and not think about algorithms or features!

DDLite is very much a work in progress, but some people have already begun developing applications with it, and initial feedback has been positive... let us know what you think, and how we can improve it, in the Issues section!

First of all, make sure all git submodules have been downloaded.

git submodule update --init --recursive
git submodule update --recursive

DeepDive Lite requires a few python packages including:

• nltk
• lxml
• requests
• numpy
• scipy
• matplotlib

We provide a simple way to install everything using virtualenv:

# set up a Python virtualenv
virtualenv .virtualenv
source .virtualenv/bin/activate

pip install --requirement python-package-requirement.txt

Note: if you have an issue with the matplotlib install related to the module freetype, see this post; if you have an issue installing ipython, try upgrading setuptools

Alternatively, they could be installed system-wide if sudo pip is used instead of pip in the last command without the virtualenv setup and activation.

In addition the Stanford CoreNLP parser jars need to be downloaded; this can be done using:

./install-parser.sh

Finally, DeepDive Lite is built specifically with usage in Jupyter/IPython notebooks in mind. The jupyter command is installed as part of the above installation steps, so the following command within the virtualenv opens our demo notebook.

jupyter notebook examples/GeneTaggerExample_Extraction.ipynb

The best way to learn how to use is to open up the demo notebooks in the examples folder. GeneTaggerExample_Extraction.ipynb walks through the candidate extraction workflow for an entity tagging task. GeneTaggerExample_Learning.ipynb picks up where the extraction notebook left off. The learning notebook demonstrates the labeling function iteration workflow and learning methods. For examples of extracting relations, see GenePhenRelationExample_Extraction.ipynb and GenePhenRelationExample_Learning.ipynb.

The flowchart below illustrates the labeling function iteration workflow.

First, we generate candidates, a hold out set of candidates (using MindTagger or gold standard labels), and initial set of labeling functions L⁽⁰⁾ for a CandidateModel. The next step is to examine the coverage and accuracy of labeling functions using CandidateModel.plot_lf_stats(), CandidateModel.top_conflict_lfs(), CandidateModel.lowest_coverage_lfs(), and CandidateModel.lowest_empirical_accuracy_lfs(). If coverage is the primary issue, we write a new candidate function and append it to L⁽⁰⁾ to form L⁽¹⁾. If accuracy is the primary issue instead, we form L⁽¹⁾ by revising an existing labeling function which may be implemented incorrectly. This process continues until we are satisfied with the labeling function set. We then learn a model, and depending on the performance over the hold out set, revaluate our labeling function set as before.

Several parts of this workflow could result in overfitting, so tie your bootstraps with care:

• Generating a sufficiently large and diverse hold out set before iterating on labeling functions is important for accurate evaluation.
• A labeling function with low empirical accuracy on the hold out set could work well on the entire data set. This is not a reason to delete it (unless the implementation is incorrect).
• Metrics obtained by altering learning parameters directly to maximize performance against the hold out set are upwards biased and not representative of general performance.

Here are a few practical tips for working with DeepDive Lite:

• Use autoreload
• Keep working source code in another file
• Pickle extractions often and with unique names
• Entire objects (extractions and features) subclassed from Candidates can be pickled
• Document past labeling functions either remotely or with the CandidateModel log

Update coming...

DDLite provides a number of options for learning predictive models. The simplest option is to just DDLiteModel.train_model(), but this uses only default parameter values. DDLiteModel.train_model accepts the following arguments

The following parameters can be passed to DDLiteModel.learn_lf_accuracies, which executes the first stage of the data programming learning pipeline.

For the logistic regression model (DDLiteModel.train_model(method="lr", ...)), the regularization parameter can be tuned automatically. This requires a validation set defined using DDLiteModel.set_holdout(validation_frac=p) with p > 0, and either multiple or no values passed as the regularization parameter. The rest of the parameters passed to the logistic regression model via model_opts are as follows: