dossier.models
is a Python package that provides experimental active learning
models. They are meant to be used as search engines through dossier.web
web
services.
dossier.models
is on PyPI and can be installed with pip
:
pip install dossier.models
Currently, dossier.models
requires Python 2.7. It is not yet Python 3
compatible.
API documentation with examples is available as part of the Dossier Stack documentation: http://dossier-stack.readthedocs.org
dossier.models
comes with an example web application that demonstrates how to
use all of the Dossier Stack components to do active learning. The following is
a step-by-step guide to get you up and running with a simple example of
SortingDesk. This guide assumes basic familiarity with standard Python tools
like pip
and virtualenv
.
This guide also requires a database of some sort to store data. You can use any of the backends supported by kvlayer (like PostgreSQL, HBase or MySQL). For this guide, we'll use Redis since it requires very little setup. Just make sure it is installed and running on your system.
Here are a couple of screenshots of SortingDesk in action:
First, you should create a new Python virtual environment and install
dossier.models
from PyPI:
$ virtualenv dossier
$ source ./dossier/bin/activate
$ pip install dossier.models
Depending upon your system setup, this may take a bit of time since
dossier.models
depends on numpy
, scipy
and scikit-learn
.
Now verify that dossier.models
is installed correctly:
$ python -c 'import dossier.models'
If all is well, then the command should complete successfully without any output.
Next, we need to setup configuration so that Dossier Stack knows which database to use and which indexes to create on feature collections. You can grab a sample configuration from GitHub:
$ curl -O https://raw.githubusercontent.com/dossier/dossier.models/master/data/config.yaml
The config looks like this:
kvlayer:
app_name: dossier
namespace: models
storage_type: redis
storage_addresses: ['localhost:6379']
dossier.store:
feature_indexes: ['name', 'keywords']
The first section configures your database credentials. This config assumes
you're using Redis running on localhost
on port 6379
(the default).
The second section tells Dossier Stack which indexes to create on feature
collections. This configuration is dependent on the features in your data.
In this sample configuration, we've chosen name
and keywords
because both
are features in the sample data set.
To download and load the sample data set, grab it from GitHub and use the
dossier.store
command to load it:
$ curl -O https://raw.githubusercontent.com/dossier/dossier.models/master/data/example.fc
$ dossier.store -c config.yaml load --id-feature content_id example.fc
The dossier.store
command allows you to interact with feature collections
stored in your database. The --id-feature
flag tells dossier.store
to use
the value of the content_id
feature as the feature collection's primary key.
If this flag is omitted, then a uuid
is generated instead.
You can confirm that data was added to your database with the ids
command:
$ dossier.store -c config.yaml ids
doc11
doc12
doc21
doc22
doc23
...
Finally, you can run the web application bundled with dossier.models
:
$ dossier.models -c config.yaml
Open your browser to
http://localhost:8080/SortingDesk to
see an example of SortingDesk
with the sample data. If you click on the X
link on an item in the queue, a negative label will be added between it and the
query indicated at the top of the page. Or you can drag an item from the queue
into a bin---or drop it anywhere on the body page to create a new bin. Bins can
also be dragged on to other bins to merge them. Go ahead and try it. You can
confirm that a label was made with the dossier.label
command:
$ dossier.label -c config.yaml list
Label(doc22, doc42, annotator=unknown, 2014-11-26 16:02:01, value=CorefValue.Negative)
You should also be able to see labels being added in the output of the
dossier.models
command if you're running it in your terminal.
There is also a simpler example using plain SortingQueue
available at
http://localhost:8080/SortingQueue.