This repository contains code for training deep learning systems to do question answering tasks. Our primary focus is on Aristo's science questions, though we can run various models on several popular datasets.

This code is a mix of scala (for data processing / pipeline management) and python (for actually training and executing deep models with Keras / Theano / TensorFlow).

This repository implements several variants of memory networks, including the models found in these papers:

• The original MemNN, from Memory Networks, by Weston, Chopra and Bordes
• End-to-end memory networks, by Sukhbaatar and others (close, but still in progress)
• Dynamic memory networks, by Kumar and others (close, but still in progress)
• DMN+, from Dynamic Memory Networks for Visual and Textual Question Answering, by Xiong, Merity and Socher (close, but still in progress)
• The attentive reader, from Teaching Machines to Read and Comprehend, by Hermann and others
• Windowed-memory MemNNs, from The Goldilocks Principle: Reading Children's Books with Explicit Memory Representations (in progress)

As well as some of our own, as-yet-unpublished variants. There is a lot of similarity between the models in these papers, and our code is structured in a way to allow for easily switching between these models. For a description of how we've built an extensible memory network architecture in this library, see this readme.

This code allows for easy experimentation with the following datasets:

• AI2 Elementary school science questions (no diagrams)
• The Facebook Children's Book Test dataset
• The Facebook bAbI dataset
• The NewsQA dataset
• The Stanford Question Answering Dataset (SQuAD)
• The Who Did What dataset

And more to come... In the near future, we hope to also include easy experimentation with CNN/Daily Mail and SimpleQuestions.

This code is a mix of scala and python. The intent is that the data processing and experiment pipeline code is in scala, and the deep learning code is in python. The recommended approach is to set up your experiments in scala code, then run them through sbt. Some documentation on how to do this is found in the README for the org.allenai.deep_qa.experiments package.

If for whatever reason you don't want to gain the benefits of the scala pipeline when running experiments, you can run the python code manually. To do this, from the base directory, you run the command python src/main/python/run_solver.py [model_config]. You must use python >= 3.5, as we make heavy use of the type annotations introduced in python 3.5 to aid in code readability (I recommend using anaconda to set up python 3, if you don't have it set up already).

You can see some examples of what model configuration files look like in the example experiments directory. We try to keep these up to date, but the way parameters are specified is still sometimes in a state of flux, so we make no promises that these are actually usable with the current master (and you'll have to provide your own input files to use them, in any event). Looking at the most recently added or changed example experiment should be your best bet to get an accurate format. And if you find one that's out of date, submitting a pull request to fix it would be really nice!

The best way currently to get an idea for what options are available in this configuration file, and what those options mean, is to look at the class mentioned in the solver_class field. Looking at the dynamic_memory_network.json example, we can see that it's using a MultipleTrueFalseMemoryNetworkSolver as it's solver_class. If we go to that class's __init__ method, in the code, we don't see any parameters, because MultipleTrueFalseMemoryNetworkSolver has no unique parameters of its own. So, we continue up the class hierarchy to MemoryNetworkSolver, and we can see the parameters that it takes: things like num_memory_layers, knowledge_encoder, entailment_model, and so on. If you continue on to its super class, TextTrainer, you'll find more parameters, this time for things that deal with word embeddings and sentence encoders. Finally, you can continue to the base class, Trainer, to see parameters for things like whether and where models should be saved, how to run training, specifying debug output, running pre-training, and other things. It would be nice to automatically generate some website to document all of these parameters, but I don't know how to do that and don't have the time to dedicate to making it happen. So for now, just read the comments that are in the code.

There are several places where we give lists of available choices for particular options. For example, there is a list of concrete solver classes that are valid options for the solver_class parameter in a model config file. One way to find lists of available options for these parameters (other than just by tracing the handling of parameters in the code) is by searching github for get_choice or get_choice_with_default. This might point you, for instance, to the knowledge_encoders field in memory_network.py, which is imported from layers/knowledge_encoders.py, where it is defined at the bottom of the file. In general, the places where there are these kinds of options are in the solver class (already mentioned), and the various layers we have implemented - each kind of Layer will typically specify a list of options either at the bottom of the corresponding file, or in an associated __init__.py file (as is done with the sentence encoders).

We've tried to also give reasonable documentation throughout the code, both in docstring comments and in READMEs distributed throughout the code packages, so browsing github should be pretty informative if you're confused about something. If you're still confused about how something works, open an issue asking to improve documentation of a particular piece of the code (or, if you've figured it out after searching a bit, submit a pull request containing documentation improvements that would have helped you).

If you use this code and think something could be improved, pull requests are very welcome. Opening an issue is ok, too, but we're a lot more likely to respond to a PR. The primary maintainer of this code is Matt Gardner, with a lot of help from Pradeep Dasigi (who was the initial author of this codebase), Mark Neumann and Nelson Liu.

This code is released under the terms of the Apache 2 license.