This repository contains the three WikiReading datasets as used and described in WikiReading: A Novel Large-scale Language Understanding Task over Wikipedia, Hewlett, et al, ACL 2016 (the English WikiReading dataset) and Byte-level Machine Reading across Morphologically Varied Languages, Kenter et al, AAAI-18 (the Turkish and Russian datasets).

Run get_data.sh to download data the English WikiReading dataset.

Run get_ru_data.sh and get_tr_data.sh to get the Russian and Turkish version of the WikiReading data, respectively.

If you use the data or the results reported in the papers, please feel free to cite them.

@inproceedings {hewlett2016wikireading,
 title = {{WIKIREADING}: A Novel Large-scale Language Understanding Task over {Wikipedia}},
 booktitle = {Proceedings of the The 54th Annual Meeting of the Association for Computational Linguistics (ACL 2016)},
 author = {Daniel Hewlett and Alexandre Lacoste and Llion Jones and Illia Polosukhin and Andrew Fandrianto and Jay Han and Matthew Kelcey and David Berthelot},
 year = {2016}
}

and

@inproceedings{byte-level2018kenter,
  title={Byte-level Machine Reading across Morphologically Varied Languages},
  author={Tom Kenter and Llion Jones and Daniel Hewlett},
  booktitle={Proceedings of the The Thirty-Second AAAI Conference on Artificial Intelligence (AAAI-18)},
  year={2018}
}

Train, validation, and test datasets are in TFRecord or streamed JSON (one JSON object per line). They are 45GB, 5GB, and 3GB respectively. For example test.tar.gz contains 15 files whose union is the whole test set. We split them to help speed up training/testing by parallelizing reads. Any one of the shards can be opened with a TFRecordReader or with your favorite JSON reader for every line. Download a sample TFRecord shard or a sample JSON shard of the validation set (1/15th) to play around with if disk space is limited.

Each instance contains these features (some features may be empty).