This repository contains the code used in the paper:

Paragraph-level Commonsense Transformers with Recurrent Memory. Saadia Gabriel, Chandra Bhagavatula, Vered Shwartz, Ronan Le Bras, Maxwell Forbes, Yejin Choi. AAAI 2021. [ArXiv]

This is a general purpose framework for aligning commonsense knowledge with narrative text. The repo contains

1. A framework for distantly supervised paragraph-level commonsense knowledge alignment; and
2. Modeling code for finetuning pretrained transformers to generate paragraph-level commonsense inferences.

Our models infer about the social commonsense underlying narratives (i.e. actions or changes in emotional states of characters related to what would likely happen or has likely already happened at a particular point in a narrative):

python >= 3.6 (I suggest using a virtual environment)

Note: For now, the code assumes that stories contain at most 5 sentences and models generate inferences for up to 5 sentence stories.

pip install -r requirements.txt 
cd data
wget https://storage.googleapis.com/ai2-mosaic/public/atomic/v1.0/atomic_data.tgz
tar -xvzf atomic_data.tgz 

Note: torch and torchvision may need to be installed with a specified cuda version to run on GPUs. (For example, if the cuda version is 10.1, the install command for torch should be torch=={torch_version}+101)

Data for distant supervision should be a file "train-processed.jsonl" in the data folder. The file should contain the following keys:

dict_keys(['storyid', 'storytitle', 'sentence1', 'sentence2', 'sentence3', 'sentence4', 'sentence5', 'sentence1_tokens', 'sentence1_noun_phrases', 'sentence1_verb_phrases', 'sentence2_tokens', 'sentence2_noun_phrases', 'sentence2_verb_phrases', 'sentence3_tokens', 'sentence3_noun_phrases', 'sentence3_verb_phrases', 'sentence4_tokens', 'sentence4_noun_phrases', 'sentence4_verb_phrases', 'sentence5_tokens', 'sentence5_noun_phrases', 'sentence5_verb_phrases'])

Get the preprocessed data from link and place in data folder

cd src/ds
python distant_supervision.py --target_dir ../../data/atomic 

Get pretrained comet model from link and place in data folder

cd src/ds
python distant_supervision.py --comet --comet_location ../../data --target_dir ../../data/atomic 

Combine distantly supervised data into a single file "all_data.jsonl" by running combine_files.py in ds folder

Split data using split.py in ds folder

Change format between comet and heuristic data by setting comet = True or comet = False in split.py file

For comet data, files are in the format "c_atomic_{split}.txt"

For heuristic data, files are in the format "h_atomic_{split}.txt"

cd src/gpt (or src/gpt2) 
python finetune_model.py --log_dir ./log --model_dir ./models --data_dir ../../data --use_mem False --comet True 

cd src/gpt (or src/gpt2) 
python finetune_model.py --log_dir ./mem_log --model_dir ./mem_models --data_dir ../../data --use_mem True --comet True

cd src/gpt (or src/gpt2) 
python decode.py --model_type ./models/model --original_file '../../data/c_atomic_test.jsonl' --data_dir ../../data --save_dir ../../data/gen_data --save_filename 'outputs.jsonl' --load_epoch 8 

cd src/gpt (or src/gpt2) 
python decode.py --model_type ./mem_models/model --original_file '../../data/c_atomic_test.jsonl' --data_dir ../../data --save_dir ../../data/gen_data --save_filename 'outputs.jsonl' --load_epoch 9 --use_mem True

Note: Make sure to decode using one GPU for memory model. (If using a multi GPU setting, specify CUDA_VISIBLE_DEVICES={device_id} before the python command)

Data

Evaluation scripts are included under src/eval, including scripts to reformat generated inferences for NLI evaluation. To run the full NLI evaluation, refer to the original github repo for SemBERT.

Note: Models were trained in a multi GPU setting, so ensure args.use_multigpu is set to True when decoding from pretrained models.

To most easily run pretrained models with your own data, use the narrative inference demo repo here. Demo code will be integrated into the main github repo at a later point in time.

Please cite this repository using the following reference:

@inproceedings{Gabriel2021ParagraphLevelCT,
title={Paragraph-level Commonsense Transformers with Recurrent Memory},
author={Gabriel, Saadia and Bhagavatula, Chandra and Shwartz, Vered and Le Bras, Ronan and Forbes, Maxwell and Choi, Yejin},
booktitle={AAAI},
year={2021},
}