This tool computes the correlations between a given theta and human judgements.

A theta is a scoring function that takes as input a summary and a document collections and outputs a score.

It needs as input a dataset of manual annotations with summary manual scored with pyramid and responsiveness scores. The tool computes the pearson's r, spearman's rho and NDCG@10 between the theta and the two manual annotations.

If you reuse this software, please use the following citation:

@inproceedings{TUD-CS-2017-0074,
	title = {{A Principled Framework for Evaluating Summarizers: Comparing Models of Summary Quality against Human Judgments}},
	author = {Peyrard, Maxime and Eckle-Kohler, Judith},
	publisher = {Association for Computational Linguistics},
	volume = {Volume 2: Short Papers},
	booktitle = {{Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (ACL 2017)}},
	pages = {(to appear)},
	month = aug,
	year = {2017},
	location = {Vancouver, Canada},
}

Abstract: We present a new framework for evaluating extractive summarizers, which is based on a principled representation as optimization problem. We prove that every extractive summarizer can be decomposed into an objective function and an optimization technique. We perform a comparative analysis and evaluation of several objective functions embedded in well-known summarizers regarding their correlation with human judgments. Our comparison of these correlations across two datasets yields surprising insights into the role and performance of objective functions in the different summarizers.

Contact person: Maxime Peyrard, peyrard@aiphes.tu-darmstadt.de

http://www.ukp.tu-darmstadt.de/

• Python 2.7
• Numpy 1.11.1 (http://www.numpy.org)
• nltk 3.2.1 (http://www.nltk.org)

An example of theta is provided: jensen-shanon divergence between summary and the document collection. Some sample data are also provided to show the structure that the tool expects.

In order to replace the sample data with a real dataset, the user should modify the following variables in the file read_data:

• MANUAL_ANNOTATIONS_FOLDER
• DATASET_FOLDER

To test the installation just run: python example.py