A project using the UK REF data sets to collaboratively re-rank the publishing venues and UK Computer Science departments.

The UK Research Excellence Framework (REF) that took place in 2014 was a peer-reviewed process that attempted to objectively rank UK Higher Education academic departments based on the quality of their research. Academic staff from each department were asked to submit their top four outputs from between 2009 and 2014 for peer review. The submitted pieces were reviewed -- at great expense (of time and money) -- and the departments were ranked based on the opinions of the reviewers. These submissions, as well as the outcomes, can be accessed from the REF 2014 website.

Despite being peer-reviewed, the REF process was not uncontroversial: A number of top UK scientists devoted a very large amount of time to evaluating the thousands of submitted works; the final judgments of quality depend on what is still a relatively small college of experts and may still contain biases; the task of selecting the experts to sit on the expert panels implies a pre-existing judgment of research quality. The final rankings produced were widely discussed.

Was all of this expense necessary?

We note that the in mere act of selecting papers for review, authors and institutions are inherently making a value judgment about the quality of their recent work. Thus, it can be assumed that, for any particular author, the works selected tend to be of a higher quality than works that they did not choose to include.

This selection enables us to perform an independent evaluation of publication venues, and hence academic institutions. By aggregating over the subjective quality decisions of tens of institutions and hundreds of authors, we create a participatory ranking system. We show that, when applied to the field of Computer Science, this peer-to-peer ranking reproduces some of the findings of the REF, whilst also creating some surprises.

We matched the works submitted for evaluation with publications recorded in the dblp database, an open-source record of computer science publications, using custom fuzzy matching techniques. From this database, we were also able to identify other published work that authors chose not to submit to the REF process between 2008 and 2014. Based on the combination of these selected and unselected works, we evaluated various publication venues (conferences and journals) to see whether research presented at some venues was systematically selected by authors for inclusion in the REF submission over research that they presented in other (presumably less prestigious) venues.

To estimate the venue quality, we made flow graphs describing researchers' quality judgments about different publication venues: given a set of selected and unselected publications at specific venues, we build a directed graph from all the venues of unselected papers to all the venues of selected ones. We then compute the stationary distribution of this directed graph, representing the probability of reaching a venue after a long random walk through this graph. Heuristically, the steps follow the subjective quality judgments and higher quality venues are more likely to be reached.

Once we estimate a quality score for venues we use them as proxies for judging the quality of research per department: for each author put forward we chose their best 12 papers evaluated by venue, and aggregate their score into the score of the institution. We experimented with selecting the top-4 or even all papers, without any major effect on most rankings. We call this the Peer Score, and the resulting ranking the Peer Rank. We publish the full procedure for computing them.

The following table summarizes the Peer Rank and Peer Score of the top-75 UK Computer Science departments. We also compare the Peer Rank with the REF Output rank, and provide the difference in ranks. We observe some departments are not majorly re-ranked, while others see their position change significantly. As a reminder, Peer Rank and Score are computed using the submissions alone (see above) -- we do not use the judgments and outcomes of the REF peer-review.

As a means to ranking departments we also produced a ranking of computer science academic venues. Note that larger communities and larger venues have a disproportionately high ranking score. Just for fun we also made a table summarizing how many papers from authors of each institutions were used by other institutions as part of their REF submission.

So we conclude, that it is possible to rank both publishing venues and institutions from individual self-rankings from each researcher. It requires no appointment of expert peer-reviewers (solving the chicken-and-egg problem of assessing expertise), and incurs little expense. However, the subjective judgments of authors had to be accurate because of the expectation of high-quality peer review: thus, the submissions should come under some degree of scrutiny to incentivize accurate selection of higher quality works. Note that we were only able to perform this analysis on the Computer Science corpus thanks to the existence of the DBLP open data records of publications.

In terms of the results, we were surprised by how widely the Peer Ranking differed from the REF Output scores from some institutions. This could be due to venues being only a weak proxy for research quality. However, we would not have expected ranking of large departments to be affected by this to the extent observed. Our analysis only considers papers in conferences and journals, introducing possibly a bias. The fraction of REF authors and papers automatically matched to DBLP records is high for all institutions but sometimes far from perfect.

In terms of public policy, the fact that self-rankings can be aggregated to rank institutions is very interesting: they can be used to rank the quality of the peer-review process of established scientific venues, and this in turn to rank departments according to which venues they publish in. Furthermore, since self-ranking is not as socially awkward as ranking other peoples' works, an open and transparent process may be used to rank venues and departments. One may even foresee a process by which a continuous assesement of research quality is put in place, by requiring UK academics to highlight their top-4 works alongside depositing them into open access repositories, as they are now required. However, the expectation of peer-review is what provides incentives to do so diligently -- and preserving this expectation is a key to the success of this method.

• George Danezis
• Rebecca Murphy

The matching and ranking was implemented and tested in about 1000 lines of Python code. All our findings are publicly reproducible and only depend on public datasets: you need to download and uncompress the DBLP xml database and dtd file into the data folder. Then, execute extractdblppapers.py (builds a list of paper records), extractrefdata.py (builds databases of UK ref data), matchdblp.py (matches DBLP records likely to be in the REF), matchauthors.py (matches REF authors to DBLP authors and papers) and finally extractUKdblp.py (performs the ranking).

• REF Submissions for Computer Science. We extract CSV tables from those files, included here.
• DBLP