Ewan Carr^*, Rebecca Bendayan^*, Daniel Bean, Matthew Stammers, Wenjuan Wang, Huayu Zhang, Thomas Searle, Zeljko Kraljevic, Anthony Shek, Hang T T Phan, Walter Muruet, Anthony J Shinton, Ting Shi, Xin Zhang, Andrew Pickles, Daniel Stahl, Rosita Zakeri, Kevin O'Gallagher, Amos Folarin, Lukasz Roguski, Florina Borca, James Batchelor, Xiaodong Wu, Jiaxing Sun, Ashwin Pinto, Bruce Guthrie, Cormac Breen, Abdel Douiri, Honghan Wu, Vasa Curcin, James T Teo^#, Ajay Shah^#, Richard Dobson^# > ^*#Joint authors

doi: 2020.04.24.20078006

• This repository provides pre-trained models to validate models in the medRxiv pre-print.
• Please get in touch if you would like to collaborate on this replication (clinical contact: jamesteo@nhs.net; technical: ewan.carr@kcl.ac.uk).
• If you use code/trained models from this repository, please cite the pre-print as a condition of use.

The file replicate.py will fit a series of models using pre-trained models. Specifically, it:

1. Imports a CSV file containing the required features and outcome.
2. For each feature set, it loads a pre-trained model (see [here][training/trained_models]) and tests this on the new data.

Some notes:

• The code does not perform any training or cross-validation, with the exception of KNN imputation, see below.
• Some code for data cleaning is provided (cleaning.R) but this is quite specific to the structure of the source data. It should demonstrate how we prepared the training and validation datasets, but will likely require modification before running on replication samples.

• All models are written in Python using scikit-learn. A minimal set of packages is required (pandas, numpy, scikit-learn; see requirements.txt).

• The recommended way to use this repo is in a virtual environment. If you’re using conda, you can use:

  conda env create -f conda.yml
  

• If you're using Singularity, you can use:

  singularity build python.simg singularity.def
  

See conda.yml and singularity.def in the software folder.

We have provided some simulated data for testing purposes (simulated.csv). This is randomly generated data based on the means/SD of the training sample. For example, you can fit all models to the simulated dataset:

git clone https://github.com/ewancarr/NEWS2-COVID-19
cd NEWS2-COVID-19
conda env create -f software/conda.yml
conda activate news2
python replicate.py

• The replicate.py script requires a dataset (validation.csv) containing the measures in the table below.
• Not all columns are required for replication.
  • At a minimum, you need: y, age, news2, crp_sqrt, neutrophils, estimatedgfr, and albumin.
  • This would allow validation of the supplemented NEWS2 score model from the paper (i.e. Models 1 and 5 from Table 3, p. 19).

The study cohort was defined as:

• All adult inpatients testing positive for SARS-Cov2 by reverse transcription polymerase chain reaction (RT-PCR);
• all patients included in the study had symptoms consistent with COVID-19 disease (e.g. cough, fever, dyspnoea, myalgia, delirium).
• We excluded subjects who were seen in the emergency department but not admitted.

• The training sample included patients testing positive for SARS-Cov2 between 1^st and 30^th March 2020. The external validation sample included patients testing positive on/after 31^st March 2020.
• Where possible, these timeframes should be used in replications.

The primary outcome (y) was patient status at 14 days after symptom onset, or admission to hospital where symptom onset was missing, categorised as:

1. Hospital admission but no transfer to ICU or death (WHO-COVID-19 Outcome Scale 3-5; y = 0)
2. Transfer to ICU or death (WHO-COVID-19 Outcome Scales 6-8; y = 1)

If date of symptom onset is unavailable it may be necessary to impute or use date of hospital admission. Most endpoints happen at 10-14 days, so you may have to work out how to define this.

Please refer to the above table for details of the required measures. The aim of this analysis is to assess the improvement in predictive performance achieved by supplementing NEWS2 with a small number of blood/physiological parameters. The base model includes:

age, news2, crp, neutrophils, estimatedgfr, albumin

A secondary model, if data are available, additionally includes:

oxsat, troponint, lymphocytes_log10

The provided script (replicate.py) includes several other feature sets which can be estimated subject to data availability.

All features must be measured at or shortly after hospital admission (within 24 hours).

• During training, missing feature information was imputed using KNN imputation (sklearn.impute.KNNImputer).
• For the purposes of temporal external validation, as presented in the paper, we used fitted imputation model to impute in the validation dataset.
• However, this repository does not provide the pre-trained KNN model, since the fitted object contains data that cannot be shared publicly.
• Therefore, replicate.py will train the KNN imputation on the provided validation dataset (see here).

The below table can be used to compare/comment on differences between training and replication samples.