A case study in model failure? COVID-19 daily deaths and ICU bed utilisation predictions in New York state
- PDF / 3,067,927 Bytes
- 10 Pages / 595.276 x 790.866 pts Page_size
- 36 Downloads / 141 Views
COVID-19
A case study in model failure? COVID‑19 daily deaths and ICU bed utilisation predictions in New York state Vincent Chin1,2 · Noelle I. Samia3 · Roman Marchant1,2 · Ori Rosen4 · John P. A. Ioannidis5,6,7,8,9,10 · Martin A. Tanner3 · Sally Cripps1,2 Received: 14 June 2020 / Accepted: 21 July 2020 © Springer Nature B.V. 2020
Abstract Forecasting models have been influential in shaping decision-making in the COVID-19 pandemic. However, there is concern that their predictions may have been misleading. Here, we dissect the predictions made by four models for the daily COVID-19 death counts between March 25 and June 5 in New York state, as well as the predictions of ICU bed utilisation made by the influential IHME model. We evaluated the accuracy of the point estimates and the accuracy of the uncertainty estimates of the model predictions. First, we compared the “ground truth” data sources on daily deaths against which these models were trained. Three different data sources were used by these models, and these had substantial differences in recorded daily death counts. Two additional data sources that we examined also provided different death counts per day. For accuracy of prediction, all models fared very poorly. Only 10.2% of the predictions fell within 10% of their training ground truth, irrespective of distance into the future. For accurate assessment of uncertainty, only one model matched relatively well the nominal 95% coverage, but that model did not start predictions until April 16, thus had no impact on early, major decisions. For ICU bed utilisation, the IHME model was highly inaccurate; the point estimates only started to match ground truth after the pandemic wave had started to wane. We conclude that trustworthy models require trustworthy input data to be trained upon. Moreover, models need to be subjected to prespecified real time performance tests, before their results are provided to policy makers and public health officials. Keywords COVID-19 · Hospital resource utilisation · Model evaluation · Uncertainty quantification * Sally Cripps [email protected] 1
ARC Centre for Data Analytics for Resources and Environments, Sydney, Australia
2
School of Mathematics and Statistics, The University of Sydney, Sydney, Australia
3
Department of Statistics, Northwestern University, Chicago, USA
4
Department of Mathematical Sciences, University of Texas at El Paso, El Paso, USA
5
Stanford Prevention Research Center, Stanford, USA
6
Department of Medicine, Stanford University, Stanford, USA
7
Department of Epidemiology and Population Health, Stanford University, Stanford, USA
8
Department of Biomedical Data Sciences, Stanford University, Stanford, USA
9
Department of Statistics, Stanford University, Stanford, USA
10
Meta‑Research Innovation Center at Stanford (METRICS), Stanford University, Stanford, USA
Introduction I don’t have a crystal ball. Everybody’s entitled to their own opinion, but I don’t operate here on opinion. I operate on facts and on data and on numbers
Data Loading...
