Are the Coronavirus Models Too Pessimistic?

Unfortunately, if anything, reality has often turned out somewhat worse than predicted.

The White House keeps thinking that the major coronavirus models are too pessimistic and that things will turn out better than they predict.  The Administration is wrong to downplay the models’ forecasts. Rather than being too pessimistic, models have frequently erred in the direction of optimism. Despite continuing uncertainties, there’s no reason to discount their prediction downward.

Fivethirtyeight.org has a nifty feature where you can check out what some leading models were predicting as of various dates. That allows comparisons between model forecasts made at different times covering different dates. The results are instructive, as shown by the table at the end of this post.   Here are the main-takeaways.

Over-optimism by models. If anything, the models haven’t been pessimistic enough, particularly the IMHE model preferred by the White House. Predictions for deaths two weeks in the future have often turned out to be too low.  This could be an indication that social distancing has been less effective than expected or less rigorous than necessary. Or there’s something else we don’t understand about the disease.

Models are constantly tweaked. You can see that from the predicted number of deaths for April 25 made by each model ten days earlier and four days earlier.  There’s even a shift in predictions for late May between predictions made on May 1 and those made on May 5, which was quite significant for some models. (In fact, on Monday, the IMHE upped its August 4 prediction from 74,000 to 135,000).  These changes make it a little hard to know which modeling approach is better. 

There’s been notable convergence. If you compare the predictions for two weeks out (and now four weeks out), there seems to be a clear trend toward closer clustering.  Hopefully, this is a sign that learning and improved date are increasing the validity of predictions.

One more important thing to note.  All of these models have been operating under conditions where social distancing rules were in effect across the country.  As those rules are relaxed, the modelers will have to figure out how to modify the models.  Given the lag time between new infections and deaths from the disease, we probably won’t see the relaxations fully reflected in the death rates until late May. Because the amount of testing and the testing criteria are constantly shifting, it may be hard to interpret changes in the number of new cases observed.  Changes in hospitalizations will lag by a week or ten days, but may be more revealing.

The Trump Administration seems a bit adrift right now.  A recent draft report by staff does consider the possibility that things could get a lot worse.  Yet the White House also favors a home-brewed model by one of its economists that “shows deaths dropping precipitously in May — and essentially going to zero by May 15.”  The Administration needs to stop cherrypicking models and recognize the possibility that reality may end up worse, not better, than some established models predict.

Table

Key: IMHE (University of Washington model), UT (University of Texas), NE (Northeastern), Colum. (Columbia).  “Date Listed” is the date when the prediction appeared on the 538.org listing.

coronavirus, models versus reality, public health, scientific uncertainty, Trump Administration