Mini-Troughput: The Triple Lindy Stats Flop
This week on the Twitters, a mini-controversy erupted over COVID-19 projections. The latest model projections for the toll of Coronavirus indicates that if social distancing is ended, cases will surge to 200,000 a day with 3000 deaths a day by June. However, the White House is skeptical, favoring much more optimistic projections. In response to the story, they tweeted this out:
To better visualize observed data, we also continually update a curve-fitting exercise to summarize COVID-19’s observed trajectory. Particularly with irregular data, curve fitting can improve data visualization. As shown, IHME’s mortality curves have matched the data fairly well. pic.twitter.com/NtJcOdA98R
— CEA (@WhiteHouseCEA) May 5, 2020
That shows the predictions of various models for the progression of the pandemic. The Institute for Health Metrics and Evaluation (IHME) models, which the White House had been favoring, are very optimistic and have come under fire. But even their most recent revision, which does not account for the end of social distancing, predicts that COVID will continue to ravage the country at this level for a few more weeks before slowing fading away (the green line). The one the White House now favors and wants to base policy on is the red line. This is a “cubic model” from economist Kevin Hassett that goes beyond even the optimism of the IHME model to predict that COVID deaths are about to fall dramatically, disappearing within the next couple of weeks.
If you heard a loud noise on Tuesday morning, that was probably the sound of every epidemiologist, every economist, every mathematician, every scientist and every virologist simultaneously face-palming and then crashing to the ground in a dead faint. You may have noticed shares in smelling salt companies surge on the Dow. The reason is because while that “cubic model” fit might look reasonable, it is utter absolute garbage. It is the kind of thing that would get a D if it were turned by a freshman in stats class.
Walk with me a bit.
How Data Becomes Prediction
Science lives in a dizzying sea of numbers that we call “data”. Data are the foundation stones of all science because they are what is real … or rather, what is measured. Data can be misleading. They can be wrong. They can be biased. But they are the tether between the reality of the world and our way of understanding that reality that we call “science”.
In the case of COVID, the data are the number of positive tests, the number of confirmed deaths, the number of hospitalizations. Those facts are a shadowy representation of the underlying reality — how many people have caught the virus, how many have gotten seriously ill and how many have died of it. There has been a great deal of debate about the connection between those two things — whether we are over- or under-counting COVID deaths, how many of the infected are not being tested, etc. These are what we call “biases” in the data. And they’re something you have to account for if you’re to understand what the data are telling you.
Assuming you can understand the biases, you can then use your data to breach the underground chamber that contains the fundamental questions you are trying to answer. In this case: How infectious is the disease? How much exposure does someone need to become sick? How deadly is the disease? How long will it be before someone knows they’re sick and seeks help? How will society respond to the disease? How will that response change from place to place? If you can get answers to those questions, even partial ones, you can start making predictions about what will happen, given certain assumptions.
Models are one of the tools we use to answer fundamental questions. They essentially turn the process around. If we assume certain answers to the fundamental questions, what would the data look like? Can we change those answers until the prediction of the model matches the data we have on hand? These models are not conjured out of the air but are based on prior knowledge … in this case, how previous pandemics have behaved.
Models are a contentious subject in science. My view is that they are useful tools but are slippery when they try to make predictions. The reason is that there are sometimes many models that can correctly “predict” what has already happened. The make-or-break point is whether they correctly predict the future. And the scientific literature is liberally littered with models that explained the past perfectly but then crashed and burned when they tried to predict the future.
The road from data to bias-free data to theory to prediction is long, winding and has many false branches. Scientists have devoted their lives to this. And many have been working round-the-clock on the COVID crisis to figure it out. The difficulty of prediction is why the models for COVID-19’s spread and ultimate toll have varied so much, especially in the early days when we knew a lot less than we know now. We knew that COVID-19 could infect millions. We knew it could kill hundreds of thousands. But just how many and just how fast depended a lot on your assumptions — about the disease itself and about society’s response.
And most epidemiologists have been up-front about this, stating their assumptions, their uncertainties and welcoming criticism. Models makers rarely have a single prediction but usually a variety of predictions for various possible courses. Figuring this stuff out is hard. It involves a lot of work, a great deal of knowledge, a familiarity with past epidemics, attention to your underlying assumptions and a significant amount of humility. But unless we plan on a return to reading sheep entrails, it is our only option.
Curve Fitting, For the Loss
The White House, however, has taken a different approach. From the beginning, they have favored models that predicted a much milder epidemic. But what they have now done, apparently, is throw over even these models in favor of putting the number of cases into a spreadsheet and applying a cubic fit from the pull-down menu. It’s a sophisticated version of connect-the-dots but where half the dots weren’t there so you just drew the lines where you think they should be. They are pretending that this constitutes a model.
But it doesn’t. A simple mathematical fit to data does not account for biases in said data. It tells you nothing about the underlying fundamentals. It reveals no underlying information about the disease or our society. If epidemiological models are a level, this is holding your thumb up, closing both eyes and saying, “Eh, looks fine to me.”
The simple truth is that you can rarely apply such simplistic analysis to real-life behavior. And you especially can not apply it to something that has already shown that it doesn’t behave that way. The infection curves we’ve seen in countries has been a sharp rise, a plateau and then a slow decline. We have not seen the sharp bell curve predicted above. This “cubic model” fit assumes that we have perfect information on the virus, which we do not have, and that the virus behaves in a way that it clearly does not.
What’s more, remember what I said about how models can crash and burn? Fitting a simple equation is a “model” that is particularly vulnerable to crashing and burning. The COVID epidemic is still in progress. We have not seen any fall in cases so far in the United States. As a result, that dramatic drop in case-loads is not based on any data at all. It is entirely theoretical, assumed because they used the equation to fit one side of the data. This is the equivalent of assuming that since one side of the Titanic doesn’t have any holes in, the other side must not either.
This comic from XKCD illustrates the problem of fitting mathematical formulae to data better than I ever could:
Note especially that last panel. The crazier the model, the bigger the assumptions, the wilder the results you get when you extend it beyond your data sample. If you’ve got a mathematical model that explains, say, stock market prices from 1950 to 2010, don’t be surprised if it predicts that the Dow hit -24,000 in 2012. It is intrinsic to mathematics that you get crazy predictions when you go outside your data sample. Mark Twain probably expressed it best, when talking about how engineering projects had shortened the length of the Mississippi River over the years:
In the space of one hundred and seventy-six years the Lower Mississippi has shortened itself two hundred and forty-two miles. That is an average of a trifle over one mile and a third per year. Therefore, any calm person, who is not blind or idiotic, can see that in the Old Oolitic Silurian Period,’ just a million years ago next November, the Lower Mississippi River was upwards of one million three hundred thousand miles long, and stuck out over the Gulf of Mexico like a fishing-rod. And by the same token any person can see that seven hundred and forty-two years from now the Lower Mississippi will be only a mile and three-quarters long, and Cairo and New Orleans will have joined their streets together, and be plodding comfortably along under a single mayor and a mutual board of aldermen. There is something fascinating about science. One gets such wholesale returns of conjecture out of such a trifling investment of fact.”
Another illustration of the point:
According to predictions from my “cubic model” fitted on yesterday’s data, today’s temperature is approaching absolute zero and we are all dead, mercifully. pic.twitter.com/jfNgpa2wBH
— John Voorheis (@john_voorheis) May 5, 2020
— Kareem El-Badry (@kjb_astro) May 6, 2020
Fit a cubic model to my total number of citations by year and I’m very excited to share I will have -850 citations by the year 2040 pic.twitter.com/0WEbgkfdQX
— Mark Hibbins (@mhibbinsbiology) May 6, 2020
I should note that this is not Hassett’s first goat rodeo with this sort of thing. He was co-author on a book that predicted the Dow would hit 36,000. And he used similar “fitting” to argue that tax cuts would pay for themselves:
It’s come to my attention the “economist” in charge of this ridiculous claim is the same one who made this pants on head level dataviz circa 2004. This in monstrous incompetence during a pandemic pic.twitter.com/7Xax48YZEw
— Sean – now with 10% more social distancing! (@SeanfromSeabeck) May 5, 2020
The White House isn’t the only group pulling this sort of flim-flammery on COVID-19 either. This website contains similar fits to the data from a variety of countries. But while some countries (the Netherlands, France, Germany) have followed that curve shape reasonably, others (Italy, the US) have not. Moreover, the models keep changing. Two weeks ago, they predicted the virus would be mostly gone by mid-May and entirely gone by Labor Day. Today’s analysis shows that extending out to June and October, respectively.
This touches on one of the subtle deceptions of the White House tweet. They show multiple IHME models that have changed as more information on the epidemic has come in. But they show only one cubic fit. This would give you the impression that even the IHME, as optimistic as they are, doesn’t know what they’re doing and the cubic fitters do. What is ignored is that if you had run their “cubic model” say, a month ago, it would have predicted that the epidemic would be over by now. Simplistic curve-fitting is extremely vulnerable to assuming that however many cases you have right now — be it two, two thousand or two million — is the peak. And that you’ll get back to zero as fast as you got there from zero.
In the end, the reason the White House is taking this “model” seriously is because it’s telling them what they want to hear, what even the IHME won’t tell them: that we can end the lockdowns and the virus will simply disappear within a couple of weeks. There is absolutely no reason to believe this. Social distancing and lockdowns have brought the infectiousness of the virus down to just about 1 so that the virus is leveling off. But they haven’t brought it below 1, so that the virus starts to fade1 Models may come and go. But if history and this pandemic has taught us anything, it’s that “opening the economy” is not going to drop the infection rate.
- To be fair, treating the United States as one giant block of data is silly. We are a far-flung nation with a broad range of responses to this. And the virus is doing different things in different places. The NYT has a breakdown by state that shows that, in places like New York, the virus peaked a few weeks ago and is now in decline while in places like Iowa, it’s still on the rise.