Torture Numbers and They’ll Confess to Anything
Photo by Theschutzcompany via Wikimedia Commons
We don’t know how many people have died of Coronavirus.
It seems weird, doesn’t it?? In our world with cutting edge medical technology and the ability to collect and track information at the touch of our fingertips, not to mention advanced mathematical methods to analyze statistical data, we are still unable to say exactly how many have died of COVID-19.
So why is it, when we look back at the past, we read some numbers in a history book and assume they mean anything? Why do we feel we know with certainty how many children died of polio in 1940? Why do we feel we know with certainty how many people died of smallpox in 1640? We don’t even know how many people are dying of coronavirus today!
And given our experience with the coronavirus, why should we assume numbers are telling us anything reliable in the present day? After all, the numbers told us Hillary Clinton was going to be the president. The numbers told us that a high carb, low fat diet would be good for us. The numbers told us that people younger than 80 wouldn’t even really be affected by COVID-19.
Numbers lie, and even when they’re not lying outright, they can mislead. They make us believe things as fact when in fact the reality is entirely different. Numbers were once used to “prove” that women and minorities were mentally inferior to white males. Numbers were used to “prove” that babies and animals don’t feel pain (they do). Numbers have been used to “prove” all sorts of things, from the horrifying to the banal. The numbers told me that it never frosts in my area on May 4, and I planted zucchini on that basis. Last night it was 26 degrees. The numbers told me that pea seeds cannot sprout when they’re more than 5 years old, but I tested some old seeds I found just to see, and they sprouted like they were fresh. A 103 year old woman survived coronavirus!
Science-minded people understand about averages and outliers and playing things conservatively (don’t rely on those 12 year old pea seeds to feed you, even if they may sprout and grow) but science-minded people need to also understand that most folks aren’t science-minded. Most folks listen to numbers and hear certainty and guarantees. Most folks think numbers are something concrete like, well, concrete, and not something malleable like clay or flowing like water, even though numbers are both malleable and flowing.
There is something incredibly comforting about looking at some data and thinking “ok, these numbers are illustrating to me the truth of this uncertain and dangerous world in which I live”. There is something incredibly comforting about looking at some data and feeling that you’re in control of something that is inherently uncontrollable, that the numbers have proven the scary thing cannot hurt you. But numbers can tell us only averages and probabilities. They’re clumsy and imprecise. Predictions cannot explain outliers and random fluctuations and meteor strikes and black swans, and furthermore, predictions are very often wrong. Numbers don’t deserve the faith we put in them.
Despite their failings, we use numbers not only to tell us about the past and the present, but to predict the future. We assume we know what the temperature will be in 100 years. We assume we know how many roads to build and where to build them. We assume we know how much toilet paper grocery stores will need to have on hand in any given week. We assume we know what the economy will do because we shut it down and we assume we know how coronavirus would have spread if we hadn’t. We make social policy based on these numbers, and it is very, very easy for people who set social policy, who think they know what’s best for us all, to capitalize on the trust and faith folks put in numbers rather than explaining their limitations.
But the truth is we don’t really know anything. All we can do is make our best guess based on the tools we have, and numbers are just a tool. Tools operate best in the hands of a skilled person using them for their proper task. Like a pipe wrench, numbers can be used to fix something that is broken or to bash in a skull depending on the intent of the person wielding it.
Science-minded people need to be better at explaining the limitations of data both to the people in charge of the government, and to the average guy and gal on the streets, even when it makes the experts’ job harder, even when it forces them to reveal how little even the experts actually know about the world, and that unknowable entity, the future.
Because putting blind faith in numbers is never a good thing. Just ask my zucchini.
We don’t understand things without a narrative, though.
We don’t understand them with a narrative, of course… but without a narrative we don’t even feel like we understand them.Report
But you CAN communicate complicated scientific principles in the form of a narrative. I know this because I do it every day. I see other people who are professionals in the same field as I am who choose not to do that, because it’s easier not to. It’s much, much easier for medical professionals, for example, to just cut to the chase and tell people “do this, not that” and use numbers as proven evidence even though the numbers they use are unreliable or even debunked.Report
We do this all the time. We do it until we are sick of it. We do it until we are blue in the face.
And we get two pushbacks.
The first (from the policymakers and politicians) is that they don’t want to hear about our uncertainty. They want facts and figures they can make decisions from and they regard uncertainty and caveats and assumptions as detracting from the certainty they need to function, because they want that certainty as a shield to political negatives.
The second pushback – which comes from all quarters and in many voices – is that people’s “gut” or “common sense” tells them something different, and all our caveating and uncertainty discussing seems to them like we scientists don’t know what we’re doing because we WON’T give them “A NUMBER” (as opposed to we can’t give them one number because thats not how statistics work).
So, lets make sure we put the onus on all the parties and not just the scientists. effective communication on this is a two way street, and no matter what we scientists do if the politicians and the public don’t buy what we are selling that’s also on them.Report
And this is why I very carefully chose the word “science minded people” rather than scientists There are many journalists, writers, public health specialists, politicians, etc who are science minded, indeed, even reporting, writing about, or setting public policy scientific issues, who outwardly misrepresent numbers and data because it’s easier.Report
I’d be interested in knowing who they are, and how you know, so I can avoid them in the future.Report
I know, because I have spent the last 12 years working in the field of fertility nutrition with tens of thousands of people worldwide who are told things by naturopaths, nurses, doctors, and even highly qualified medical specialists that are completely untrue.
I know because over my life I’ve done tons of professional-level research and reading into medicine, nutrition, and health issues, and yet I see firsthand and often journalists and even health workers reporting completely misleading news about studies that are not proven and are later debunked (and we’ve all seen this, haven’t we?? You read the fine print and realize that the study in question really isn’t at all definitive, but it’s written up in every major news organization)
Additionally, I know because of my own experiences as a patient with a medical mystery illness when I was repeatedly told things that were untrue or misleading, that were told to me by doctors to hurry me out of the office with reassurances that they couldn’t give, only to find out that I did actually have a health problem all along despite their claims and empty reassurance.
As for how to avoid them, you just have to be willing to hunt down everything yourself to see if what they say is true, rather than accepting anything at face value, EVEN when they’re telling you what you would very much like to be true. Most people don’t have the time or ability to do that, and that’s why I wish “science minded people” would try a little harder to communicate the uncertainty of what is suspected vs. what is known.Report
The general point is indisputable, but anybody in particular I should watch out for, like anyone anyone might care about with an established record for misrepresentation?Report
Also, you may not be aware of this but there are many people who most laymen would deem scientists – doctors, nurses, nutritionists, health educators, etc. who absolutely DO commit these types of numerical shenanigans with great frequency because they are in a rush, because they learned something back in medical school and haven’t bothered to keep current with research, or in some cases because they really don’t understand the data themselves. These are the people tasked with doing the most communicating with patients, and so even though there are of course many scientists who don’t make these types of errors themselves, they absolutely do exist and cause serious and lasting harm to the confidence of laymen in the ability of data to describe reality.Report
This is also true in business. In fact, it’s an important skill to learn: how to communicate with “business types.”
Of course, it helps a lot if the business types learn to communicate with the engineers. It’s a give and take. When there is conscientiousness on both sides, along with good faith, things go well.
I think it’s part of a good company culture, when the suites and the nerds can speak each other’s language and learn how to “bottom line” things for each other.Report
Make no mistake, that is the real reason they want to play ‘hear no evil’ about uncertainty. Because people make decisions based upon uncertain data all the time, and enjoy the benefits of getting it right, or suffer the consequences of getting it wrong. Except politicians, who are, as a group, largely cowards*.
*PS I place a lot of CEOs, etc into the same group.Report
Good post. I would add, however, that in this instance I think the media is failing more than scientists.
I’ve watched/listened to a ton of longer-form interviews with epidemiologists/researchers/statisticians these past 6-8 weeks, and I can’t think of one the didn’t emphasize over and over that the data they had so far was incomplete and flawed, and that it would be a long time before we could say very much for certain. But when I read or see media reports about whatever models/info they are presenting, I almost never see that caveat. It’s just presented as “sciences say that…”
It’s a really dangerous thing to be cavalier about. You can already see the public’s faith in epidemiological science starting to erode.Report
I was making a distinction (that I evidently should have been clearer about) where I used “science minded” to encompass science writers, the media, any number of health care professionals, health educators, even educated laymen. There are PLENTY of people, not only in the media but in many fields who use shortcuts and shorthand and rely on dubious data to just basically quickly get people to do what they say, painting themselves as all-knowing experts since it is a heck of a lot easier and quicker than actually explaining “here is what we think, but we don’t KNOW it as a fact”.Report
I would push very hard about calling people who who use shortcuts and shorthand and rely on dubious data to just basically quickly get people to do what they say, painting themselves as all-knowing experts since it is a heck of a lot easier and quicker than actually explaining “here is what we think, but we don’t KNOW it as a fact”. “science minded”
By definition, every science minded person understands how science and the scientific method work. And they know that “certainty”, like “Truth”, are Platonic ideals that cannot be reached, just approximated to. Painting yourself as an all-knowing expert and using dubious data, well, that’s more what stable geniuses doReport
People who purport to be scientists, who work in the field of science/medicine, who report on science and claim to respect science have to be called something, and I chose to call them “science minded.”
I disagree that every “scientist” (or at least the people who most laymen consider scientists, you may quibble but I’m talking about public perspective) really knows, internally how the scientific method works. You’re giving them credit that they may not deserve, and additionally their motives are not always universally pure. I’ve seen a few too many nutritionists with official titles telling people that the pH of their body can change and make them sick, and a few too many “holistic doctors” with MD’s peddling herbal remedies.Report
Every scientist (no scare quotes) does know how the scientific method works. As does every good engineer, and every good MD, and even every nutritionist, I’ve ever met professionally or socially .
I cannot vouch for how people (or quacks) present themselves, particularly if they are selling something to “laymen”(scare quotes intended)Report
I see this as a viscous circle of fear.
Scientists give data with caveats.
Media discard caveats in favor of varying degrees of sensationalism.
Public gets scared and increases ad revenue, etc.
Scientists update models with more caveats.
Media repeats previous.
Public starts to not trust scientists.
Media gets scared and ups the fear mongering because they still want the public to trust the scientists, but selling fear is pretty much all the media knows how to do these days.Report
“Science-minded people need to be better at explaining the limitations of data both to the people in charge of the government, and to the average guy and gal on the streets, even when it makes the experts’ job harder, even when it forces them to reveal how little even the experts actually know about the world, and that unknowable entity, the future.”
Agree with this. However, there’s a problem. People who talk about uncertainties and limitations tend to sound wishy-washy. The public has a lot more trust in someone who states things definitively, even he’s massively and frequently wrong.Report
Another problem we have is that the output of any mathematical model produces as many significant figures as an astronomer’s plot of planetary positions (as NASA might use to plot a deep space mission), because that’s how equations and computers work. Better presentations show error bars, which is good but even those are essentially a wild guess because it’s all being based on human behavior and we’re intervening constantly in what’s unfolding.
The Newtonian wind-up universe model died when we realized his perfect equations quickly became useless at plotting the future position of billiard balls because each collision acted as an massive error multiplier. So that buy collision three or four the calculation was garbage, since the future state of some systems is essentially unpredictable due to their nature.
“By June 1st our state is going to have 8,500 to 9,500 cases” is actually less “true” than saying something like “Well, we’re holding out pretty well, and if we stay the course I think things might be a little bit better come June, God willing and the creek don’t rise.” But which statements sounds smarter and more authoritative?
Yet politically, the governors making the second type of statement will likely retain far more public trust than the ones making the first, whose accumulating list of failed predictions make them look like the Emperor with no clothes, while the second were the ones who suffering through the uncertainties and made hard calls with hope tempered by caution, just like the rest of us.Report