At Cato Unbound, Dan Gardner and Philip Tetlock observe that expert predictions often miss the mark:
In the most comprehensive analysis of expert prediction ever conducted, Philip Tetlock assembled a group of some 280 anonymous volunteers—economists, political scientists, intelligence analysts, journalists—whose work involved forecasting to some degree or other. These experts were then asked about a wide array of subjects. Will inflation rise, fall, or stay the same? Will the presidential election be won by a Republican or Democrat? Will there be open war on the Korean peninsula? Time frames varied. So did the relative turbulence of the moment when the questions were asked, as the experiment went on for years. In all, the experts made some 28,000 predictions. Time passed, the veracity of the predictions was determined, the data analyzed, and the average expert’s forecasts were revealed to be only slightly more accurate than random guessing—or, to put more harshly, only a bit better than the proverbial dart-throwing chimpanzee. And the average expert performed slightly worse than a still more mindless competition: simple extrapolation algorithms that automatically predicted more of the same.
But this should not in any sense be read as a case for what-do-they-know populism. On one level, it’s a call for suspicion about ideologically driven predictions:
Cynics resonate to these results and sometimes cite them to justify a stance of populist know-nothingism. But we would be wrong to stop there, because Tetlock also discovered that the experts could be divided roughly into two overlapping yet statistically distinguishable groups. One group would actually have been beaten rather soundly even by the chimp, not to mention the more formidable extrapolation algorithm. The other would have beaten the chimp and sometimes even the extrapolation algorithm, although not by a wide margin…
One group of experts tended to use one analytical tool in many different domains; they preferred keeping their analysis simple and elegant by minimizing “distractions.” These experts zeroed in on only essential information, and they were unusually confident—they were far more likely to say something is “certain” or “impossible.” In explaining their forecasts, they often built up a lot of intellectual momentum in favor of their preferred conclusions. For instance, they were more likely to say “moreover” than “however.”
The other lot used a wide assortment of analytical tools, sought out information from diverse sources, were comfortable with complexity and uncertainty, and were much less sure of themselves—they tended to talk in terms of possibilities and probabilities and were often happy to say “maybe.” In explaining their forecasts, they frequently shifted intellectual gears, sprinkling their speech with transition markers such as “although,” “but,” and “however.”
Using terms drawn from a scrap of ancient Greek poetry, the philosopher Isaiah Berlin once noted how, in the world of knowledge, “the fox knows many things but the hedgehog knows one big thing.” Drawing on this ancient insight, Tetlock dubbed the two camps hedgehogs and foxes.
On another level, it’s an empirical research project:
Consider a major new research project funded by the Intelligence Advanced Research Projects Activity, a branch of the intelligence community.
In an unprecedented “forecasting tournament,” five teams will compete to see who can most accurately predict future political and economic developments. One of the five is Tetlock’s “Good Judgment” Team, which will measure individual differences in thinking styles among 2,400 volunteers (e.g., fox versus hedgehog) and then assign volunteers to experimental conditions designed to encourage alternative problem-solving approaches to forecasting problems. The volunteers will then make individual forecasts which statisticians will aggregate in various ways in pursuit of optimal combinations of perspectives. It’s hoped that combining superior styles of thinking with the famous “wisdom of crowds” will significantly boost forecast accuracy beyond the untutored control groups of forecasters who are left to fend for themselves.
Other teams will use different methods, including prediction markets and Bayesian networks, but all the results will be directly comparable, and so, with a little luck, we will learn more about which methods work better and under what conditions. This sort of research holds out the promise of improving our ability to peer into the future.
In his response, Robin Hanson suggests that expert predictions aren’t really about trying to predict the future. They’re about signaling group affiliation and building authority. Experts make predictions — and more importantly, non-experts demand predictions — so that they can seem smart and well-connected. Branché , as the French would say. (See how easy signaling is? And how fun!)
Robin suggests bringing back the pre-Victorian attitude about wagering. If you put money on it, you’ll be relatively more motivated by accuracy and relatively less by making a good impression. If we think it dishonorable to predict without a wager, we will see a lot more accuracy, if such is anywhere to be found.
I’d like to see an Elo rating system for head-to-head forecasting challenges. Unlike wagering, a rating system in which you essentially wager non-monetary rating points wouldn’t implicate the law at all.
I’m not sure the institutional support would exist for it, but saying that you’re a grandmaster-level forecaster would be one heck of a credential.
I need to read this whole thing. Thanks, Jason!Report
I’m glad to see the comments are working. I was expecting three hundred or so by now, most of them accusing me of being a corporate stooge, and I was sure the system had broken.Report
I’m on vacation. Don’t know what excuse these other laggards have.Report
You corporate stooge!
Feel better?Report
Robin suggests bringing back the pre-Victorian attitude about wagering.
Wagering, hell, bring back the code duello. Because an armed society is an accurate society.Report
I’d just like to suggest the other possibility that chimps are developing abilities far greater than those of humans and we’d better start stockpiling guns right now.Report
I’d think the “grandmaster-level forecasters” profit from their analyses instead of give it away for free or nearly so.Report
But this should not in any sense be read as a case for what-do-they-know populism. On one level, it’s a call for suspicion about ideologically driven predictions:
What about ideologically driven suspicions?Report
If Tetlock is right, then it doesn’t much matter how you come by your suspicions about expert forecasting. Just have them, and you’ll get fooled a lot less often. For what it’s worth, I don’t think he’s much of an ideologue, either.Report
For what it’s worth, I don’t think he’s much of an ideologue, either.
He certainly isn’t.Report
Like philosophy, it’s all in identifying and clarifying the issues and variables. Weighing them toward a conclusion, a prediction, is where mileage varies. Perhaps the expert is no better—or worse because of his prejudices—than the average joe.
We certainly hope that a president is wiser than his advisors, his experts, since ideally his experts will present conflicting predictions/recommendations and the exec must choose between them.
I like Dick Morris’ analyses of things. Can’t remember any of his predictions being right, ever. Krugman probably figures in here too, predicting 9 of the last 2 recessions, or however that riff goes.
It could be that keen analysis and conclusion-drawing are separate talents, that deriving usable information isn’t the same thing as using it. Skill is not synonymous with wisdom. Add in bias on the part of the analyst, and bad conclusions can follow more often than not.Report
I agree you should have suspicions. My point, admittedly too oblique, is just that the term/epithet “ideologue” is over used — these days anyone with a political point of view of any sort is routinely labelled an ideologue by anyone not sharing that point of view. Of course, those who engage in that sort of labelling are often just projecting their own flaws on others. There are those who imagine themselves quite free of any sort of prior commitment, but then that just opens them up to Keynes quote about “practical men” as the “slaves of some defunct economist” (or philosopher or worse).Report
I agree that opportunities to profit from accurate forecasting would be a huge win – as opposed to just having a system where people are mostly paid for what the payers want to hear.
It’s too bad there seems to be a lot of animosity and murky legal issues around forecasting markets.
What I observe is mainly that people predict not what they want the world to be but an outcome that flows from the way they want the world to work. So conservatives predict economic catastrophe from tax hikes, liberals predict human suffering catastrophe from spending cuts. It’s not that they want the disaster, it’s that they want the cause and effect to work that way.Report
And everyone predicts catastrophe as a way of saying “Listen to me! This is important!”.Report
Tetlock Alert: Government by “experts?” Uh-oh.
The basic point of the op-ed is that, as political scientist Ken Kersch puts it in a great scholarly review of [Stephen] Breyer’s book “Active Liberty”, Breyer’s intellectual roots are less in the sort of modern liberalism that animated the likes of William Brennan, and more “in pre–New Deal, early twentieth century progressivism, an outlook with an animating faith in government by expert, acting as stand-ins for the (uninformed) people at large.”
http://volokh.com/2011/07/14/follow-up-on-breyer-op-ed/Report
Am I the only one here who sees the infinite regress problem?
How do we assess the assessors?
Also, Jason, I recall The One Best Way did not take kindly to Nassim Nicholas Taleb’s book The Black Swan, the main contention of which is that society places far too much confidence in “experts”.Report
I’ve been thinking a lot about this during the Arab Spring. I’m convinced that you can’t predict revolution with any accuracy. Every society has simmering tensions. They require a critical mass of people, and that mass is a sum of each individual’s decision whether or not to act. When the next big one occurs, people will be blaming the intelligence services for not seeing it coming, others will be saying that they saw it coming, when really everyone who studied the situation knew the underlying reason for the revolution but no one could give a sound reason for believing that critical mass would occur at a specific moment.
I had the same thoughts during the last NCAA tournament, and I think the same analysis applies. So I think the next question is, what can be forecasted? Stocks, basketball, and elections if the forecaster has access to better information. (Of course, that begs the question, “what information is better?”.) Revolutions, inflation, and war are more the sum of invididual wills, and I don’t think that forecasting those is possible.Report
Remember the lefties douchebagging Glenn Beck for advertising buying gold?
No, I guess they don’t. But now all the teabaggers got gold at 100% profit, and the lefty green energy investments…oh, wait, I bet there weren’t any. Lefties only put somebody else’s money where their mouth is.
This is just too rent-seeking fucking funny:
http://www.altenergystocks.com/archives/2011/07/ges_mark_vachon_gas_is_massive.htmlReport