Experts all the way down.

• Author: Tetlock, Philip E.
• Position: Wrong: Why Experts Keep Failing Us And How to Know When Not to Trust Them, Being Wrong: Adventures in the Margin of Error and Proofiness: The Dark Arts of Mathematical Deception - Critical essay

David H. Freedman, Wrong: Why Experts* Keep Failing Us And How to Know When Not to Trust Them (New York: Little, Brown and Co., 2010), 304 pp., $25.99.

Kathryn Schulz, Being Wrong: Adventures in the Margin of Error (New York: Ecco, 2010), 416 pp., $26.99.

Charles Seife, Proofiness: The Dark Arts of Mathematical Deception (New York: Viking Adult, 2010), 304 pp., $25.95.

Chafe though we might at our helplessness, even the most abrasive cynics cannot avoid relying on various species of experts--economists or oncologists or climatologists or ...--in forming opinions. The world is just too complex to decode it on our own. And as the world has grown evermore complicated, there has been a corresponding surge in demand for metaexpertise--for those who promise to guide us through the intellectual labyrinth, distinguishing the snake-oil salesmen from the fact profferers. Of course, just because we want something does not mean that it exists. But it is a good bet that people will pop up claiming they can deliver the answers gift wrapped tomorrow morning.

Thank goodness, it would seem then, that there are three new books by accomplished journalists David Freedman, Kathryn Schulz and Charles Seife to help us laymen figure out who is engaging in trickery-by-data-distortion and who not, so we can all better decode whom and what to trust.

I am often pigeonholed as an "expert on experts" because I have a long-standing interest in the (rather large) gaps between the confidence of political pundits and their forecasting accuracy. In reviewing these books by experts on experts, one could say I have morphed into an even-higher life form: an expert on "experts on experts."

Before ascending further on the Great Ladder of Being (all it would take is for a blogger to review this review--and for me to reply--to move up to Rung Number Five), I had better stop the infinite regress. Why not start by asking: How much traction can we get from heeding these authors' advice in coping with that recurring challenge of modern existence--separating the wheat from the chaff?

Each author offers what appear to be sensible, sometimes deeply insightful, guidelines. Seife warns us in Proofiness about how easily we can be seduced by an assortment of specious statistical arguments--and about the hazards of Potemkin numbers that give weak arguments an aura of credibility. He notes especially that we are never more vulnerable to this sort of "proofiness" than when false claims of precision cut along convenient ideological lines. In essence, when experts' sureties affirm our own beliefs (Democratic, Republican or otherwise), we are quick to take them on faith.

But we should be wary of almost all "facts" delivered to us (whether they jibe with our vision of reality or not), for as David Freedman documents in Wrong, experts know a lot less than they claim--and that this is, as Marxists were fond of saying, no accident. There are such powerful and perverse institutional incentives for experts to overclaim the validity of their data and their conclusions, we should not be shocked that many ambitious scientists succumb to the I-have-figured-out-all-the-answers temptation (indeed, the surprising thing is perhaps that so many resist the siren calls of media acclaim).

And Schulz explores in her own subtly seductive way the experience of "Being Wrong"; why it is so hard to admit error but how, with the help of brilliant philosophers and clever perceptual illusions, we might learn to stop taking ourselves so damned seriously and embrace William James's liberating insight: "Our errors are surely not such awfully solemn things. In a world where we are so certain to incur them in spite of all our caution, a certain lightness of heart seems healthier than this excessive nervousness on their behalf."

It is hard to argue with writers as attractively open-minded as this threesome--and I caught only a few errors, none consequential. And certainly if we all internalized the authors' collective wisdom we would be quite a bit better off. If we mastered the rudiments of statistical reasoning, recognized that experts are all too human and cultivated the fine art of self-interrogation, we'd reduce our exposure to some expensive mistakes.

On the personal side of the ledger, we would pay less money for dubious financial-services advice (and shift into low-transaction-cost, diversified mutual funds); we would be less easily swept up in panics over food additives or vaccinations; and, we would be less likely to gamble our lives on alternative medical treatments. Richer and healthier are not bad returns on an investment of less than $100 in books.

On the public-policy side of the ledger, we would be quicker to sense when we are being tricked by politicians who make their arguments in nominal dollars rather than inflation-adjusted ones, who imply that correlation means causation, and who insinuate that they have policy formulas that allow us to escape the tedious trade-offs that genuine domestic- and foreign-policy experts know all too well--those between consumption and investment, equality and efficiency, deterrence and reassurance.

And if all that were not reward enough, we could dodge a lot of silly watercooler arguments. For instance, Seife's discussion of "electile dysfunction" shows how meaningless it was to get hot and bothered over who really won the Bush-Gore--or the Franken-Coleman--election. Both were, despite the media brouhaha, to put it simply, ties. Many have a hard time grasping this truth because we have not deeply internalized the elementary statistical concept of measurement error. A result can be off a little here or off a little there; a definitive result is an impossibility, no matter how many chads we count. Indeed, there are some situations in which we can do no better than tossing a coin--an option that our highest courts could have safely considered if most citizens were as statistically savvy as Charles Seife.

So, there are tangible senses in which we would be better off if we were all "smarter"--and "smarter" about what it means to be smart (it is not just processing speed and crunching power--it is a capacity for constructive self-criticism). But how feasible is it for the statistical layman? Can we truly become functioning educated skeptics armed with the...