RALEIGH – If there is one newspaper editorial you read this week, make it the July 15 piece in the Wall Street Journal entitled “The More You Hear, The Less You Know.” If you are not a subscriber, then unfortunately you can’t read the piece online, so go out to your local library and read the print edition. You can get the flavor of the editorial, and of the journal article on which it is based, by reading this summary from the National Center for Policy Analysis.
Essentially, it is a cautionary tale about believing every news report you hear touting a “breakthrough new study” that “proves beyond a doubt” something or other. The example here is medical, but the same insight applies to many other fields of study, including public policy.
The underlying paper, printed in the Journal of the American Medical Association, took a look at 49 widely touted studies reporting the results of clinical trials. Of those 49, 45 purported to provide evidence supporting the effectiveness of a medical intervention – be it a pharmaceutical, a procedure, or something else. The authors of the article considered whether follow-up studies had offered evidence supporting the original finding. That’s the way science is supposed to work: scientists form an hypothesis based on existing evidence or inferences, then test the hypothesis with a controlled experiment, then they or others attempt to replicate the findings of the initial experiment.
Of the 45 cases studied, there were 14 instances in which subsequent studies contradicted rather than supported the initial findings. For example, a study found that hormone therapy reduced the incidence of coronary artery disease in women. But subsequent research linked hormone therapy to an increased risk of coronary artery disease among women. Somewhat less distressing was the example of Vitamin E, which initial research suggested might be useful as a heart protector but in subsequent studies demonstrated little effect.
The proper conclusion here is not that there are lots of incompetent scientists out there fabricating results or publishing sloppy work. It is, instead, that researchers, regulators, the media, and the general public should look before they leap. The nature of science is to advance propositions that may be disproved through experiment but are not necessarily proved by experiment, and certainly not by a single experiment. We may never know for certain whether a particular medical therapy, or environmental remedy, or other action will achieve the goal we seek. We can’t allow that to lead to perpetual inaction, naturally. But it is reasonable to wait for confirmation, to withhold judgment and forego costly interventions until our knowledge base is wider and deeper.
Sometimes the rule is: don’t just do something, stand there.
Hood is president of the John Locke Foundation.