From today's WSJ, just the first six paragraphs:
Wall Street Journal writes:
Most Science Studies Appear to Be Tainted By Sloppy Analysis
We all make mistakes and, if you believe medical scholar John Ioannidis, scientists make more than their fair share. By his calculations, most published research findings are wrong.
Dr. Ioannidis is an epidemiologist who studies research methods at the University of Ioannina School of Medicine in Greece and Tufts University in Medford, Mass. In a series of influential analytical reports, he has documented how, in thousands of peer-reviewed research papers published every year, there may be so much less than meets the eye.
These flawed findings, for the most part, stem not from fraud or formal misconduct, but from more mundane misbehavior: miscalculation, poor study design or self-serving data analysis. "There is an increasing concern that in modern research, false findings may be the majority or even the vast majority of published research claims," Dr. Ioannidis said. "A new claim about a research finding is more likely to be false than true."
The hotter the field of research the more likely its published findings should be viewed skeptically, he determined.
Take the discovery that the risk of disease may vary between men and women, depending on their genes. Studies have prominently reported such sex differences for hypertension, schizophrenia and multiple sclerosis, as well as lung cancer and heart attacks. In research published last month in the Journal of the American Medical Association, Dr. Ioannidis and his colleagues analyzed 432 published research claims concerning gender and genes.
Upon closer scrutiny, almost none of them held up. Only one was replicated.
It's obvious who's to blame for this sad state of affairs: THE MEDIA!
Why do I say that? Why aren't I excoriating the scientists?
The truth is, I'm happy to hear that there are researchers out there working hard to replicate prior research. Failure to replicate is a good thing. It means the prior finding is not confirmed, and that no scientific consensus will develop around it, which is a another good thing. We don't want scientists operating under the false assumption that bogus research results are true.
I blame the media because they're the one's giving a false imprimatur to unreplicated research findings. Long before a scientific finding has been replicated and tested and confirmed, they're filling the print press, airways and Internet with stories that begin with phrases like, "In this week's New England Journal of Medicine it was reported that...", and then they continue on to describe recent research results.
What is one to think when confronted with medical news presented in this compelling way? How many people think, "Well, it's just preliminary." Does anyone actually say, "We'll have to wait for more studies."
No, of course not, not in any meaningful numbers. What actually happens is that people say, "Oh my God, eating frabis is going to kill me, and as much as I love it I'm going to have to give it up."
Anyone who has lived on this planet long enough, or at least in the United States part of it, is fully aware of the whipsaw results of these medical reports. I still remember the cholesterol controversy. For years it was bad for you, many gave up eggs, only to find later that, well, I'm sure everyone knows the story.
So take your media reports about the latest scientific findings with a grain of salt, a whole salt mine in fact, especially in the field of medicine where you might actually be influenced to change your behavior. The scientists are out there doing their jobs building consensus around bodies of replicated research, and when there's legitimate new knowledge relevant for you then your doctor, assuming he's competent and not a quack or a chiropractor, will tell you about it.
--Percy
™ Version 4.2