Register | Sign In


Understanding through Discussion


EvC Forum active members: 65 (9162 total)
3 online now:
Newest Member: popoi
Post Volume: Total: 915,817 Year: 3,074/9,624 Month: 919/1,588 Week: 102/223 Day: 13/17 Hour: 0/0


Thread  Details

Email This Thread
Newer Topic | Older Topic
  
Author Topic:   Great News: Most Scientific Research Findings are Wrong!
Percy
Member
Posts: 22392
From: New Hampshire
Joined: 12-23-2000
Member Rating: 5.3


Message 1 of 6 (421813)
09-14-2007 4:06 PM


From today's WSJ, just the first six paragraphs:
Wall Street Journal writes:
Most Science Studies Appear to Be Tainted By Sloppy Analysis
We all make mistakes and, if you believe medical scholar John Ioannidis, scientists make more than their fair share. By his calculations, most published research findings are wrong.
Dr. Ioannidis is an epidemiologist who studies research methods at the University of Ioannina School of Medicine in Greece and Tufts University in Medford, Mass. In a series of influential analytical reports, he has documented how, in thousands of peer-reviewed research papers published every year, there may be so much less than meets the eye.
These flawed findings, for the most part, stem not from fraud or formal misconduct, but from more mundane misbehavior: miscalculation, poor study design or self-serving data analysis. "There is an increasing concern that in modern research, false findings may be the majority or even the vast majority of published research claims," Dr. Ioannidis said. "A new claim about a research finding is more likely to be false than true."
The hotter the field of research the more likely its published findings should be viewed skeptically, he determined.
Take the discovery that the risk of disease may vary between men and women, depending on their genes. Studies have prominently reported such sex differences for hypertension, schizophrenia and multiple sclerosis, as well as lung cancer and heart attacks. In research published last month in the Journal of the American Medical Association, Dr. Ioannidis and his colleagues analyzed 432 published research claims concerning gender and genes.
Upon closer scrutiny, almost none of them held up. Only one was replicated.
It's obvious who's to blame for this sad state of affairs: THE MEDIA!
Why do I say that? Why aren't I excoriating the scientists?
The truth is, I'm happy to hear that there are researchers out there working hard to replicate prior research. Failure to replicate is a good thing. It means the prior finding is not confirmed, and that no scientific consensus will develop around it, which is a another good thing. We don't want scientists operating under the false assumption that bogus research results are true.
I blame the media because they're the one's giving a false imprimatur to unreplicated research findings. Long before a scientific finding has been replicated and tested and confirmed, they're filling the print press, airways and Internet with stories that begin with phrases like, "In this week's New England Journal of Medicine it was reported that...", and then they continue on to describe recent research results.
What is one to think when confronted with medical news presented in this compelling way? How many people think, "Well, it's just preliminary." Does anyone actually say, "We'll have to wait for more studies."
No, of course not, not in any meaningful numbers. What actually happens is that people say, "Oh my God, eating frabis is going to kill me, and as much as I love it I'm going to have to give it up."
Anyone who has lived on this planet long enough, or at least in the United States part of it, is fully aware of the whipsaw results of these medical reports. I still remember the cholesterol controversy. For years it was bad for you, many gave up eggs, only to find later that, well, I'm sure everyone knows the story.
So take your media reports about the latest scientific findings with a grain of salt, a whole salt mine in fact, especially in the field of medicine where you might actually be influenced to change your behavior. The scientists are out there doing their jobs building consensus around bodies of replicated research, and when there's legitimate new knowledge relevant for you then your doctor, assuming he's competent and not a quack or a chiropractor, will tell you about it.
--Percy

Replies to this message:
 Message 2 by Modulous, posted 09-14-2007 4:11 PM Percy has replied
 Message 4 by Jon, posted 09-15-2007 1:06 AM Percy has not replied

  
Modulous
Member
Posts: 7801
From: Manchester, UK
Joined: 05-01-2005


Message 2 of 6 (421814)
09-14-2007 4:11 PM
Reply to: Message 1 by Percy
09-14-2007 4:06 PM


Further reading/debate
Either the same paper, or just a similar one from the same people, was discussed in most scientific papers are wrong?

This message is a reply to:
 Message 1 by Percy, posted 09-14-2007 4:06 PM Percy has replied

Replies to this message:
 Message 3 by Percy, posted 09-14-2007 6:17 PM Modulous has not replied

  
Percy
Member
Posts: 22392
From: New Hampshire
Joined: 12-23-2000
Member Rating: 5.3


Message 3 of 6 (421841)
09-14-2007 6:17 PM
Reply to: Message 2 by Modulous
09-14-2007 4:11 PM


Re: Further reading/debate
Different paper, same guy, Dr. John Ioannidis. According to the WSJ article, he's published "a series of influential analytical reports" making his point that the findings of most research papers, at least in the medical field, are not corroborated by further research.
For myself, I don't find this a bad thing. Just as we're somewhat amazed to discover how much pruning natural selection performs in the case of fish and turtles, it's equally surprising to find just how many scientific papers are in essence "deselected" by the scientific process.
It is often lamented that we don't see enough papers reporting negative results, that the results of such efforts often wind up in file drawers instead of journals. I think the criticism is a bit too strong, as any negative results must be kind of compelling for a journal to accept a paper reporting it. The reaction at most journals would be, "Here's a paper reporting yet another failed effort to fuzbat the wizsteins of norburs. Who cares. Rejected!"
And given the "publish or perish" environment in which many scientists work, I find completely unsurprising any efforts to place negative results in as positive light as possible, and to try to dredge at least some contribution, and even to over-analyze their data.
Dr. Ioannidis's reports don't make me concerned, and I don't think anyone else should be concerned, either. This is just the way science works, and it's also the way it has worked in the past. Just because someone is going to the trouble of reporting the details of just how messy a process science is doesn't mean we should be worried about it.
I'm more concerned about people like Dr. Ioannidis who go around raising alarms about things that are probably just about as they should be. This isn't to say we shouldn't strive to improve, we most certainly should, but the raising of alarms tends to cause pressures for solutions that are visible rather than effective, and even worse, lend the impression that improvements are possible when that isn't necessarily so. Most scientists are, by definition and just like members of any field, average. Exhortations to be better aren't likely to be rewarded.
Reading this over it sounds lecturey, sorry, this isn't really directed at you. These are just the thoughts I already had in my head about this.
--Percy

This message is a reply to:
 Message 2 by Modulous, posted 09-14-2007 4:11 PM Modulous has not replied

Replies to this message:
 Message 6 by Omnivorous, posted 11-17-2007 10:45 PM Percy has not replied

  
Jon
Inactive Member


Message 4 of 6 (421917)
09-15-2007 1:06 AM
Reply to: Message 1 by Percy
09-14-2007 4:06 PM


It's obvious who's to blame for this sad state of affairs: THE MEDIA!
I think it's feminists.

This message is a reply to:
 Message 1 by Percy, posted 09-14-2007 4:06 PM Percy has not replied

  
Pharaoh205 
Inactive Member


Message 5 of 6 (434866)
11-17-2007 8:34 PM


spam removed
spam
Edited by AdminNosy, : No reason given.

  
Omnivorous
Member
Posts: 3978
From: Adirondackia
Joined: 07-21-2005
Member Rating: 7.3


Message 6 of 6 (434902)
11-17-2007 10:45 PM
Reply to: Message 3 by Percy
09-14-2007 6:17 PM


There was never a chance our experimental drug would help you. Sorry.
Percy writes:
It is often lamented that we don't see enough papers reporting negative results, that the results of such efforts often wind up in file drawers instead of journals. I think the criticism is a bit too strong, as any negative results must be kind of compelling for a journal to accept a paper reporting it.
It seems to me that the predicament of sick people who volunteer for medical research is the truly lamentable aspect of the non-publication of negative findings in medical studies.
Those who care most are likely the desperately sick people who were, say, in a drug-receiving cohort in a study that unknowingly repeated previous studies: they never had a chance of any benefit, and if negative results were centrally reported, they could have participated in a study with at least some possibility of benefit.
In addition, patients are frequently refused participation in one study expressly because of their prior participation in another. So the patients at hand risked untoward events without any real chance of benefit in the unnecessary study, and forfeited any chance of participating in another.
The rest of us, of course, also suffer negative impacts: from the waste of funding, public and private, and from opportunities lost when unnecessary, redundant studies crowd out genuinely new work.
The reaction at most journals would be, "Here's a paper reporting yet another failed effort to fuzbat the wizsteins of norburs. Who cares. Rejected!"
Let's rephrase your spoof quote: "Here's a paper reporting yet another failed effort to treat pancreatic cancer with extremely toxic norburitine. Who cares. Rejected!"
Oncologists, of course, generally do report both positive and negative results to central databases, but many fields of medicine and most branches of science do not. That's pretty stupidly wasteful of time, lives, and money.
I can't see anything good about that, or any good reason to refrain from changing it.
The competition for journal space is well and good, but negative results are important and should be reported: a winner-take-all perspective is as pernicious in research as it is in economies.
Note that in the field of archeology, another field that depends on peer-reviewed journals, both funding and access to sites are being withdrawn when publication lags, whether that be in a journal, private press, database, website, etc. Not being a winner in the journal joust is no excuse for not publishing results.

Real things always push back.
-William James
Save lives! Click here!
Join the World Community Grid with Team EvC!
---------------------------------------

This message is a reply to:
 Message 3 by Percy, posted 09-14-2007 6:17 PM Percy has not replied

  
Newer Topic | Older Topic
Jump to:


Copyright 2001-2023 by EvC Forum, All Rights Reserved

™ Version 4.2
Innovative software from Qwixotic © 2024