Register | Sign In


Understanding through Discussion


EvC Forum active members: 60 (9208 total)
2 online now:
Newest Member: Skylink
Post Volume: Total: 919,430 Year: 6,687/9,624 Month: 27/238 Week: 27/22 Day: 0/9 Hour: 0/0


Thread  Details

Email This Thread
Newer Topic | Older Topic
  
Author Topic:   Failure to Replicate
Tempe 12ft Chicken
Member (Idle past 586 days)
Posts: 438
From: Tempe, Az.
Joined: 10-25-2012


Message 1 of 6 (767323)
08-28-2015 10:39 AM


A new study that is being released soon in Science discusses how reproducible 100 different studies in the field of Psychology are. The final results were not very good for the field and could possibly lead to several retractions.
According to the researchers:
Cathleen O'Grady writes:
Of the 100 original studies, 97 had results that were statistically significant; only a third of the replications, however, had statistically significant results. Around half of the replications had effect sizes that were roughly comparable to the original studies. The teams conducting the replications reported whether they considered the effect to be replicated, and only 39 percent of them said it did. These criteria suggest that fewer than half of the originals were successfully replicated.
Now, I have always heard that psychology and social science is less structured than some of the more rigorous fields, such as physics or geology, but the question is what, if any, takeaways should we find for the scientific community from these results. Should certain areas of science be required to increase repeat experiments to avoid this statistical error or should this new paradigm be applied across the board for all sciences. If it is for all sciences, what methods can be undertaken to ensure that journals print more repeat experiments and failures to replicate instead of only focusing on the "sexy" new ideas?
I was thinking either "Is it Science" or Miscellaneous, but will defer to moderator decision.
100 Psychology Experiments Repeated Less Than Half Successful

The theory of evolution by cumulative natural selection is the only theory we know of that is in principle capable of explaining the existence of organized complexity. - Richard Dawkins
Creationists make it sound as though a 'theory' is something you dreamt up after being drunk all night. - Issac Asimov
If you removed all the arteries, veins, & capillaries from a person’s body, and tied them end-to-endthe person will die. - Neil Degrasse Tyson
What would Buddha do? Nothing! What does the Buddhist terrorist do? Goes into the middle of the street, takes the gas, *pfft*, Self-Barbecue. The Christian and the Muslim on either side are yelling, "What the Fuck are you doing?" The Buddhist says, "Making you deal with your shit. - Robin Williams

Replies to this message:
 Message 3 by 1.61803, posted 08-28-2015 11:17 AM Tempe 12ft Chicken has replied
 Message 6 by caffeine, posted 08-29-2015 8:36 AM Tempe 12ft Chicken has not replied

  
AdminNosy
Administrator
Posts: 4755
From: Vancouver, BC, Canada
Joined: 11-11-2003


Message 2 of 6 (767325)
08-28-2015 10:55 AM


Thread Copied from Proposed New Topics Forum
Thread copied here from the Failure to Replicate thread in the Proposed New Topics forum.

  
1.61803
Member (Idle past 1754 days)
Posts: 2928
From: Lone Star State USA
Joined: 02-19-2004


Message 3 of 6 (767327)
08-28-2015 11:17 AM
Reply to: Message 1 by Tempe 12ft Chicken
08-28-2015 10:39 AM


So more than half psychologist are full of bovine excrement. Who knew?

"You were not there for the beginning. You will not be there for the end. Your knowledge of what is going on can only be superficial and relative" William S. Burroughs

This message is a reply to:
 Message 1 by Tempe 12ft Chicken, posted 08-28-2015 10:39 AM Tempe 12ft Chicken has replied

Replies to this message:
 Message 4 by Tempe 12ft Chicken, posted 08-28-2015 2:12 PM 1.61803 has not replied

  
Tempe 12ft Chicken
Member (Idle past 586 days)
Posts: 438
From: Tempe, Az.
Joined: 10-25-2012


Message 4 of 6 (767349)
08-28-2015 2:12 PM
Reply to: Message 3 by 1.61803
08-28-2015 11:17 AM


That does seem to be the case in regards to at least the 100 studies this group has attempted to replicate. Now, they do state that they achieved similar results in some, but that the original researchers had just made too many assumptions in their conclusions from those results. However, do you think that running a similar program of trying to replicate past studies is advisable in any of the other branches of science also?
One article I read about this mentioned the large amount of scientists that are being trained, combined with the minimum amount of public funding has led researchers (the article is still discussing psych, but I'm wondering what we can extrapolate to other sciences, if anything) to bolster claims to try and ensure funding for their study. Do you think that this affects any other area of science, and if so, is this effect equal to that in the social sciences? One area I can think of is in topics such as String Theory, where there is a lot of current discussion going on and many different ideas. Could this broad list of options open this area of physics up to similar risks from studies that cannot be replicated?

The theory of evolution by cumulative natural selection is the only theory we know of that is in principle capable of explaining the existence of organized complexity. - Richard Dawkins
Creationists make it sound as though a 'theory' is something you dreamt up after being drunk all night. - Issac Asimov
If you removed all the arteries, veins, & capillaries from a person’s body, and tied them end-to-endthe person will die. - Neil Degrasse Tyson
What would Buddha do? Nothing! What does the Buddhist terrorist do? Goes into the middle of the street, takes the gas, *pfft*, Self-Barbecue. The Christian and the Muslim on either side are yelling, "What the Fuck are you doing?" The Buddhist says, "Making you deal with your shit. - Robin Williams

This message is a reply to:
 Message 3 by 1.61803, posted 08-28-2015 11:17 AM 1.61803 has not replied

Replies to this message:
 Message 5 by NosyNed, posted 08-28-2015 5:19 PM Tempe 12ft Chicken has not replied

  
NosyNed
Member
Posts: 9011
From: Canada
Joined: 04-04-2003


Message 5 of 6 (767376)
08-28-2015 5:19 PM
Reply to: Message 4 by Tempe 12ft Chicken
08-28-2015 2:12 PM


String Theory
Do you think that this affects any other area of science, and if so, is this effect equal to that in the social sciences? One area I can think of is in topics such as String Theory, where there is a lot of current discussion going on and many different ideas. Could this broad list of options open this area of physics up to similar risks from studies that cannot be replicated?
As I see it a "study" as used here is some form of experiment. There are no experiments (or damned few) that apply to string theory as yet. It is all very interesting mathematics.
The math is checked very rigorously (no one wants to publish and find they didn't carry the 1) so I doubt that this is a serious issue in that field.

This message is a reply to:
 Message 4 by Tempe 12ft Chicken, posted 08-28-2015 2:12 PM Tempe 12ft Chicken has not replied

  
caffeine
Member (Idle past 1275 days)
Posts: 1800
From: Prague, Czech Republic
Joined: 10-22-2008


(1)
Message 6 of 6 (767424)
08-29-2015 8:36 AM
Reply to: Message 1 by Tempe 12ft Chicken
08-28-2015 10:39 AM


If it is for all sciences, what methods can be undertaken to ensure that journals print more repeat experiments and failures to replicate instead of only focusing on the "sexy" new ideas?
It's most certainly a problem for all sciences, but there is at least one method to help which is already being employed in some fields. There was a study publish recently in PLOS One that illustates this well. It compared studies into treatment for cardiovascular disease funded by the NHLBI. Prior to the year 2000, 57% showed a positive effect of the treatment studied. Since 2000, 8% did.
The difference? In 2000 compulsory pre-registration of trials was introduced. This means that negative results are now published, and the aims of the trial had to be clearly specified in advance, to prevent posthoc playing around with statistics to make it look like you discovered something else worthwhile.
Likelihood of Null Effects of Large NHLBI Clinical Trials Has Increased over Time | PLOS ONE

This message is a reply to:
 Message 1 by Tempe 12ft Chicken, posted 08-28-2015 10:39 AM Tempe 12ft Chicken has not replied

  
Newer Topic | Older Topic
Jump to:


Copyright 2001-2023 by EvC Forum, All Rights Reserved

™ Version 4.2
Innovative software from Qwixotic © 2024