Register | Sign In


Understanding through Discussion


EvC Forum active members: 65 (9162 total)
6 online now:
Newest Member: popoi
Post Volume: Total: 915,817 Year: 3,074/9,624 Month: 919/1,588 Week: 102/223 Day: 0/13 Hour: 0/0


Thread  Details

Email This Thread
Newer Topic | Older Topic
  
Author Topic:   The State of Peer Review
Rrhain
Member
Posts: 6351
From: San Diego, CA, USA
Joined: 05-03-2003


Message 1 of 11 (72239)
12-11-2003 2:17 AM


Apparently, back in August, the Office of Management and Budget for the Bush Administration issued a bulletin regarding the new policy for peer review regarding government related research. Specifically, the Office of Information and Regulatory Affairs and the White House Office of Science and Technology Policy would be in charge and that reviewers would need to be "independent of the agency" for which the review was being done.
In and of itself, this doesn't seem so bad, until one recognizes that "independent of the agency" means receives no government funds. Well, just what science groups don't receive government funds? That's right, industry.
By this policy, the government's review of science regarding, say, the health effects of smoking would not be allowed to use studies by such groups as the American Cancer Society, the American Heart Association, etc., and would instead rely upon studies put forward by the tobacco industries.
Please, please tell me that most people understand how this is a despicable thing to do. While the idea that one wants to keep the the reviewers independent is spot on, the problem is that government is so involved in funding research that to claim that anybody who receives government funding is tainted means we are left with special-interest groups reviewing the material.
Just as the shift in the federal advisory committees were made to favor those from industry, the shift to essentially turning over all review to industry is a disaster waiting to happen.
My prediction: Since the Bush administration already claims that global warning "needs more study," by making most of the available reviewers companies who would be negatively impacted by regulations on emissions, it will be found that global warming has no evidence at all for it and restrictions on emissions will be rolled back.
Science for special interests
------------------
Rrhain
WWJD? JWRTFM!

Replies to this message:
 Message 2 by Mammuthus, posted 12-11-2003 5:25 AM Rrhain has not replied

  
Mammuthus
Member (Idle past 6475 days)
Posts: 3085
From: Munich, Germany
Joined: 08-09-2002


Message 2 of 11 (72250)
12-11-2003 5:25 AM
Reply to: Message 1 by Rrhain
12-11-2003 2:17 AM


While this is additional terrible news regarding peer review, the current scientific peer review system is a mess in any case. The concept behind peer review is admirable. However, in practice, it is often used by researchers to knock down or scoop their competition, particularly in higher profile journals. Ask any scientist who has gone through the peer review process and I bet you they would tell you a story where it was clear the reviewer had not read the paper and merely rejected the work out of spite. In my own tenure in science I have seen 1) a well known scientist use his sexual relationship with a journal editor to get a rejected paper published over the recommendations of the reviewers 2) same scientist contact "friends" in a top journal to make sure a competing group with less clout never publishes in that journal 3) seen personal attacks from anonymous reviewers that have nothing to do with the actual topic of the paper 4) seen an entire project stolen down to the exact PCR primers used in the experiment when the reviewer rejected the paper and then copied the work and published it before the rejectee could get their work published elsewhere. And this is just publishing...it gets worse when you live off of grants and have to go through the same crap to get funding. The entire system does need to be reviewed and modified so that its original purposes of correcting mistakes, finding fraud, maintaining an extremely high standard of scientific quality are restored. However, instituting a system like the admistration wants which would bias everything even further is a completely retarded idea. If anything, powerful interests, including famous scientists, should have less influence over peer review and the system should be made more egalatarian so that a Nobel prize winner faces the same chance of rejection in publishing or in obtaining funds as a Ph.D. student.

This message is a reply to:
 Message 1 by Rrhain, posted 12-11-2003 2:17 AM Rrhain has not replied

Replies to this message:
 Message 3 by helena, posted 12-11-2003 7:13 AM Mammuthus has replied

  
helena 
Suspended Member (Idle past 5845 days)
Posts: 80
Joined: 03-27-2008


Message 3 of 11 (72255)
12-11-2003 7:13 AM
Reply to: Message 2 by Mammuthus
12-11-2003 5:25 AM


I do think that you are a bit too harsh on the peer review system. While I must agree that politics come into play a lot, especially in the higher profile journals, it seems to me that good or deserving work will eventually be published (at least that's my experience). Of course, the paper might not be published in the journal it deserves to be published in, but that's all vanity anyways.
What I must agree to is the fact that referees show the tendency not to read / try to comprehend the paper at all. This is worsened by the fact that most journals nowadays send the paper out to multiple referees, increasing the workload on individual referees enormously.
The system proposed above is not only a step in the wrong direction but rather over the cliff.
What I would like to see as a reform would be to publish the referee's comments (anonymously) alongside the paper. What that could accomplish is (a) to point readers to some possible flaws / problems and (b) to lessen the amount of personal attacks in referee reports (because reading such comments would just cause sympathy for the authors).
As far as funding is concerned, I think that not only the referee but also the applicant should stay anonymous (i.e. only the passages relevant to the project should be handed to the referee). There is still some probability that the referee could guess the identity of the applicant but putting less emphasis on one's own works in the references would certainly help staying anonymous.

This message is a reply to:
 Message 2 by Mammuthus, posted 12-11-2003 5:25 AM Mammuthus has replied

Replies to this message:
 Message 4 by Mammuthus, posted 12-11-2003 7:59 AM helena has replied
 Message 5 by Loudmouth, posted 12-11-2003 3:10 PM helena has not replied

  
Mammuthus
Member (Idle past 6475 days)
Posts: 3085
From: Munich, Germany
Joined: 08-09-2002


Message 4 of 11 (72258)
12-11-2003 7:59 AM
Reply to: Message 3 by helena
12-11-2003 7:13 AM


Hi Alex,
I do not advocate scrapping peer review. But I do think the current process needs to be reviewed itself and corrected or modified where needed. If anything it is critical because more and more decisions are made based on the impact factors of journals that scientists publish in as opposed to the actual quality or significance of the work. It would be naive to think that science is immune to fraud, politics, or manipulation of such a system when people's careers or advancement are on the line. It developes a culture of short term "newsworthiness" thinking as opposed to actual scientific persuit. Rrhain's post shows that now an additional factor, those with a vested financial interest, are seeking to bias the peer review system in their favor which does not make the situation any better and would be incredibly harmful to science...a couple of assholes who get their kicks dinging papers is one thing..but a coordinated effort by industry to influence what and how things are published would be a disaster.

This message is a reply to:
 Message 3 by helena, posted 12-11-2003 7:13 AM helena has replied

Replies to this message:
 Message 6 by helena, posted 12-11-2003 5:01 PM Mammuthus has replied

  
Loudmouth
Inactive Member


Message 5 of 11 (72319)
12-11-2003 3:10 PM
Reply to: Message 3 by helena
12-11-2003 7:13 AM


quote:
What I would like to see as a reform would be to publish the referee's comments (anonymously) alongside the paper. What that could accomplish is (a) to point readers to some possible flaws / problems and (b) to lessen the amount of personal attacks in referee reports (because reading such comments would just cause sympathy for the authors).
Actually, this is how it happens in the journals our lab submits to (mainly microbial and human immunity). I'm not sure if our names are on the submission or not, but we do get feedback from each reviewer and also a judgement by the journal on acceptance, acceptance with revisions (as suggested by reviewers), or denial. Also, we can also state which people we would like to exclude from the review board, although they are taken just as suggestions. Usually, from my experience, reviewers in good standing with the journal and who are also listed among your citations are the chosen the most.
I would agree with Mammuthus that every investigator feels a little antsy when sending out manuscripts, fraud has occurred in the past, especially in competitive fields. Even moving away from peer review (investigators in the same field) to a review board may not entirely cure the problem, but it may help a bit. The practice of science is apolitical, but Science as a whole has quickly become political. Unfortunately, it comes down to money, not enough to go around. The more you publish, the more likely you are to get funded, plain and simple. I don't know how often this happens, but I would be willing to bet that good grants have been denied because of a lack of publications on the part of the investigator. Until anyone who wants to research can get the funding they want we will continue to see politics in science. In other words, this may not have a cure.

This message is a reply to:
 Message 3 by helena, posted 12-11-2003 7:13 AM helena has not replied

  
helena 
Suspended Member (Idle past 5845 days)
Posts: 80
Joined: 03-27-2008


Message 6 of 11 (72345)
12-11-2003 5:01 PM
Reply to: Message 4 by Mammuthus
12-11-2003 7:59 AM


quote:
It would be naive to think that science is immune to fraud, politics, or manipulation of such a system when people's careers or advancement are on the line. It developes a culture of short term "newsworthiness" thinking as opposed to actual scientific persuit.
I agree a hundred percent. The problem is mainly that there are not many factors that can be used to evaluate a research institution, except papers published, patents filed, and (possibly) student satisfaction. Therefore, one cannot "pursue the advancement of science" without risking to lose one's job.
As for fraud: I'm working in a field closely related to J.H. Schoen's work, so go figure
Concerning peer review again: I think that there are no real alternatives. It is even extremely hard to reform the system: in fact, the best qualified referees are probably those working on related issue and thus being in direct competition.
As stated above, more referees don't help either as this increases the number of "bad" or "too quick" reviews.
Professional review boards don't make a lot of sense either because that would engender even more politics and pressure exerted (as you would probably know the members of a journal's review board).
There have been suggestions that an article should be posted on the net for a certain period of time and be subject to open review. I just don't think that would work out with confidentiality being completely broken.
As for influence of certain interest groups: I think that this is already happening. There are "scientific" advisors to most governments that more or less determine the general funding landscape. Lobbyist groups are always working on getting funding for research in their field. One cannot deny that EC calls for proposals are strongly influenced by industry interests (especially in Material Science / Physics and I would guess in Microbiology too). For them to participate in the actual review process is a smaller step than you might think..

This message is a reply to:
 Message 4 by Mammuthus, posted 12-11-2003 7:59 AM Mammuthus has replied

Replies to this message:
 Message 7 by Mammuthus, posted 12-12-2003 3:34 AM helena has replied

  
Mammuthus
Member (Idle past 6475 days)
Posts: 3085
From: Munich, Germany
Joined: 08-09-2002


Message 7 of 11 (72466)
12-12-2003 3:34 AM
Reply to: Message 6 by helena
12-11-2003 5:01 PM


quote:
Concerning peer review again: I think that there are no real alternatives. It is even extremely hard to reform the system:
Hi Alex,
I agree that there is no real alternative to peer review (though I do believe reproducibility of results in other labs is more critical i.e. that is where Schoen got nailed). But I do think there could be some possibilities that could at least be tried to improve the system. I think that the review process should either be completely open i.e. both the reviewer and reviewee know each others identity, or completely closed i.e. niether knows who is who. The idea of posting the reviews with the article is fine but impractical. Who wants to slog through tons of reviews every time one reads a paper? I think it would get ignored.
Regarding other measures, the journal impact factor chasing is a relatively recent phenomenon. I think it is rather harmful. It was always a major accomplishment to get a paper published in the top journal in ones field or in general journals like Nature or Science...but now we are at an extreme (my department is like this) where people are chasing after and tracking differences in impact factors between journals with impacts of 2.3 and 2.4 because of the 0.1 difference...and despite the fact that they vary from year to year tremendously. I don't really know what can be done about this aspect of modern science. This is driven as Loudmouth said, by too little money and too many people. The risk as I see it, if trends continue , is that academic research will completely abandon basic research in the persuit of immediately applicable and hot topics and will converge with the goals of industry, to make stuff you can sell. While this can produce results, many fields will suffer tremendously from such a narrow focus and set of goals.
I don't have the answers to these problems but I think they are pretty serious and are being ignored by universities and the scientific establishment. Even if I did have the answers, nobody would listen to me since I am not famous

This message is a reply to:
 Message 6 by helena, posted 12-11-2003 5:01 PM helena has replied

Replies to this message:
 Message 8 by helena, posted 12-12-2003 4:11 AM Mammuthus has not replied

  
helena 
Suspended Member (Idle past 5845 days)
Posts: 80
Joined: 03-27-2008


Message 8 of 11 (72469)
12-12-2003 4:11 AM
Reply to: Message 7 by Mammuthus
12-12-2003 3:34 AM


Hi Mammuthus,
I don't think double open / double blind would work:
As for double blind, I think that any person established in his field to be considered as a referee could tell from the work, the references, even the general writing style from which group the paper was. So there is no way that the system is ever going to be really double blind.
Concerning completely open review, I think that would be extremely problematic. Knowing exactly who the referee is would put even more power in the hands of the "leading" / most prominent scientists in the field, as nobody in his right mind would openly challenge their work (there's grants, cluster projects you either get in or don't, conferences where having a talk is essential to interest people in your work...)
Deciding between impact factors of 2.3 and 2.4 seems a bit extreme to me. While we also have the tendency to go for the high impact journals, there is still the underlying wish that one's work be read. Thus, the journal is most often selected by its audience with impact factor being the second most important factor. Come to think of it: No, impact is probably the most important factor but the decision between journals of approximately equal impact factor is made based on the audience of the journal.
I second Loudmouth's opinion that it is mainly the competition for funding that leads to these impact factor races. My question, however, still remains: Is there any sensible system to evaluate a research institution that is not based on impact factors?
best regards

This message is a reply to:
 Message 7 by Mammuthus, posted 12-12-2003 3:34 AM Mammuthus has not replied

  
Minnemooseus
Member
Posts: 3941
From: Duluth, Minnesota, U.S. (West end of Lake Superior)
Joined: 11-11-2001
Member Rating: 10.0


Message 9 of 11 (142083)
09-13-2004 3:28 PM


At the-scientist.com (and elsewhere)
Nice little opinion piece on peer review:
Stop Whispering About Peer Review -
It validates work for researchers and can do the same for a wider audience | By Tracey Brown
http://www.the-scientist.com/yr2004/sep/opinion_040913.html
quote:
Tracey Brown is director of Sense About Science whose report on peer review is available online at Sense about Science – Because evidence matters
The full-blown paper is at JPG to PDF Made Easy: 3 Benefits In Using PDFBear For JPG to PDF Conversion - Sense About Science Blog
Moose
Added by edit: Adminnemooseus has just closed the earlier but still redundant topic Peer review under scrutiny. You may wish to also look back at that topic.
This message has been edited by minnemooseus, 09-13-2004 02:38 PM

Replies to this message:
 Message 10 by Rei, posted 09-13-2004 3:40 PM Minnemooseus has not replied

  
Rei
Member (Idle past 7013 days)
Posts: 1546
From: Iowa City, IA
Joined: 09-03-2003


Message 10 of 11 (142087)
09-13-2004 3:40 PM
Reply to: Message 9 by Minnemooseus
09-13-2004 3:28 PM


Re: At the-scientist.com (and elsewhere)
There was an interesting article on Slashdot recently (I could look it up if you'd like) about how several journals are now going to require any group, if the want their studies to be included in a journal, to inform the journal of the studies being conducted *before* they're being done. This is designed to stop groups from conducting several studies and omitting results that they don't like.

"Illuminant light,
illuminate me."

This message is a reply to:
 Message 9 by Minnemooseus, posted 09-13-2004 3:28 PM Minnemooseus has not replied

  
Minnemooseus
Member
Posts: 3941
From: Duluth, Minnesota, U.S. (West end of Lake Superior)
Joined: 11-11-2001
Member Rating: 10.0


Message 11 of 11 (271417)
12-21-2005 12:26 PM


Bump for Custard
Off-topic elsewhere was:
Although after reading the article in the NYT yesterday about how bad peer review has become, it's kind of unnerving to see the judge use scientific peer review as a reason why science and evolution ought to have more credence than ID.
I think this is the topic to follow up on the above quoted.
Moose

Professor, geology, Whatsamatta U
Evolution - Changes in the environment, caused by the interactions of the components of the environment.
"Do not meddle in the affairs of cats, for they are subtle and will piss on your computer." - Bruce Graham

  
Newer Topic | Older Topic
Jump to:


Copyright 2001-2023 by EvC Forum, All Rights Reserved

™ Version 4.2
Innovative software from Qwixotic © 2024