Transparency of manuscript reviewing is important and therefore some journals have started to use open review.
In
all the medical BMC journals, reviewers are asked to sign their
reviews and the pre-publication history of each paper is posted on
the web with the published article.
http://www.biomedcentral.com/info/about/peerreview/
The comments of reviewers, and the replies by the authors may be informative for the readers of the published paper. All relevant considerations cannot be included in the final report, for example, because of space limitations. Therefore the reviewer comments and replies by authors are of additional interest to those readers who are seriously interested in the topic.
However, in the case of rejection the BMC policy is semi-transparent. Although the reviewer comments are made available as net versions (below), the refutations of the reviewer comments are not.
Since the comments of reviewers and replies by authors are important when considering the validity of a scientific paper, I am briefly describing the reviewer comments and their counter-comments for the paper below.
************************************
In
Spring 2010, I submitted my manuscript
“Subgroup
analysis of large trials can guide further research: a case study of
vitamin E and pneumonia”
to
the journal “Trials” http://www.trialsjournal.com/
I received two reviewer comments:
Reviewer
1
http://www.trialsjournal.com/imedia/6765528423760221_comment.pdf
Reviewer
2
http://www.trialsjournal.com/imedia/1246630932376444_comment.pdf
The manuscript was rejected on the basis of these two reviewer comments.
However,
the associate editor wrote that:
“If at some stage you
are able to fully address all of the referees' concerns, you may wish
to consider submitting a new manuscript to Trials. If you are able to
do this, a full covering letter, explaining the revisions made,
should accompany the submission.”
Editor's
rejection
letter:
http://www.mv.helsinki.fi/home/hemila/PPH/CE/Trials_Reject_1.pdf
Since the reviewer comments did not point out any fatal flaws in my manuscript, we followed editor's suggestion and wrote replies in which we fully addressed all of the referees' concerns:
Replies
to Reviewer 1
comments:
http://www.mv.helsinki.fi/home/hemila/PPH/CE/Trials_Reject_1.pdf
Replies
to Reviewer 2
comments:
http://www.mv.helsinki.fi/home/hemila/PPH/CE/Trials_Reviewer_2.pdf
These comments and replies describe which kinds of issues have been explicitly considered in addition to the text published in the final paper.
Since
reviewer 1 and 2 comments were long, we wrote in the second
submission letter:
“It
is possible that we have missed or misunderstood some important
issues, and in such a case we would hope for a possibility to reply
to comments that still need our
reply.”
http://www.ltdk.helsinki.fi/users/hemila/PPH/CE/Trials_Submission_2.pdf
The manuscript was rejected a second time:
Second
editor's rejection
letter:
http://www.mv.helsinki.fi/home/hemila/PPH/CE/Trials_Reject_2.pdf
This second rejection was based on a very short reviewer comment:
Reviewer
3
http://www.trialsjournal.com/imedia/6870941364447350_comment.pdf
The
only points of reviewer 3 were:
1) the replies to reviewers 1 and
2 were too long
2) the approach of manuscript was “somewhat
unorthodox”
Neither of these statements are scientific arguments pointing out errors or unjustified conclusions in the manuscript. Since I was puzzled by such kind of editorial policy, I wrote a commentary to several editors of the Trials journal. In the commentary I expressed my surprise that my manuscript was rejected on the basis that my replies were “too long” and approach “somewhat unorthodox”, which are subjective opinions and not scientific arguments.
Commentary
on the editorial policy of
“Trials”:
http://www.mv.helsinki.fi/home/hemila/PPH/CE/Trials_EdPolicy.pdf
I did not receive any replies to the above email.
The
manuscript was finally published in another journal:
Hemilä
H, Kaprio J.
Subgroup analysis of large trials can guide further
research: a case study of vitamin E and pneumonia.
Clinical
Epidemiology 2011;3:51-59
http://dx.doi.org/10.2147/CLEP.S16114
*******************************************************************************
Thus, it is surprising that a manuscript reporting strong evidence of heterogeneity in the effect of vitamin E supplementation is rejected on the basis of Reviewer 3 comments.
There seem to be two potential reasons for such an unscientific rejection.
1)
In my manuscript I challenge the views of a senior editor of the
Trials journal about subgroup analysis (ref. 2). Thus, it seems
possible that the real reason for rejection was to prevent the
publication of findings that challenge the wisdom
of strongly opposing subgroup analysis. It is possible that the
editors rejecting my manuscript simply wanted to stifle debate,
although stifling debate has been criticized e.g. by a senior editor
of Trials journal:
“The
key characteristic of science is not its infallibility, a quality it
clearly does not and cannot have, but its self-correcting ability.
The decision by medical editors to stifle debate is
misguided...”
http://dx.doi.org/10.1371/journal.pmed.0020126
2) Two of the senior editors of the Trials journal keep up citing a 1975 study which claimed that vitamin C does not have any effect on the common cold and claimed that the observed benefit was explained by the placebo effect. However, the 1975 study was shown to be erroneously analyzed over a decade ago. Citation bias is a common phenomenon which usually originates from preconceptions of the authors, when they pick “illustrative examples” to support their argument. Thus, citing a vitamin C paper that was shown erroneously analyzed over a decade earlier may be a reflection of general bias of the senior editors against vitamins in general, including vitamin E, which was the topic of the rejected manuscript.
The
1975 study:
http://jama.ama-assn.org/cgi/content/abstract/231/10/1038
Refutation
of the 1975 analysis:
Hemilä
H. Vitamin C, the placebo effect, and the common cold: a case study
of how preconceptions influence the analysis of results.
J Clin
Epidemiol
1996;49:1079-84
http://dx.doi.org/10.1016/0895-4356(96)00189-8
http://helda.helsinki.fi//handle/10250/8082
Links to references are added
Hemilä
H. Response to the dissent by Thomas Chalmers
J Clin Epidemiol
1996;49:1087
http://dx.doi.org/10.1016/0895-4356(96)00191-6
http://helda.helsinki.fi//handle/10250/8079
Hemilä
H. Analysis of clinical data with breached blindness
Stat
Med 2006;25:1434-7
http://dx.doi.org/10.1002/sim.2347
The
most influential trial on vitamin C and the common cold: Karlowski et
al. (1975)
http://www.mv.helsinki.fi/home/hemila/karlowski
Citation
bias by senior editor 1, see description in:
Assessment
of the importance of double-blinding should be based on systematic
reviews
http://dx.doi.org/10.1111/j.1538-7836.2008.03006.x
https://helda.helsinki.fi/handle/10138/25297
Links to references are added
Citation
bias by senior editor 2, see description in:
Citation
bias in the CONSORT comments on
blinding.
http://www.bmj.com/content/340/bmj.c869/reply#bmj_el_235882
Harri
Hemilä
harri.hemila@helsinki.fi