<h3 id="post-2710" class="storytitle">Richard Lindzen, the famous anthropogenic warming skeptic, in the climate science headlines again, involved in what may become another example of a breakdown in the peer review process for science publishing. The science professionals involved in the following commentary are not easily dismissed with obscurantist obfuscation:</h3>
<h3 class="storytitle"><a href="http://www.realclimate.org/index.php/archives/2010/01/lc-grl-comments-on-peer-review-and-peer-reviewed-comments/#more-2710">http://www.realclimate.org/index.php/archives/2010/01/lc-grl-comments-on-peer-review-and-peer-reviewed-comments/#more-2710</a></h3>
<h3 class="storytitle">L&C, GRL, comments on peer review and peer-reviewed comments</h3>
<div class="meta">Filed under:
<ul class="post-categories">
<li><a title="View all posts in Climate Science" href="http://www.realclimate.org/index.php/archives/category/climate-science/" rel="category tag">Climate Science</a></li></ul>— gavin @ 10 January 2010 </div>
<div class="entry">
<p>I said <a href="http://www.realclimate.org/index.php/archives/2010/01/first-published-response-to-lindzen-and-choi/">on Friday</a> that I didn’t think that Lindzen and Choi (2009) was obviously nonsense. Well, a <a href="http://www.realclimate.org/index.php/archives/2010/01/first-published-response-to-lindzen-and-choi/comment-page-1/#comment-153754">number</a> of <a href="http://www.realclimate.org/index.php/archives/2010/01/first-published-response-to-lindzen-and-choi/comment-page-1/#comment-153884">people</a> have disagreed with me, and in doing so, have presented some of the <a href="http://www.realclimate.org/index.php/archives/2010/01/first-published-response-to-lindzen-and-choi/comment-page-2/#comment-153998">back story</a> on the how the response was handled. I think this deserves to be more widely known in the hope that it will generate some discussion in the community for how such situations might be dealt with in the future.<br>
<span id="more-2710"></span></p>
<p>From Chris O’Dell:</p>
<blockquote>
<p>Given the large number of comments on the peer-review process in general and in the LC09 case in particular, it is probably worthwhile to give a bit more backstory to our Trenberth et al. paper. On my first reading of LC09, I was quite amazed and thought if the results were true, it would be incredible (and, in fact, a good thing!) and hence warranted independent checking. Very simple attempts to reproduce the LC09 numbers simply didn’t work out and revealed some flaws in their process. To find out more, I contacted Dr. Takmeng Wong at NASA Langley, a member of the CERES and ERBE science teams (and major player in the ERBE data set) and found out to my surprise that no one on these teams was a reviewer of LC09. Dr. Wong was doing his own verification of LC09 and so we decided to team up.</p>
<p>After some further checking, I came across a paper very similar to LC09 but written 3 years earlier – Forster & Gregory (2006) , hereafter FG06. FG06, however, came to essentially opposite conclusions from LC09, namely that the data implied an overall positive feedback to the earth’s climate system, though the results were somewhat uncertain for various reasons as described in the paper (they attempted a proper error analysis). The big question of course was, how is it that LC09 did not even bother to reference FG06, let alone explain the major differences in their results? Maybe Lindzen & Choi didn’t know about the existence of FG06, but certainly at least one reviewer should have. And if they also didn’t, well then, a very poor choice of reviewers was made.</p>
<p>This became clear when Dr. Wong presented a <a href="http://science.larc.nasa.gov/ceres/STM/2009-11/index.html">joint analysis</a> he & I made at the CERES science team meeting held in Fort Collins, Colorado in November. At this meeting, Drs. Trenberth and Fasullo approached us and said they had done much the same thing as we had, and had already submitted a paper to GRL, specifically a comment paper on LC09. This comment was rejected out of hand by GRL, with essentially no reason given. With some more inquiry, it was discovered that:</p>
<ol>
<li>The reviews of LC09 were “extremely favorable”</li>
<li>GRL doesn’t like comments and is thinking of doing away with them altogether.</li>
<li>GRL wouldn’t accept comments on LC09 (and certainly not multiple comments), and instead it was recommended that the four of us submit a stand-alone paper rather than a comment on LC09.</li></ol>
<p>We all felt strongly that we simply wanted to publish a comment directly on LC09, but gave in to GRL and submitted a stand-alone paper. This is why, for instance, LC09 is not directly referenced in our paper abstract. The implication of statement (1) above is that LC09 basically skated through the peer-review process unchanged, and the selected reviewers had no problems with the paper. This, and for GRL to summarily reject all comments on LC09 appears extremely sketchy.</p>
<p>In my opinion, there is a case to be made on the peer-review process being flawed, at least for certain papers. Many commenters say the system isn’t perfect, but it in general works. I would counter that it certainly could be better. For AGU journals, authors are invited to give a list of proposed reviewers for their paper. When the editor is lazy or tight on time or whatever, they may just use the suggested reviewers, whether or not those reviewers are appropriate for the paper in question. Also, when a comment on a paper is submitted, the comment goes to the editor that accepted the original paper – a clear conflict of interest.</p>
<p>So yes, the system may work most of the time, but LC09 is a clear example that it doesn’t work all of the time. I’m not saying LC09 should have been rejected or wasn’t ultimately worthy of publication, but reviewers should have required major modifications before it was accepted for publication. </p>
</blockquote>
<p>To me this raises a number of questions. Why are the editors at GRL apparently not following the <a href="http://www.agu.org/pubs/authors/policies/comments_replies.shtml">published editorial policy</a> on comments? The current policy might not be ideal, and perhaps should be changed, but surely not by fiat, and surely not without announcing that policy change? This particular example has ended up divorcing the response from the original paper and clearly makes it harder to follow the development of this analysis in the literature. Additionally, in cases where there appears to have been lapses in peer-review (for whatever reason), is there not an argument for having a different editor deal with the comment/response? Perhaps a new online journal which independently publishes peer-reviewed comments and responses is called for? </p>
<p>Everyone involved in the peer-review process knows full well the difficulty in finding suitable reviewers who have the time and inclination to do a good review. The pressures on editors both to be seen to be fair, and to actually be fair to the authors (and the readers!) are strong, and occasionally things will go wrong. The measure of such a system is not whether it is perfect, but whether it deals appropriately and quickly with problems when they (inevitably) arise. </p>
<p><small><font size="2">NB. Comments on how to improve the situation are welcome, but please avoid simply criticising papers that you personally think shouldn’t have been published in the form they were.</font></small></p>
</div>
<div class="feedback"> <a title="Comment on L&C, GRL, comments on peer review and peer-reviewed comments" onclick="wpopen(this.href); return false" href="http://www.realclimate.org/?comments_popup=2710">Comments (pop-up) (18)</a></div>
<div class="feedback">------------------------------------------</div>
<div class="feedback">Vision2020 Post: Ted Moffett</div>