[Vision2020] MIT Meteorologist Lindzen's Recent Climate Science Paper Poorly Peer Reviewed, With Direct Critical Comments Rejected by GRL?

[Ted Moffett]

Richard Lindzen, the famous anthropogenic warming skeptic, in the climate
science headlines again, involved in what may become another example of a
breakdown in the peer review process for science publishing.  The science
professionals involved in the following commentary are not easily dismissed
with obscurantist obfuscation:
http://www.realclimate.org/index.php/archives/2010/01/lc-grl-comments-on-peer-review-and-peer-reviewed-comments/#more-2710
L&C,
GRL, comments on peer review and peer-reviewed comments
Filed under:

   - Climate Science<http://www.realclimate.org/index.php/archives/category/climate-science/>

— gavin @ 10 January 2010

I said on Friday<http://www.realclimate.org/index.php/archives/2010/01/first-published-response-to-lindzen-and-choi/>that
I didn’t think that Lindzen and Choi (2009) was obviously nonsense.
Well, a number<http://www.realclimate.org/index.php/archives/2010/01/first-published-response-to-lindzen-and-choi/comment-page-1/#comment-153754>of
people<http://www.realclimate.org/index.php/archives/2010/01/first-published-response-to-lindzen-and-choi/comment-page-1/#comment-153884>have
disagreed with me, and in doing so, have presented some of the back
story<http://www.realclimate.org/index.php/archives/2010/01/first-published-response-to-lindzen-and-choi/comment-page-2/#comment-153998>on
the how the response was handled. I think this deserves to be more
widely
known in the hope that it will generate some discussion in the community for
how such situations might be dealt with in the future.

>From Chris O’Dell:

Given the large number of comments on the peer-review process in general and
in the LC09 case in particular, it is probably worthwhile to give a bit more
backstory to our Trenberth et al. paper. On my first reading of LC09, I was
quite amazed and thought if the results were true, it would be incredible
(and, in fact, a good thing!) and hence warranted independent checking. Very
simple attempts to reproduce the LC09 numbers simply didn’t work out and
revealed some flaws in their process. To find out more, I contacted Dr.
Takmeng Wong at NASA Langley, a member of the CERES and ERBE science teams
(and major player in the ERBE data set) and found out to my surprise that no
one on these teams was a reviewer of LC09. Dr. Wong was doing his own
verification of LC09 and so we decided to team up.

After some further checking, I came across a paper very similar to LC09 but
written 3 years earlier – Forster & Gregory (2006) , hereafter FG06. FG06,
however, came to essentially opposite conclusions from LC09, namely that the
data implied an overall positive feedback to the earth’s climate system,
though the results were somewhat uncertain for various reasons as described
in the paper (they attempted a proper error analysis). The big question of
course was, how is it that LC09 did not even bother to reference FG06, let
alone explain the major differences in their results? Maybe Lindzen & Choi
didn’t know about the existence of FG06, but certainly at least one reviewer
should have. And if they also didn’t, well then, a very poor choice of
reviewers was made.

This became clear when Dr. Wong presented a joint
analysis<http://science.larc.nasa.gov/ceres/STM/2009-11/index.html>he
& I made at the CERES science team meeting held in Fort Collins,
Colorado
in November. At this meeting, Drs. Trenberth and Fasullo approached us and
said they had done much the same thing as we had, and had already submitted
a paper to GRL, specifically a comment paper on LC09. This comment was
rejected out of hand by GRL, with essentially no reason given. With some
more inquiry, it was discovered that:

   1. The reviews of LC09 were “extremely favorable”
   2. GRL doesn’t like comments and is thinking of doing away with them
   altogether.
   3. GRL wouldn’t accept comments on LC09 (and certainly not multiple
   comments), and instead it was recommended that the four of us submit a
   stand-alone paper rather than a comment on LC09.

We all felt strongly that we simply wanted to publish a comment directly on
LC09, but gave in to GRL and submitted a stand-alone paper. This is why, for
instance, LC09 is not directly referenced in our paper abstract. The
implication of statement (1) above is that LC09 basically skated through the
peer-review process unchanged, and the selected reviewers had no problems
with the paper. This, and for GRL to summarily reject all comments on LC09
appears extremely sketchy.

In my opinion, there is a case to be made on the peer-review process being
flawed, at least for certain papers. Many commenters say the system isn’t
perfect, but it in general works. I would counter that it certainly could be
better. For AGU journals, authors are invited to give a list of proposed
reviewers for their paper. When the editor is lazy or tight on time or
whatever, they may just use the suggested reviewers, whether or not those
reviewers are appropriate for the paper in question. Also, when a comment on
a paper is submitted, the comment goes to the editor that accepted the
original paper – a clear conflict of interest.

So yes, the system may work most of the time, but LC09 is a clear example
that it doesn’t work all of the time. I’m not saying LC09 should have been
rejected or wasn’t ultimately worthy of publication, but reviewers should
have required major modifications before it was accepted for publication.

To me this raises a number of questions. Why are the editors at GRL
apparently not following the published editorial
policy<http://www.agu.org/pubs/authors/policies/comments_replies.shtml>on
comments? The current policy might not be ideal, and perhaps should be
changed, but surely not by fiat, and surely not without announcing that
policy change? This particular example has ended up divorcing the response
from the original paper and clearly makes it harder to follow the
development of this analysis in the literature. Additionally, in cases where
there appears to have been lapses in peer-review (for whatever reason), is
there not an argument for having a different editor deal with the
comment/response? Perhaps a new online journal which independently publishes
peer-reviewed comments and responses is called for?

Everyone involved in the peer-review process knows full well the difficulty
in finding suitable reviewers who have the time and inclination to do a good
review. The pressures on editors both to be seen to be fair, and to actually
be fair to the authors (and the readers!) are strong, and occasionally
things will go wrong. The measure of such a system is not whether it is
perfect, but whether it deals appropriately and quickly with problems when
they (inevitably) arise.

NB. Comments on how to improve the situation are welcome, but please avoid
simply criticising papers that you personally think shouldn’t have been
published in the form they were.
 Comments (pop-up) (18) <http://www.realclimate.org/?comments_popup=2710>
------------------------------------------
Vision2020 Post: Ted Moffett
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.fsr.com/pipermail/vision2020/attachments/20100110/ef29f8c7/attachment-0001.html