The Peer Review Fetish

The following statement in a ClimateProgress post this past weekend jumped out at me:

“Peer review is the basis of modern scientific endeavour.”

This strikes me as an accurate reflection of the way that many – perhaps most – in the climate world think. It’s also wrong.

The sentiment is not restricted to one side of the policy fight. Yes, climate campaigners regularly point out that the work that backs them up is peer reviewed. Yet many of last year’s attacks on the IPCC (and often, by extension, the enterprise of addressing climate change) focused on its use of non-peer-reviewed sources too.

Why is the statement wrong? Modern science rests on a mix of transparency, replicability, and peer evaluation and challenge. Huge numbers of (sometimes peer-reviewed) papers get published. Some of them stand the test of time. Others don’t. What separates the wheat from the chaff is ultimately whether the work withstands broad scrutiny.

Peer review is only a small part of this dynamic. Journals (and book editors) make a point of sending papers out to two or three people for (usually blind) review. Positive reviews indicate that the reviewers think that the paper is of sufficient quality for the journal that’s considering it. They do not necessarily indicate agreement with the paper’s conclusions. Indeed they do not even necessarily mean that the reviewers think that the paper is all that good: that depends on the journal it’s being considered for.

A couple reviewers, of course, are a poor substitute for mass scrutiny. Sometimes reviewers are chosen poorly; other times they’re lazy. For a complex, interdisciplinary paper, it’s often impossible to find reviewers who actually understand the whole thing.

Once a paper is published, the real evaluation begins. I would place a lot more trust in a paper that withstood assessment by hundreds of other researchers after publication, even if it was not initially peer reviewed, than one that was published in a peer reviewed journal but was subsequently trashed by large numbers of competent researchers. Better yet, I’d look for both peer review and widespread acceptance.

This, however, does not appear to be how most people in the climate world think. Journalists are too often allowed to turn off their brains once they’re handed a piece of peer reviewed work (or told that another piece of work has not undergone peer review). Peer review converts the paper into gospel; all that remains is to preach its existence to the masses. Conversely, to attack a good study that is not peer reviewed, all that seems to be required is an attack on that procedural shortcoming; substance, too often, becomes secondary.

I can understand why peer review has become the gold standard in some quarters. The IPCC, in particular, needs a simple screen for quality, given the immense sprawl of its activities. And, to be fair, many deeply flawed studies only see the light of day because they avoid the barrier of peer review. That, however, merely indicates that peer review is often a useful minimum standard. It shouldn’t be read as anything more.

This is not an attack on the importance of science, or rigorous argument, in informing climate policy. Quite the opposite. Conflating peer review with scientific soundness impoverishes our appreciation of the scientific process. Peer review should be one criterion that people use in assessing the strength of any given piece of research – nothing more, nothing less.