Seldom in the field of human conflict has so much been written by so many people on a subject about which they know nothing. Or so I’d like to hope: in the sense that I’d hope that the denialist chatter about peer review was the nadir. But I do know something about peer review, though my knowledge is 7 years out of date. Nonetheless, I don’t hesitate to comment. If you’re wondering (or I’m wondering, coming back to this later) all this kicks off from the ship of fools nonsense, which has elevated peer review to super-star status for its 15 minutes in the blog-o-light.
For a working scientist, peer review is just part of the job. You write up your work, you show it to your colleagues (if you work somewhere like BAS, its likely mandatory that it gets passed around a bit just to make sure you’re not saying anything really dumb; your division head corrects a couple of typos. If you’re new, this is a really helpful part of the process; if you’re not new then likely the internal review becomes a formality), you send it to the best journal you think you can get away with, and eventually you get the reviews back. These will be a mixture of “please cite my paper” (usually disguised as “you need to consider X”), typos, and the occasional well-considered thoughtful comment that genuinely improves things. You sigh, you happily incorporate the thoughtful stuff, you work out how much of the not-very-helpful stuff you can get away with blowing off, and you resubmit (naturally, I don’t know what happens when someone senior in the field submits, since I never was). And sometimes you get a reviewer who really really doesn’t like your paper for what you regard as invalid reasons, and you have to decide whether to fight to the death or go elsewhere.
For a “skeptic” – many of whom are on display at JoNova – peer review is a process about which they know nothing, except that it produces answers they don’t like (note: for those who read my previous censorship post and didn’t see the update, I’ll say that I was wrong about her site: I’m being allowed to comment freely). What’s probably most striking about that post is the level of ignorance on display: about peer review itself, and how it works, but also about prior art. You’d think that problems with PR had only just been discovered. I did try to point that out but as you’d expect, it fell on stony ground.
Peer review is nothing more than argument from authority and should be considered entirely irrelevant when evaluating the science
Comment #1 at JoNova, by “Truthseeker”. Of course, you know the old proverb: if someone calls themselves Truthseeker, then…”. But anyway: TS’s argument is a very common one: what we really care about is the quality of the science: what do we need a bunch of anonymous gatekeepers for?
Weeeellll… there are several answers to this. Let’s start with the most obvious: there’s an avalanche of papers out there, and not enough eyeballs to read them all. Journals like Nature publish less than 10% of what’s submitted (although note they are sifting for (ideally) both high quality and “excitement”; arguably, they veer off to the latter when pushed and may sometimes neglect the former). I found Rejection rates for journals publishing in the atmospheric sciences, from which I’ve taken the figure, and this quote: “Seventy-nine percent of journals have rejection rates of 25%–60%”.
OK, so hopefully you accept that we need some kind of gatekeepers to staunch the flow, but how then do we account for the common notion that peer review improves or proves quality or scientific merit? I have two answers:
* in practice, we find it does. Science is what works, bitches. Compare it to other ways of doing things.
* it doesn’t prove merit. Many many papers languish unread and uncited in reviewed journals. The ultimate test of the worth of your work is whether people choose to read and then build on what you’ve done. All peer review does is help you (the reader) by removing some drivel and pointing you towards some hopefully interesting stuff; and you (the writer) by providing a higher chance of people reading your stuff. There’s a reason people fight like rats in a sack to get their work into Nature, after all.
For a completely opposite approach, we already have full open-access no-peer-review publication: blogs. Anyone can write what they like and reach the entire world (I’m ignoring arXiv, about which I know nothing). Which suffer from the obvious problems.
“Peer-review” is an ENCLOSED system that no one can challenge
Comment #4.2 from Joe Lalonde. If you’re a “skeptic” seeing all your favourites shot down and reduced to producing their own journals, this is likely to seem true. As a normal scientist faced with some silly reviewer who refuses to see it your way and who is mysteriously backed up by the journal editor, it sometimes seems the same. But actually it isn’t.
Example from my own humble oeuvre: On the Consistent Scaling of Terms in the Sea-Ice Dynamics Equation by me and a cast of luminaries. That was initially rejected by not one but two referees. Ref 1 said it was true, but so obviously true that it wasn’t worth publishing. Ref 2 said it was obviously false. We managed to persuade the editor that ref 2 was wrong, but that because of ref 2, ref 1 must also be wrong (you might have hoped that the editor would have noticed this contradiction by himself, but editors are busy people).
And of course, there are already open-review journals. They aren’t in a majority, but they exist.
The other point is that most reviewers have experience of the bastard review system themselves, and can be quite sympathetic.
Anyway: as an outsider, who thinks of themselves as an outsider, and talks to no-one but outsiders, its very easy to get the wrong idea.
The effectiveness, and the desirability, of peer review is negated where a ostensibly scientific subject is politicized
Comment 5.3 by Eric Simpson.
Continuing: If one side ends up controlling peer review, and if that side is pushing for a “cause” that has nothing to do with the science, peer review is worse than worthless. Again, this is what it looks like from the denial-o-sphere: they are all so distant from the real science, that all the scientists look to them to be clustered together. But in reality there is vibrant discourse – well, sometimes. The comment is ill-posed, because the stuff about “sides” doesn’t really work. And there are so so few decent test-cases. I can’t think of a single paper that the “skeptics” can put forward that should have been published, that wasn’t. I suppose they’d retreat to “but the system is so biased against us we don’t even try” sort of paranoia. But that’s just paranoia.
Let the free market review the papers
(this is JoNova’s idea). Hmm, well, maybe. Its easy to forget sometimes that PR has evolved into its current form, and perhaps things have changed. The rise of the internet makes swift and open feedback entirely practical. But what is JN’s programme?
one named editor solely makes the decision to publish, and they can ask advice from reviewers, whomever they should choose. The reputation of that one editor should depend on the value of the papers they pass… They need to be paid, and the best ones, more. Editors are, currently, usually unpaid. They do the work either out of love, or because it reflects credit on their career. Paying them – presumably, significant sums – would change the game (one obvious problem: institutes generally OK people taking time off to edit, because its for the general good, and because they aren’t being paid. If they *were* being paid, that might change). I’d be concerned that editors would then have a (strong, financial) incentive to stuff papers into the journal. Which is the last think you want. I don’t find the rest of her suggestion terribly well thought-out either.
disappointingly, people on that blog haven’t taken up her idea and subjected it to constructive criticism, which is a semi-ironic implicit comment on the entire “skeptic” worldview.
What would you do then?
Pah, you mean in an ideal world or in practice? As I and many many other people have said, addressing the flow at its source would be best: which would mean stop judging people on sheer paper count. But that’s wrapped up in so many things, including the centralisation of decision-making, that its hard to see as realistic.
In practice I think a transition to an open review system would be very helpful, and solve some of the existing problems. That could also include journals listing papers they rejected before review, if you could work out some way around the copyright or priority problems.
[Update: thanks for your comments. I find it fairly amusing that the majority of commenters here are able to say My experience of peer review has been… as opposed to the denial-o-sphere’s fairy tales about what they imagine peer review is like. Of course, since they aren’t in any way restricted by reality, the d-o-s puts in far more comments.]
[Uupdate: see-also VV’s Peer review helps fringe ideas gain credibility.]
Refs
* Their own private reality
* http://www.realclimate.org/index.php/archives/2005/01/peer-review-a-necessary-but-not-sufficient-condition/
* http://www.realclimate.org/index.php/archives/2005/01/peer-review-ii/
* http://julesandjames.blogspot.co.uk/2013/03/peer-review-problems-at-egu-journals.html
* Bad Science
* Unless you plan to do something really bad, why do you insists being anonymous?
* Some links from Eli
* Fix the incentive structure and the preprints will follow – David L. Stern
* Peer review: Troubled from the start Alex Csiszar, Nature, 19 April 2016