Having never thought about peer review in much detail, I came across two independent articles on the subject today. The first came courtesy of my friend Gill Norman who used it as light lunchtime reading before posting it on Facebook to depress the rest of us. It’s an article from Breast Cancer Research which summarises the evidence of failure in peer review. My knee-jerk reaction was that peer review is a kind of public scrutiny and that public scrutiny is a (Popperian) good. Having read that piece, as well as this recent blog post by Andrew Gelman, I’m not so sure.
If you don’t know who reviewed your article (which is standard practice) there’s nothing public about the scrutiny. And, if Gelman is anything to go by, reviewers don’t spend much time on each paper – he claims to spend 15 minutes. At that rate, they might be able to say whether you had chosen an appropriate method, but I doubt they would have time to do much else (such as assess whether the method had been correctly applied, the literature had been fairly reviewed or the results clearly presented).
So why do we do it? My first thought was to turn to economics: perhaps peer review performs a signalling function? A source (OK, it was Gill) made an alternative suggestion: “a biologist would do just as well as an economist: the journals turn bits of their anatomy bright red and the academics show up with titbits, elaborate dances and possibly a well constructed nest/burrow/set of antlers etc…”
I’m interested in Gelman’s suggestion that some disciplines (like mine, for example) might move away from peer review and towards self-publication on the internet. I had a quick look one of early publication sites – the Social Science Research Network. When you search the SSRN for a particular topic, the results are presented in order of downloads – another form of signalling. Once these papers start to pick up citations they become credible even without peer review. They also get credibility from the names or institutions of their authors (as, indeed, do papers in peer-reviewed journals). I wonder what the downside is for authors? I’d like to know how easy they are to find in a literature search and whether they will have the same longevity as equivalent peer reviewed articles. I suppose a lot will depend on whether funders go for it…