The Perils of Peer Review

There is an awful lot of crap taxonomy out there. Incoherent ramblings, near-unidentifiable taxa, or "new" taxa of dubious distinction from their previously-published relatives are all too common. Because inadequate taxonomic works can create an enormous hurdle for subsequent workers*, many suggestions have been made on how to reduce the number of such publications. One point that I've heard raised a number of times recently is that the current ICZN places very few limits on where a new taxon can be published, and the suggestion has been made (at least informally) that only names published in peer-reviewed publications should be acceptable. While I can see the appeal in this proposal, I disagree - I don't think the ICZN should introduce such a requirement.

*As Charles Michener (1963) put it: "In other sciences the work of incompetents is merely ignored; in taxonomy, because of priority, it is preserved, and too much of the time of subsequent taxonomists is devoted to straightening out work of such people".

At the risk of stating the obvious, peer review is not a guarantee of quality. A lot of absolute dreck has survived the peer review process unscathed; a lot of excellent non-peer reviewed work has been published. Indeed, it is worth keeping in mind that formal peer review is, for the most part, a quite modern phenomenon. Until fairly recently, it simply wouldn't have been logistically feasible to send copies of a manuscript to multiple reviewers - especially for researchers in places like Australasia who were often working in isolation and for whom the only suitable reviewers would have been on the other side of the globe. A significant number of major taxonomic monographs that are still being referred to today were never peer reviewed.

There are also limitations to peer review that are particularly applicable to taxonomy. A reviewer of a manuscript is always forced to exercise a certain degree of faith in the author(s) he/she is reviewing. They can re-run the data analyses reported in the manuscript to see if the sums add up, they can try and think of any other analyses that the author(s) should have done but didn't, but generally they must first make the assumption that the author(s) are reporting their data accurately (almost the only alternative would be to re-run the entire investigation from scratch, an investment of time that most reviewers would simply not have the freedom to make). Unless there are great glaring holes in the reported data, most reviewers may not be likely to spot cases of fabrication, omission or basic error*. This is a particular limitation in taxonomy because of its often interpretive and material-dependent nature. If you present me with a manuscript saying "Specimen group A and specimen group B have the following distinct characters, and should be regarded as separate species", then I am limited in how much I can check this assertion because most likely I do not have access to your specimens.

*To give a practical example of what I'm trying to say: imagine you said to me, "Today I saw two red cars and one blue car; therefore, red cars are more common than blue cars". I can criticise your conclusion on the basis of the data you've given me (for a start, I would hardly think that three observations is a large enough sample to be statistically significant), but I would have no way of knowing whether you actually had seen two red cars and one blue car.

Of course, while both these points are worth keeping in mind, neither one is in itself a reason not to introduce a peer review requirement into taxonomy. The recency of peer review would not be likely to be a problem because most revisions of the Code have not been retroactive. And while peer review may not ensure that only quality work is published, it usually still acts as a general filter to prevent the worst abominations. The real reason why I don't think that the Code should introduce a peer review requirement is that if you are going to require something, you must first be clear about what it is that you are requiring.

Last year one of the big news items for vertebrate palaeontology was Aetogate, in which allegations of academic misconduct were levelled against an American palaeontologist and his associates. Without wanting to comment specifically on that whole sordid affair, one of the major take-home messages that I thought came out of it was the difficulty of defining "peer review". Among the accusations being made was that the journal publishing many of the allegedly offending articles could not really claim to be "peer reviewed", because the "reviewers" were made up of people closely connected to the journal itself and so not impartial. If the reviewers of a manuscript have a vested interest in the progress of that manuscript, then obviously their suitability as reviewers is questionable.

Unfortunately, the solution to this problem is not as simple as requiring that a manuscript must be reviewed by someone other than the author's nearest and dearest. If one is working in a specialist field that encompasses only a small number of workers, then there simply may not be any qualified reviewers other than one's nearest and dearest (or furthest and most loathed, which is arguably just as bad). And because the details of the review process are not usually made public (for perfectly valid reasons) there is usually no direct way of telling after publication what level of review a manuscript has passed through. As I've discussed in a previous post, it can often be contentious enough establishing whether or not a name has been properly published. Imagine the confusion that could ensue if subsequent workers had to establish whether it had been properly reviewed.

So what is the appropriate solution to inadequate taxonomy? As it happens, the ICZN already has some powerful tools up its sleeve. It has the ability to eliminate problematic names or, if necessary, entire publications from nomenclature. It is also worth noting that many of the workers who have suggested a peer review requirement are not considering just the simple quality of taxonomic work, but more its currently dispersed nature. In that case, a central registration system (currently being proposed and developed for the ICZN) may remove a large number of the current issues. Registration is a concept that is not without its issues, but at least determining whether or not a name has been registered would be much more straightforward than determining whether or not a name has been reviewed.

REFERENCES

Michener, C. D. 1963. Some future developments in taxonomy. Systematic Zoology 12 (4): 151-172.