the youngest peer reviewers are often the harshest...

...illustrated by recent articles in the Chronicle and The Onion.

While I can't comment on otter research, the complaints of the humanities scholar about how graduate students read texts ring true for young scientists too.

Why is it that so many graduate students are reflexively critical of what they read? Why are they so eager to dismiss, dislike, and disrespect work that has already been done before they can even hope to begin to contribute?

I think there are some upsides to the arrogance that grad students initially bring to reading the established literature. Grad students, some of them at least, are the future of any field. It's a wonderful thing that they critically evaluate for themselves everything that's become conventional wisdom, because some of that conventional wisdom is wrong. And as we become more entrenched in academic science it's harder to spot the flaws in the conventional wisdom that we've embraced for 10, 20, 30 years...

I remain a very detailed reader and reviewer, but I'm not nearly as acerbic as I was as a graduate student. I also find I have much less time to take on accepted ideas when they are assumed in papers I review... most recently, I lost hours of a week's vacation working on my laptop trying to convince myself (or disprove the author's assertion) of one aspect of one virus' lifestyle. In the end, I now believe there isn't much/any data against the conventional wisdom, but it is crazy that I chose to spend a substantial fraction of my time at a beach house chasing this idea down. I'm too busy now, but I do envy the perspective and time that graduate students have when they read the literature, and when they assist in or write their own peer reviews.

Failures of Peer Review, Part 2

I've been confronting a few failures of peer review recently, and I think it's time to revive peer reviewing posts. Like most failures of peer review, it may very well be the editors who dropped the ball and not the reviewers -- or a combination of the two.

1. I came across a 2008 paper with an incredibly low resolution figure that prevented me from reading the labels of the data points (and there were >100 labels). This wasn't a problem of my computer (the pdf looked just as bad -- even more pixelated and harder to resolve at higher magnifications -- with several viewers), didn't get better on the print out, wasn't available at a better resolution in the html version), I asked others to try their hand at reading it with no success, etc. Now, this wasn't a very important figure to read to get the conclusions of the paper, I admit... it's more that the figure had to exist to get the paper published. But it was important to me for a study I'm trying to start, and I had to spend several hours deciphering the figure. I know that the reviewers could not have been able to evaluate the data in the published version of the figure, so either the journal published a lower resolution version than they gave the reviewers or neither/none of the reviewers actually read/checked this figure.

2. Twice in the last month I've been asked by senior scientists what I think of a recent paper in our field. I happen to know the paper well, since I reviewed it and recommended rejection on a number of solid grounds. Even though I still think the study is flawed, I must have been in the minority, since the paper was accepted with only a few changes. However few people will read the paper thoroughly enough to see the newly introduced hedging statements; in this busy day and age, all most people will read or remember are the title, the abstract and the quality of the journal it was published in.

So I gave these senior scientists my detailed rationale for why one shouldn't trust the paper's conclusions, it was easy to convince them, and I doubt either one will be citing it anytime soon. But I can't shake the feeling of being a failure myself as a reviewer, since the paper did get into the literature virtually unchanged. Rereading the review, I was constructive in my criticism, but I clearly challenged the premises of the paper, and backed up my argument with references. I reiterated the cause for rejection in more frank language in the confidential comments to the editor. And maybe, with another editor, my comments would have caused the paper to be rejected... but the next time I feel strongly about a rejection I will be tempted to use more emphatic language.

two cultures, two deadlines.

While many are talking and blogging about the two cultures these days, I want to highlight the gulf between our academic disciplines with an article from the Chronicle of Higher Education on the attempts to speed up publication in the humanities. While peer reviewers in science are being pushed for one or two week turnarounds from an increasing number of journals, the humanities are trying to move the average time from eliciting a book review to publication from year(s) to months.

The article says one of the reasons for the long turnaround is that "reviewing tends to be a rather thankless task, so it gets pushed to the bottom of a reviewer's agenda." Certainly, it's a similar situation in the sciences... but there seems to be a stronger system of reciprocal altruism in the sciences, where we all need to get our papers published, we all want them published quickly, and we do (largely) comply with the deadlines we're given. The article also highlighted the low number of reviews that can be published in each monthly/quarterly issue of a journal, and it could be that online-only journals will spur faster publication times in the humanities just as they have in the sciences.

It's not an apples-to-apples comparison between the fields: the publications being discussed are the peer review themselves.

I appreciate how quickly biology progresses these days, but it can be overwhelming. Having to keep up with online-early publications, the myriad table of contents emails, the automatic search emails, the increasingly short review times for journals... at times the slower publishing pace of other parts of academia could be very appealing.

Fixing peer review, part 2

I saw this over at DrugMonkey. Cell, a biology GlamourMag, claims that it is redefining its publication standards and policies in response to the digital publishing era:

One issue in particular that we at Cell will be focusing on in 2009 is redefining what constitutes a "publishable unit" in the age of electronic journals and how we can best present the information content of a scientific article online. [...] The scientific article of the future will no longer be tied to the constraints of a printing press and will take advantage of all the opportunities afforded by the web to introduce a hierarchical rather than linear structure, increased graphical representations, and embedded multimedia. Inherent in our thinking about the scientific article of the future is the need to address the current unchecked growth in the amount of supplemental and supporting material and to identify constructive, well-defined guidelines for what is reasonably and appropriately included in a unit of scientific advance.

I agree with everything DrugMonkey said -- what should be in the paper should be in the physical paper. However, I am heartened by one aspect of expanding supplementary online material: I would like to return to an earlier practice of reporting genetic analyses and require that sequence alignments used in publications be publicly available as supplementary material. This would not take up a large amount of space (the nexus file format, which I would propose as the standard, is both compact and informative -- it will preserve annotations of genomic regions if the authors wish to add them).

How would this improve science, and why is this being filed under "fixing peer review?" First and foremost, it allows future groups to build upon previously published work, and know concretely how their analysis differs from the published work. If you start with a published (as supplementary material online) alignment, and you have a serious problem with aspects of the older alignment (e.g., they neglected to take conserved secondary structures into account), this gives you an important point for discussion to explain why you results do not match up with previous results. We would no longer have to speculate -- we could test such an assertion. Secondly, it will speed up research, as fewer groups would have to reinvent the wheel and re-do the same work that has been done by other groups. And, as I argue in the first point, this does not mean that the published work would be used as an unevaluated black box -- it can, and hopefully will, be critiqued and refined. The reason it is on my mind at the moment, however, is the way this will assist peer review. I just spent far too many hours yesterday trying to replicate an alignment in a manuscript I was reviewing to test a theory I had about a phylogenetic tree in it. If my theory were correct, it would undermine all the paper's conclusions, but I could not see if my theory was even plausible without seeing at least a partial alignment. Had the authors been required to upload their alignment with their manuscript, this would have taken me less than five minutes to check. Under our current system... let's not discuss how long it took. I know some will say that I should have just suggested the reanalysis without attempting it, but I have learned the hard way that if you don't provide evidence (a hint of reanalysis, citations) for your comments in peer review, the authors sweet talk the editor and convince them you didn't know what you were talking about.

When sequence analysis was a new field, journals published the entire alignments in the physical paper journals. This quickly became impractical, as computing power and algorithms advanced, and it no longer was required or encouraged to publish your alignments. However, you are always supposed to supply your alignment to any and all people who contact you after your paper is published, and some attempts have been made at alignment repositories (EMBL's closed in 10/2008). After all, bioinformatic analyses based on nucleotide or amino acid sequences are only as accurate as the alignments underlying them. I think requiring alignments as supplementary online material would be a fantastic requirement for the increasingly digital age. Some journals already require this, but none of those I regularly read or review for. Maybe this is not what Cell meant at all... but I'm running with this.

Plus, I am in favor of anything that improves my time-to-completed-review. (I have peer reviews #5 and #6 of 2009 on my computer today. I need to speed things up or I'll never get tenure)

Fixing peer review, part 1

A colleague who knows my strong feelings on peer review sent me a recent article about "the tragedy of the reviewer commons." It doesn't address all my concerns with the peer review process, but it makes a few excellent, concrete suggestions:

"First, have colleagues pre-review your manuscript before submission. This was a time-honoured norm in publishing, but has diminished in recent years. Its decline could be due to increased pressure to publish quickly, competition between colleagues, increases in the numbers of co-authors on papers, and ironically, less time available to comment on our colleagues' papers because we are submitting and resubmitting more of our own papers."

"...journals could require a statement in the cover letter such as 'We confirm that should our study have been previously submitted to another journal, we have taken all reviewers comments into account in revising our manuscript for submission to…', has the merit of increasing author awareness and responsibility, without biasing editors because all submissions, whether or not previously rejected, would have to carry this statement."

"Our experience has been that, when they are chosen appropriately, students and post-docs are very conscientious and thorough in preparing reviews. Moreover, this is a route by which a busy senior scientist can still participate in the review process, and can mentor younger scientists in the craft of writing a good manuscript review."

Amen, my ecological brothers! (The authors are Michael Hochberg, Jonathan Chase, Nicholas Gotelli, Alan Hastings and Shahid Naeem)

I could not agree more with each of these points. As a reviewer I am constantly amazed by the shoddily prepared manuscripts that people submit to journals... and I wonder if authors had to first reveal their draft to their colleagues, who vote on their career advancement, whether the manuscripts would be in better shape even before the internal pre-review.

The authors' second point is remarkably creative, and something I would never have thought of (and something I have not heard anyone else mention). We all have been asked by multiple journals to review the same paper, and when you see that the authors did not revise the manuscript AT ALL in response to your first, rejecting review... it creates resentment. For those of us who take the time to write detailed, constructive reviews, this will offer us a direct recourse. If an author is caught in the act of wantonly ignoring a previous round of review, the reviewer can document this and affect change. Perhaps the authors will be blacklisted at that journal -- a concrete negative effect of not taking advantage of the first round of reviews. However, I do not think all reviewer's comments should affect change in a manuscript, so this policy has to be employed with some common sense by editors. I wouldn't hang up an author for not changing figure formatting that I had previously suggested, but I might talk to an editor about authors who ignored cogent criticism of their statistics.

And finally, how can we teach our students to become competent reviewers if we don't expose them to the process? Because too many scientists take their own peer reviews so negatively, I don't think that's the best way to teach younger scientists how to critique and improve papers. Plus, younger scientists often take the responsibility of reviewing to heart, and spend inordinate amounts of time on their first reviews, checking references more diligently than the busy senior scientists could have time to do.

These aren't a panacea for the problems of peer reviewing, but I would wholeheartedly endorse both the cultural and official changes to review suggested by Hochberg et al.

Failures of Community Peer Review

Some think we can harness the inclination of people to blog/comment/twitter to improve peer review through open review/community review comments on the websites of individual papers. After all, many more scientists critically read a paper than just the peer reviewers, and many of us have serious complaints with published papers... and why should we all have to independently discover the same flaws? Shouldn't all readers benefit from one astute reader's comment about the mathematical model breaking down at certain values, or an informed reader's comment about some minute detail about the model organism that calls into question the conclusions of the paper?

However, while scientists love to complain about mistakes in papers, and many scientists love to blog... few seem to comment on websites of published papers. When I started googling for some data on open/community peer review, I first found reports of Nature's 2006 experiment with it:

Most comments were not technically substantive. Feedback suggests that there is a marked reluctance among researchers to offer open comments.

That was just a four month-long trial, which authors submitting to Nature could opt into if they chose (only 5% of authors did). Journals dedicated to the community peer review process, such as PLoS One or the BMC journals, must have more success getting more than just the official peer reviewers to comment on their papers.

So I looked up two papers published this summer in these journals, for which I had been a conventional, anonymous reviewer (perhaps I should start signing my reviews Phagenista?). Both had been reviewed and re-reviewed by me and one other referee. No other readers have yet chosen to comment on either in the >4 months both have been online.

I expanded my search, figuring that my search strategy of "articles I've already read critically" was too limiting. I chose to look through all the PLoS One articles on women's health, which I thought might be a field with some controversial ideas that would encourage commenting. Of these 32 articles, only one has any comments other than the two official reviews. Again, anecdotal data suggesting the community isn't doing much community peer review.

PLoS One does not have easy-to-find statistics on the average number of comments per article, and I didn't find a list of the most-commented articles. I googled to see if others had gathered such statistics and found this and this. Similarly, I couldn't find any official statistics for BMC journals, though I counted only 17 comments posted in November. As hundreds of journals are hosted by BMC, this seems like a very low comment yield. A more comprehensive study says only 2% of articles in BMC journals attract comments. Nascent's analysis of BMC comments included some nice figures, including this one classifying seven broad categories of comments:


So we biologists are not yet enthusiastically participating in community peer review. That doesn't mean it can't work, or that it isn't a vibrant system for other sciences. Perhaps one way to increase commentary would be to encourage students to post comments on papers they read for journal clubs. The journals require comments to be posted with real names, not handles, and that might be one reason people are loathe to publically comment. I know that I have always chosen to be anonymous when reviewing for the PLoS and BMC journals, and I am apparently not alone in my concerns about remaining anonymous to the authors of the papers I critique.

What I'd like to know is whether people who do download PLoS One or BMC journal articles read what comments are there... either before reading the article, or on a subsequent visit to the paper's website. How many biologists just download these open access papers through PubMed, and therefore don't even see if there are any community peer review comments?

Failures of Peer Review, Part I

I think a lot about the state of peer review. I spend a substantial amount of time on peer reviews (and re-reviews, and re-re-reviews... it would save time to accept more papers), but that's just a symptom of my belief that without thorough, fair and constructive peer review, science doesn't work. Especially as more journals publish more papers, and as online-early articles are distributed even though they contain errors that will be corrected in proof... we're being saturated with scientific information. While Faculty of 1000 and other TV guides to the recent scientific literature can help one find the better articles in the sea of decent ones, rigorous peer review is the bedrock assumption underlying all of the articles in the literature. When we read a peer-reviewed, published article we assume that the authors are not trying to deceive us, and we assume that they correctly drew conclusions from their data. And, despite the fact that two or more scientists evaluated each of these articles prior to publication, we often find we disagree with their conclusions, or feel the published experiments were done incorrectly. The goal of peer review is to help papers go into the literature in the least flawed form possible, and as we have less and less time to devote to reading each published article, peer review becomes more important.

Or, we need to move to a wiki commenting format on each and every article, which seems like it will increase the time required to read each "paper" since one will have to read all the comments as well...

I could go on for far too long about peer review... but I was prompted to write about it today because a friend of mine sent me the best example of a peer review failure that I'd seen for a very long time.

In the reference list of von Szentpály, L. 2008. Atom-Based Thermochemistry: Predictions of the Sublimation Enthalpies of Group 12 Chalcogenides and the Formation Enthalpies of their Polonides. The Journal of Physical Chemistry A (published online 11/17/08), we find:

Reference 27: http://en.wikipedia.org/wiki/Polonium#Toxicity. Weight-for-weight, polonium is about 10^11 times more toxic than hydrocyanic acid, HCN. The element and its compounds have been found in tobacco and, particularly, cigarettes. If inhaled, polonium is deposited in the lungs, where its nucleus emits α-particles causing severest damage to the tissue. The amount of polonium taken in by a smoker is approximately equal to that taken in from all other sources.

The peer reviewers didn't insist on a more stable and authoritative reference to support this than wikipedia. Surely one sign of the impending apocalypse... but the bigger issue is that the article appears to have incorrectly cited wikipedia, at least for the information listed on that page today, 11/18/08 (the Toxicity section had not been edited since 9/5/08). The wikipedia page purports:

By mass, polonium-210 is around 250,000 times more toxic than hydrogen cyanide (the actual LD50 for 210Po is about 1 microgram for an 80 kg person (see below) compared with about 250 milligrams for hydrogen cyanide

I have to say that if I were peer reviewing an article that had a wikipedia page as a reference I would have checked it out, and likely would have noticed the discrepancy between 250,000 and 10^11 times. But, there has to be a more appropriate reference for 99.9999% of facts than wikipedia (communication and information science researchers studying wikis being one obvious exception).