However, while scientists love to complain about mistakes in papers, and many scientists love to blog... few seem to comment on websites of published papers. When I started googling for some data on open/community peer review, I first found reports of Nature's 2006 experiment with it:
Most comments were not technically substantive. Feedback suggests that there is a marked reluctance among researchers to offer open comments.
That was just a four month-long trial, which authors submitting to Nature could opt into if they chose (only 5% of authors did). Journals dedicated to the community peer review process, such as PLoS One or the BMC journals, must have more success getting more than just the official peer reviewers to comment on their papers.
So I looked up two papers published this summer in these journals, for which I had been a conventional, anonymous reviewer (perhaps I should start signing my reviews Phagenista?). Both had been reviewed and re-reviewed by me and one other referee. No other readers have yet chosen to comment on either in the >4 months both have been online.
I expanded my search, figuring that my search strategy of "articles I've already read critically" was too limiting. I chose to look through all the PLoS One articles on women's health, which I thought might be a field with some controversial ideas that would encourage commenting. Of these 32 articles, only one has any comments other than the two official reviews. Again, anecdotal data suggesting the community isn't doing much community peer review.
PLoS One does not have easy-to-find statistics on the average number of comments per article, and I didn't find a list of the most-commented articles. I googled to see if others had gathered such statistics and found this and this. Similarly, I couldn't find any official statistics for BMC journals, though I counted only 17 comments posted in November. As hundreds of journals are hosted by BMC, this seems like a very low comment yield. A more comprehensive study says only 2% of articles in BMC journals attract comments. Nascent's analysis of BMC comments included some nice figures, including this one classifying seven broad categories of comments:
So we biologists are not yet enthusiastically participating in community peer review. That doesn't mean it can't work, or that it isn't a vibrant system for other sciences. Perhaps one way to increase commentary would be to encourage students to post comments on papers they read for journal clubs. The journals require comments to be posted with real names, not handles, and that might be one reason people are loathe to publically comment. I know that I have always chosen to be anonymous when reviewing for the PLoS and BMC journals, and I am apparently not alone in my concerns about remaining anonymous to the authors of the papers I critique.
What I'd like to know is whether people who do download PLoS One or BMC journal articles read what comments are there... either before reading the article, or on a subsequent visit to the paper's website. How many biologists just download these open access papers through PubMed, and therefore don't even see if there are any community peer review comments?
2 comments:
So far there's not much in PLoS ONE that relates to me. And the papers I've read there that were related to what I do are so poor that I didn't even know where to begin my critique.
When it starts with "What the hell were you thinking-?" that's not very constructive, you know?
And I wish these things were easier. I'm tired of having five thousand logins and passwords for all the different journals and websites, etc. etc.
It would help if there were a more universal system (sort of like in the blogosphere, where I can save my pseudo-ID pretty easily in only one or two formats for all the different blog outlets and enter comments without having to think about it. Where I don't have to type it in repeatedly (or have tons of different ones to meet all the different password requirements).
In other words: they need to MAKE IT EASIER.
Hmmm... interesting. The whole community peer review thing seems like one of those ideas that look great on paper and that we all know we *should* be doing-- like composting!-- but I don't really want my name out there on reviews (or a bin of scraps of food on my kitchen counter, for easy transport to the garden composting bin).
(people here compost a lot. They are very green. They do not shop at Walmart. I know I should be like them, but I am a big fail at responsible consumerism.)
And yeah, I usually just get articles straight from pubmed (actually, more often scopus or webofscience).
Post a Comment