Failures of Community Peer Review

Some think we can harness the inclination of people to blog/comment/twitter to improve peer review through open review/community review comments on the websites of individual papers. After all, many more scientists critically read a paper than just the peer reviewers, and many of us have serious complaints with published papers... and why should we all have to independently discover the same flaws? Shouldn't all readers benefit from one astute reader's comment about the mathematical model breaking down at certain values, or an informed reader's comment about some minute detail about the model organism that calls into question the conclusions of the paper?

However, while scientists love to complain about mistakes in papers, and many scientists love to blog... few seem to comment on websites of published papers. When I started googling for some data on open/community peer review, I first found reports of Nature's 2006 experiment with it:

Most comments were not technically substantive. Feedback suggests that there is a marked reluctance among researchers to offer open comments.

That was just a four month-long trial, which authors submitting to Nature could opt into if they chose (only 5% of authors did). Journals dedicated to the community peer review process, such as PLoS One or the BMC journals, must have more success getting more than just the official peer reviewers to comment on their papers.

So I looked up two papers published this summer in these journals, for which I had been a conventional, anonymous reviewer (perhaps I should start signing my reviews Phagenista?). Both had been reviewed and re-reviewed by me and one other referee. No other readers have yet chosen to comment on either in the >4 months both have been online.

I expanded my search, figuring that my search strategy of "articles I've already read critically" was too limiting. I chose to look through all the PLoS One articles on women's health, which I thought might be a field with some controversial ideas that would encourage commenting. Of these 32 articles, only one has any comments other than the two official reviews. Again, anecdotal data suggesting the community isn't doing much community peer review.

PLoS One does not have easy-to-find statistics on the average number of comments per article, and I didn't find a list of the most-commented articles. I googled to see if others had gathered such statistics and found this and this. Similarly, I couldn't find any official statistics for BMC journals, though I counted only 17 comments posted in November. As hundreds of journals are hosted by BMC, this seems like a very low comment yield. A more comprehensive study says only 2% of articles in BMC journals attract comments. Nascent's analysis of BMC comments included some nice figures, including this one classifying seven broad categories of comments:


So we biologists are not yet enthusiastically participating in community peer review. That doesn't mean it can't work, or that it isn't a vibrant system for other sciences. Perhaps one way to increase commentary would be to encourage students to post comments on papers they read for journal clubs. The journals require comments to be posted with real names, not handles, and that might be one reason people are loathe to publically comment. I know that I have always chosen to be anonymous when reviewing for the PLoS and BMC journals, and I am apparently not alone in my concerns about remaining anonymous to the authors of the papers I critique.

What I'd like to know is whether people who do download PLoS One or BMC journal articles read what comments are there... either before reading the article, or on a subsequent visit to the paper's website. How many biologists just download these open access papers through PubMed, and therefore don't even see if there are any community peer review comments?

Failures of Peer Review, Part I

I think a lot about the state of peer review. I spend a substantial amount of time on peer reviews (and re-reviews, and re-re-reviews... it would save time to accept more papers), but that's just a symptom of my belief that without thorough, fair and constructive peer review, science doesn't work. Especially as more journals publish more papers, and as online-early articles are distributed even though they contain errors that will be corrected in proof... we're being saturated with scientific information. While Faculty of 1000 and other TV guides to the recent scientific literature can help one find the better articles in the sea of decent ones, rigorous peer review is the bedrock assumption underlying all of the articles in the literature. When we read a peer-reviewed, published article we assume that the authors are not trying to deceive us, and we assume that they correctly drew conclusions from their data. And, despite the fact that two or more scientists evaluated each of these articles prior to publication, we often find we disagree with their conclusions, or feel the published experiments were done incorrectly. The goal of peer review is to help papers go into the literature in the least flawed form possible, and as we have less and less time to devote to reading each published article, peer review becomes more important.

Or, we need to move to a wiki commenting format on each and every article, which seems like it will increase the time required to read each "paper" since one will have to read all the comments as well...

I could go on for far too long about peer review... but I was prompted to write about it today because a friend of mine sent me the best example of a peer review failure that I'd seen for a very long time.

In the reference list of von Szentpály, L. 2008. Atom-Based Thermochemistry: Predictions of the Sublimation Enthalpies of Group 12 Chalcogenides and the Formation Enthalpies of their Polonides. The Journal of Physical Chemistry A (published online 11/17/08), we find:

Reference 27: http://en.wikipedia.org/wiki/Polonium#Toxicity. Weight-for-weight, polonium is about 10^11 times more toxic than hydrocyanic acid, HCN. The element and its compounds have been found in tobacco and, particularly, cigarettes. If inhaled, polonium is deposited in the lungs, where its nucleus emits α-particles causing severest damage to the tissue. The amount of polonium taken in by a smoker is approximately equal to that taken in from all other sources.

The peer reviewers didn't insist on a more stable and authoritative reference to support this than wikipedia. Surely one sign of the impending apocalypse... but the bigger issue is that the article appears to have incorrectly cited wikipedia, at least for the information listed on that page today, 11/18/08 (the Toxicity section had not been edited since 9/5/08). The wikipedia page purports:

By mass, polonium-210 is around 250,000 times more toxic than hydrogen cyanide (the actual LD50 for 210Po is about 1 microgram for an 80 kg person (see below) compared with about 250 milligrams for hydrogen cyanide

I have to say that if I were peer reviewing an article that had a wikipedia page as a reference I would have checked it out, and likely would have noticed the discrepancy between 250,000 and 10^11 times. But, there has to be a more appropriate reference for 99.9999% of facts than wikipedia (communication and information science researchers studying wikis being one obvious exception).

Job Depressions

The recession we're in is definitely bad news for academic job seekers... while certain private, well-endowed universities (and UT) can maintain their growth and attractive packages if they choose to... the most state schools are stuck with shrinking state tax revenues, and many private schools are instituting hiring freezes. Some schools are already laying off untenured faculty, so it's not as if already having obtained a job is a guarantee of security for young scientists.

There has been substantial sturm und drang on some senior postdoc blogs, especially in their comments, about this job shortage. And while no one's perspective will be one-size-fits-all, here's my two cents on why it shouldn't matter if one postdocs for a couple extra years. Inherent in this are things like "there is never a right time to have a child," that none of us got into science for the high salaries, and that scientific discovery is what makes one want to get out of bed in the morning.

When I started grad school, my professional goal was to get an academic job without doing a postdoc. I didn't see the value in spending post-PhD years not working towards tenure. It was a perspective of a twenty-two year old who had spent the vast majority of her life in school, and who was itching to get control of her future.

My very wonderful primary PhD advisor repeatedly told me that postdocing was the best time in your scientific life. You've got research independence, you have flexibility and freedom to pursue a wildly different path in your next postdoc, and you're shielded from the demands on your time that accounting, teaching and meetings impose on you. My co-advisor only spent two years postdocing before taking up her R1 job, and she wishes she had done another postdoc with a particular husband and wife academic team at a non-US institution. Essentially, she had accepted a job offer before she realized which lab she really wanted to apprentice in.

What both of my advisors successfully convinced me of was that as long as I am working on what I want to be working on in science, it doesn't matter what my job title is. There's a great confidence in feeling that you're in control of your scientific research, and that it's a journey. When you're explaining your research plan during a job interview, and you say that this is the work you're going to do in the next five years... it's clear that you don't need this particular job. Or any particular job. That you have a dedication, a unique research perspective, and you'll bend any position you take into one that will let you do what you want to do.

If you are a postdoc who is unhappy with your current position, then leaving that postdoc is likely a good call. Hopefully after you have submitted or published much of your work from that lab. If you are in a postdoc where you don't feel fairly independent, that is probably not a good postdoc to be in and it's probably not helping you build the skill set that would need for running your own lab.

That's a very broad stroke appraisal that doesn't address many of the specific problems being discussed in blogs. Such as how to entice mentors to publish good manuscripts their postdocs are generating. Or how to convince a mentor that publishing in a journal that's not C/N/S is worthwhile. Or how to find a supervisor who will actually be a mentor. These are variations on trying to cope with ones current postdoc, but it should be said that postdocing in and of itself isn't the enemy. Some advisors are, the perspective of some fields on postdocs as competent slave labor definitely is a problem.

I do understand that there is a limit to the number of years a person can postdoc before their CV looks a bit stale. However, I have seen older candidates and people with more than six years of postdoc experience get great tenure-track jobs. These people have spent their postdocs working on what they know is an important research thread... even if others don't always agree. They publish well-reasoned papers on experiments others wouldn't have thought to do. They waited for a number of their papers to get noticed, and waited for their ideas to change what hiring committees thought were important.

I was recently asked by a science journalist to talk about my primary PhD advisor's approach to science for an article she was writing about him. The research accomplishments that I kept highlighting were all from his (6 years) of postdocing. He hasn't been sitting on his laurels as a tenured professor, but his revolutionary papers, so far, have been from his postdoc years. And, frankly, we all get tenure based on our tenure-track funding but our postdoc papers -- they are the ones that have had enough time to be well-known, or to be cited enough to help our h-indices.

There is a theme here of cream rising to the top, even if others don't realize early on that the person is cream (acknowledge their creaminess?). Will all current postdocs get TT positions? No, but then again, it's been a long time since that was the case... if it ever was. Some people prefer industry or policy jobs -- for some it was always the plan to go that route, others discover that is their passion, and some others may end up feeling like it's the best of a series of unappealing options. I don't think that non-TT jobs should be considered consolation prizes, and I think having a job you like or love is the goal. Some people realize that science isn't for them and some people aim for a level of success in science that probably won't happen for them. Almost all of us will fail to become endowed chairs at Harvard and most PhD scientists will fail to become tenured at R1 schools. But the people with good ideas, good communication skills and the stamina to survive the initiation rituals of science do succeed in having some kind of scientific career that makes them happy. At least in my discipline.

That was a lot more than two cents...

science spending spree!

Few things make me happier than ordering books without any thought to the price, because the no-cost-extension of the grant is expiring soon. I happen to work in an area where single-author monographs and contributed volumes from the 1950s-1970s are very useful, so some very expensive used books are coming my way!

I wonder how many other fields still find the literature (especially the non-journal literature) from the pre-pubmed era so useful...

A microbiologist in woman's clothing?

There's been another round of science blogs talking about the way woman present themselves as scientists... whether as a group we're too androgenous right now to attract modern young women to science professions, whether we're not taken as seriously as scientists if we dress in too feminine of a manner. I think we're all doing a fine job being ourselves, and that the younger generation of profs, at least, wouldn't think less of a professor who wears the latest fashions or sparkly shoes. Just as we don't think less of heavily tatooed scientists.

But... my only (anonymous) comment that failed to pass Female Science Professor's filter was one on how women in science's clothing is perceived by male scientists. The point of the comment was that men do look at female scientists (students) differently than they do male scientists (students), but the problem is we don't know whether or not that affects their bottom line opinion about the research, intelligence or contribution of those women. And I think this is more of an issue when the male and female scientists don't know each other well. At a conference a few years ago, I overheard two male professors discussing the necklines of female graduate students and their visible figures during one of the presentations. I hope that these men remember that the slender grad student with the plunging-neckline dress and jeans combo was presenting a poster on milk microflora... for at least as long as they remembered what her bust looked like. I always wore more modest clothes to conferences than I might wear at home, but now the incident comes up in the back of my mind when I'm packing for a meeting... because I would like to be remembered for my science and my mind first, and anything else -- being female, being American, being an able drinker, anything pertaining to my secondary sex characteristics -- second.

I don't worry about this with people who interact with me on a regular basis. If I'm wearing a nicer fabric, low-cut v-neck one day, my colleagues know from experience a crumpled t-shirt and fleece might emerge tomorrow. Even when I'm on my more modest behavior, I do wear clothes I like and that match my style. For my tenure-track job interviews, for instance, I wore very modern pantsuits with shoes I adored: Paul Green angular-cut heels, and Etienne Aigner strappy mary jane heels. And as I become more senior in my field I am sure I will care less and less about meeting clothing, both because I'll have had so many interactions with the people at the conferences, and because, I'm assuming, fewer male professors talk about the body shape of 50 year old women scientists they have known for 20+ years than the 25 year old grad students they just met.

CV roundup

The CV workshop was a mixed success. The grad students who had been most excited about it couldn't be there because of a family medical emergency, and she always contributes a lot of life to the meetings. Another grad student just left for a three month research trip to Asia. So the attendees were:

Frazzled female grad student, who's preparing for comps and looks contemptuously at anyone who tries to distract her from her current work

Ignored (by her lab) female grad student from a nearby lab, who also is facing her comps

The senior professor who sometimes joins our meetings who is an absolute grant/publishing dynamo

New male postdoc who had previously done a masters with senior professor


I brought five CVs that all had very different formats. Three were from within our department: a PhD student applying for a postdoc and two postdocs who had just received TT jobs (including mine). A third postdoc applying for TT jobs and a new assistant professor's CV rounded out the bunch. I also had copies of NSF and NIH-grant format CVs from both myself and the senior professor, but no one was particularly interested in grant-specific CVs formats.

One thing that I wish I had made clearer to Senior Prof was that this wasn't an "applying for a job" workshop. Senior prof has a very strong personal point of view -- publications, and to a smaller degree, grants, are all that matter. When he is on a job search committee, he doesn't read whole CVs, he scrolls to the publications and sees where and how much the candidate's been publishing. He also considers the reputation of the letter-writers when deciding whom to put on the short list. And while I would never say that publications weren't the most important factor in almost any application for a research job or research grant/award... that wasn't the message I wanted to send, that it doesn't matter what your CV looks like, it's your pubs, pubs, pubs.

Not every professor is as single-minded on this as Senior Prof (and most professors cannot match Senior Prof's publication rate... the coincidence is notable), and many look at CVs as not just a list of your qualifications, but also an example of how you present and organize information. That message might have gotten drowned out as we lapsed into another episode of "The Phagenista and the Senior Prof Show." Sigh... As a graduate student, lab meetings were called the "Dave, Jeremy and Phagenista Show"... but at least we were all the students!

The visiting student is likely going into science journalism after getting her PhD, and she seemed receptive to my ideas on how to spin many of her previous and current activities (volunteering for AIDS organizations, writing for the local left-wing rag) into sections on her CV. So at least one person got something out of the workshopping. Not uncoincidentally, she also brought her CV with her for us to discuss. The new postdoc said that he would consider putting his teaching experience on his CV (he never has before). Frazzled female grad student was pretty silent throughout the whole thing.

Next time, and yes there will be a next time, I'll make everyone bring their CV, and I might not invite any senior profs... or invite them for the second part of the meeting... I do think there is a value in looking at the CVs of successful scientists, and seeing that there isn't one "right way" to do things.

And, true to Senior Prof's point, one of the recently-hired CVs we went over has utterly stellar content, but was the most cluttered of all we looked at. Clearly, that postdoc been allocating his time to those many (many) high-impact papers instead of gussying up the CV, and it didn't hurt him (at least three great TT offers)!