At the recent Emerging Trends in Scholarly Publishing seminar, the fine folks at Allen Press had an interesting suggestion for how to fund PubCreds: as a nonprofit funded by membership fees from publishers. The model here is crossref.org, the folks who run the DOI system for scholarly publications. DOIs prevent broken links to articles and allow links between publications with different publishers. Publishers got together to develop crossref because it was something they all wanted, ultimately because users wanted it. The annual membership fees for publishers range from $275-50,000, depending on revenue, and there's a small fee ($1 or less) per DOI assigned. PubCreds is similar to crossref in that it's not a service any publisher could offer on its own. The key issue is whether PubCreds is, or could become, a service that publishers want to offer. Here there seems to be a collective action problem. Imagine a world in which most publishers (or alternatively, most journals in some...
More»
Last week Jeremy gave an invited talk on PubCreds at the 2011 Emerging Trends in Scholarly Publishing seminar run by Allen Press (publisher of Ecology and the ESA's other journals). His talk was well-received, and should be posted shortly on YouTube (by user 'AllenPress').UPDATE: Jeremy's slides are available here, and the video of his talk is here.Managing editors from all fields of scientific and medical publishing were a big chunk of the 200 or so attendees. Jeremy reports that all of the many people he spoke to, with one exception, named 'difficulty of obtaining reviews' as the number one problem with the peer review system. The exception is the managing editor of a nutrition journal, who has no problem obtaining reviews but nevertheless has to work 80 hours/week to maintain this problem-free state of affairs...
More»
Just an update on the latest progress on PubCreds. Owen Petchey and I have formally requested detailed data on the peer review system from a whole bunch of ecology journals. Basically, we've asked for anonymized data that would let us conduct the analyses we suggested in a previous post, to test for the 'journal-level symptoms' of a 'tragedy of the reviewer commons.' These data are available from online ms handling systems, although they aren't trivial to extract. We're very gratified that the British Ecological Society has agreed to provide data for all four of their long-running journals, for all the years for which the data exist (2003-2010). Several other journal owners also have agreed to try to provide data, or else have provided qualitative or semi-quantitative summaries of the data. We've only had one hesitant response so far (I won't say from who). We're still waiting to hear from most of the journals we've contacted; it's very early days. Once we have as much data as we...
More»
Philip Davis over at The Scholarly Kitchen helpfully pointed out some surveys that bring data to bear on the basic premise which motivates PubCreds: that the peer review system is indeed breaking down (or at least is likely to do so) because of too many papers chasing too few willing referees. I want to discuss these studies in some detail, in part because at least some people see these studies as clear-cut, unambiguous evidence that there are no problems in the current peer review system and that no problems are likely to develop. I'll discuss in most detail the 2007 global survey of 3040 academics commissioned by the Publishing Research Consortium asked about attitudes towards, and experiences with, the peer review system (a 2009 follow up survey of 4000 academics found similar results in those areas that were examined in both surveys) The most relevant findings, with my comments:-64% satisfied with the current system, 12% dissatisfied, 22% neither satisfied nor...
More»
The Scholarly Kitchen's David Crotty has an excellent post on the "burden" of peer review. While scientists increasingly seem to find pre-publication peer review an overwhelming burden, it's not clear that this feeling is entirely justified. By many measures, pre-publication peer review is a terrific bargain for science as a whole, and for individual scientists. And the alternative idea of post-publication review (let's throw everything onto the web and let "the crowd" sort it out) doesn't hold up well under scrutiny. I agree wholeheartedly with David's points. But he doesn't actually suggest any ways to tweak the existing system so as to address the real (as opposed to merely perceived) burdens imposed by that system. Real burdens do actually arise when too many manuscripts are chasing too few willing referees.Hmm...if only there were a way to ensure that the availability of willing reviewers matched the number of manuscripts submitted...-Jeremy
More»
One point I took away from the ESA Publications Committee meeting was
that it would be useful to have data quantifying the scale and nature of
any problems in the peer review system. Since the tragedy of the
reviewer commons is defined in terms of the activities of individuals,
with some submitting disproportionately to how much they review, one
would ideally like to have data on how much reviewing and submitting
individuals do. But reliable individual-level data would be
time-consuming to obtain; you'd need to conduct a rigorous social
science-type survey.In contrast, journal-level data, obtained from online ms handling
systems, is much easier to obtain. So what ‘symptoms’ of a tragedy of the
reviewer commons might one expect to see in journal-level data? Here are some
possibilities (and yes, I know that some of these have alternative
explanations; I’ll get to that).1. Increasing submissions over time. We already know this is
happening because many...
More»
Along with the launch of the ESA's new online journal, PubCred was the talk of the hallways at the ESA Meeting (don't just take my word for it--that's what several members of the Publication Committee told me!) There was a lot of feedback to digest, and I'll post a summary soon.UPDATE: I won't speak specifically about what was said at the ESA Publications Committee meeting, except to thank Scott Collins, the committee chair, for inviting me to join the discussion. For me, the big take-home from my many intereractions at the ESA meeting was the need to collect (appropriately anonymized) data, in order to better-quantify the problem (or even prove beyond reasonable doubt that one exists!), and fine-tune the solution. Journals already have a fair bit of relevant data, but they actually don't have all the data one would ideally want because no one journal has a complete picture of what any one individual does. For instance, if I don't review at all for journal A, maybe that's because I...
More»
A brief update, not on PubCreds, but on the complementary idea of sharing reviews among journals so as to reduce the burden on the peer review system. It turns out that a consortium of neuroscience journals already runs a formal system of sharing of reviews. Details are here, but the key features are:-Sharing of previous reviews is voluntary on the part of the author. If the authors don't want previous reviews shared, they submit the ms to a new journal in the usual way, so that the new journal will not even be aware that the ms was previously rejected by another journal.-Sharing of reviews is all-or-nothing. You can either choose to share every review the ms has ever received, or none. You can't cherry-pick the positive ones.-Even if provided with previous reviews, editors have the option of requesting new reviews.It seems to me that's the only reason not to adopt this would be that the benefit gained (in terms of reduced burden on the peer review system) isn't worth the cost (it...
More»
Nothing new under the sun, apparently. I've just been informed by an economist colleague of a colleague that there is an economics journal which already operates on a version of the PubCred system. The main difference is that authors have the option of paying for submissions with real money instead of PubCreds (a hybrid which Dan Bolnick also suggested in the comments accompanying his petition signature). Please do click through and read the journal's approach--it's very intriguing. I need to contact them to find out how well it works, how well the journal is doing, if they used to do things differently, etc.-Jeremy
More»
Over at Jabberwocky Ecology, Ethan White has a thoughtful and provocative post about PubCreds. Ethan invited Owen and I to respond, which we'll do here (may also be cross-posted at Jabberwocky Ecology at some point). Owen and I have chatted about our response, and I’ve taken the
lead on actually writing it.
In his post, Ethan raised two major concerns about PubCreds, who makes
the rules, and how to uniquely identify authors and reviewers. These
concerns motivated Ethan to suggest what he argues is a simpler and more
flexible system, based on real money rather than PubCreds. I’ll first
address both of Ethan’s concerns, thereby alleviating the felt need to
consider a real money system. I’ll then raise some concerns about an
alternative system based on real money. At the end I’ll address some
minor points Ethan raised in passing.
The issue of who makes the rules, while undoubtedly challenging, doesn’t
need to be as challenging as Ethan suggests. I...
More»