Pubcreds: Fixing the Peer Review Process by “Privatizing” the Reviewer Commons

Response to Jabberwocky Ecology: why we shouldn't use real money instead of PubCreds Aug 3. 2010 | Comments (1)

Over at Jabberwocky Ecology, Ethan White has a thoughtful and provocative post about PubCreds. Ethan invited Owen and I to respond, which we'll do here (may also be cross-posted at Jabberwocky Ecology at some point). Owen and I have chatted about our response, and I’ve taken the lead on actually writing it.

In his post, Ethan raised two major concerns about PubCreds, who makes the rules, and how to uniquely identify authors and reviewers. These concerns motivated Ethan to suggest what he argues is a simpler and more flexible system, based on real money rather than PubCreds. I’ll first address both of Ethan’s concerns, thereby alleviating the felt need to consider a real money system. I’ll then raise some concerns about an alternative system based on real money. At the end I’ll address some minor points Ethan raised in passing.

The issue of who makes the rules, while undoubtedly challenging, doesn’t need to be as challenging as Ethan suggests. I certainly don’t think we need to elect representatives who will meet in some kind of Pub Cred Constitutional Convention. The decision makers are the journal owners; they’re the ones who already control all other aspects of the operation of their journals, including editorial policy and pricing. For the ESA’s journals, the starting point for this kind of discussion is the Publications Committee, which makes recommendations to the Governing Board. (Indeed, the ESA Publications Committee will be discussing PubCreds this week, at the suggestion of Ecological Monographs EiC Aaron Ellison). Similarly, the BES has a Publications Committee, as do most other scientific societies that sponsor journals. So while there’s undoubtedly a collective action problem here, it’s not the problem of getting thousands of individual scientists to act collectively, it’s a problem of getting a much smaller number of journal owners to act collectively. I would make the analogy to the Dryad data-sharing and archiving system that a number of scientific societies have just agreed to support—we didn’t need to have a “constitutional convention” to make that happen. If and when one or more leading journal owners decide that they want to seriously pursue PubCreds, I imagine that they would charge their existing Publications Committee, or else a new ad hoc committee, to liaise with other journal owners and build up a consensus among them as to how to proceed. That’s basically what happened with Dryad, as I understand it. This seems to me both a practical and appropriate way to proceed.

The technical issue of uniquely identifying authors and reviewers is one that’s been raised to Owen and I a few times. It never even occurred to us that this might be an issue, which is why it’s not mentioned in our paper. Indeed, it was such a non-issue for me that, when it was first raised, I continued to feel it was a non-issue but had trouble articulating why! Then the other day Owen put his finger on it: it’s bank accounts that need unique identifiers, not people. This is true in the real-money banking system, and it’s true of the PubCred Bank as well. Please see our previous post for details on how this would work.

I hope I’ve addressed Ethan’s major concerns about PubCreds. So now I get to turn the tables and raise some concerns about his proposed “real money” alternative. ;-)   In this alternative system, authors pay a fee to submit, which is used to pay referees, and journals might be free to compete by setting their own prices for reviews and submissions. This idea has been proposed by various people in various versions. I first heard about it a few years ago when I was told that it had been proposed by Everett Fee at Limnology and Oceanography. I actually thought it was a great idea when I first heard it, and pitched it informally to the senior editors at Ecology Letters and Am Nat. The feedback I got was largely negative, and upon reflection both Owen and I agree that it’s not the way to go. The major concerns, in no particular order:

1. Ethan suggests that currency exchange issues are inconsequential; we respectfully disagree. If they are inconsequential, we assume Ethan would have no objection to using some currency other than USD as the standard in which submissions and reviews are priced? Owen, who is a Brit, says that if it’s all the same to Ethan, he’d prefer to price everything in pounds sterling. ;-) We also assume Ethan wouldn’t mind if the USD depreciates against the chosen PubCred currency, thereby increasing the real cost of his submissions? And that he wouldn’t mind paying transaction fees associated with currency conversions and transfers?

2. As Ethan notes, everyone starts off broke in PubCred land: the only way to earn PubCreds is to do reviews. This is an essential feature of the PubCred system. The same is emphatically NOT true of real money. Many authors have no research grants. While it’s true that these authors theoretically could pay for their submissions by doing a proportionate amount of reviewing, there’s no way to have a real-money overdraft system to cut some slack to authors who haven’t been asked to review in a while, or at all. And while forgoing page charges on one paper/year for authors who lack funds is one thing (many journals do), forgoing submission charges while still continuing to pay referees is a financial risk no journal would want to take, even for a limited number of submissions/author/year. Plus, here’s the flip side: if you have a grant, or are simply willing to spend some personal funds, you can buy your way out of having to review. Do we want a system where those of us who have grants and/or sufficiently large salaries can afford not to have to review, while those of us who don’t have grants are desperate to review because otherwise we can’t afford to submit anything?

3. One side effect of point #2 is that, contrary to Ethan's claim, it isn't true that all that matters is the ratio of submission costs to reviewer fees (i.e. that their absolute values don’t matter). For instance, someone who has a grant to pay part or all of their submission costs is going to need to be offered exorbitant amounts to be enticed to review. This isn’t just speculation on my part: I’ve had more than one person tell me that they’d need to be hundreds or even >$1000 USD to agree to do a review they otherwise wouldn’t do.

4. Many authors are going to be very reluctant to pay even a modest real money fee in order to submit a paper that could well be rejected, even if the paper is substantive and well-matched to the journal. That’s just the way it is, but that reluctance also has some rational foundation. Since even rejection without review costs the author a bit of money, but costs the journal little or nothing, journals will have a financial incentive to reject papers without review and pocket the portion of the submission fee that pays for this decision (and if you forbid journals from charging anything for rejection without review, you can’t afford to pay editors, and you’ve removed much of the incentive for authors to submit their papers to appropriate journals).

5. If real money is involved, publishers may see it as a revenue stream. And PubCreds doesn’t even involve real money! In a real money system, publishers will want their cut of submission fees. After all, they have to pay financial administration staff, there are overhead costs, they have shareholders to feed…This issue is perhaps the single biggest obstacle to implementing PubCreds in practice. Making it a real money system just makes this hurdle a hundred times taller.

UPDATE: Point 5 isn't well-put, and it isn't correct to attribute this point to a staff member at Wiley (I'd misremembered our conversation and mixed it up with a conversation with an academic colleague). It would be more accurate to say that publishers are likely to view investment in any PubCred-type system (whether based on real or notional money) as just that--an investment. In order to invest, publishers will need to be convinced that that investment will lead to a worthwhile return, relative to the many other investments publishers could make. That return need not come directly from the PubCred system itself. For instance, it might come about because participating in the system makes the publisher's journals better, and so more attractive to subscribers. Online ms handling systems are an example of an investment providing a return in this way. Similarly, point 6 is purely speculation on my part and probably reflects a somewhat naive and simplistic view of how publishers approach price-setting.

6. Is it really a good idea to let journals compete on prices for reviews and submission fees in a real money system, as Ethan suggests? Remember that everyone is not broke in “real money land”. The mind boggles at what publishers might try, and what effects it might have…

7. For the reasons outlined above, and just because real money is a really serious thing, I predict that it would be far harder to get a significant number of journals to agree to a real money system than to PubCreds.

That’s as much as Owen and I had to say on the big issues Ethan raised. In response to some of Ethan’s minor comments:

Ethan indicates his impression that journals are asking for multiple rounds of review, which would indeed be a bad thing for all the reasons Ethan lists. But as editors, authors, and reviewers ourselves, Owen and I haven’t noticed this. In my experience it’s a very rare practice, especially since journals compete to have short times-to-decision. This leads many mss to be rejected without review, or else rejected for trivial reasons after one round of review, forcing what’s really a revision to be resubmitted as a new ms.

As to whether journals should be given flexibility with regard to credits or payments under the PubCred system, I don’t think so (with one specific exception to be noted). If flexibility were allowed, the system would immediately break down because journals would start offering reviewers large amounts of PubCreds while charging authors few or no PubCreds (or even paying authors to submit!) If allowing journals to compete on “price” for submissions and reviews really is a good idea (and I don’t think it is), it needs to be with real money. The only pricing flexibility that could be possibly be allowed under the PubCred system would be if journals wanted to charge X PubCreds for a submission, in order to pay X-1 reviewers plus the editor (e.g., Science or Nature could charge 6 PubCreds in order to pay for 5 reviewers plus the editor). I’ve never before thought about allowing journals to charge X PubCreds in order to pay for X-1 reviews, rather than having everyone charge 3 PubCreds in order to pay for 2 reviews, but offhand I don’t see a huge obvious problem with it. This could open up an avenue for limited competition among journals. One could even imagine a journal charging as little as 1 Pub Cred for submissions, and allowing its editors to make all decisions unaided by external reviews. I’ll need to think more about how this limited flexibility would work in practice, and whether it would create any perverse incentives or unwanted side effects.

Thanks again to Ethan for taking the time to offer his thoughts, and for inviting Owen and I to respond. It’s always a pleasure when one’s ideas spark this kind of constructive feedback.