Pubcreds: Fixing the Peer Review Process by “Privatizing” the Reviewer Commons

Responses to concerns Aug 2. 2010 | Comments (5)

We've received a lot of feedback on the idea of PubCreds. Below we address most of the concerns we've heard so far. For discussion of the single most-commonly raised concern--what to do about students and others who aren't asked to review much--go here.

1) If only some journals participate, authors may try to avoid the system by submitting to non-participating journals. Indeed, leading journals have an incentive not to require the use of credits, if they want to keep the best papers coming to them in order to keep their impact factors up. Big name journals will probably have little trouble getting qualified reviewers even if they don't participate.

Ideally, all journals within a relatively well-defined field (like "ecology") will participate voluntarily, because doing so is in their individual interests, and because doing so incurs little or no associated cost. Journals that wish to attract mss from authors who want to avoid the PubCred system, and that are currently able to easily attract quality reviews, may choose not to participate initially. But their ability to attract quality reviews may decline as more journals join the system, giving them an incentive to join as well.

2) Quality of reviews will decrease as people perform them quickly just to earn credits.
and
3) Some reviews are more useful than others, and the system should take this into account.

We favor a useful review earning 1 PubCred, and one that is not useful earning none. All the handling editors with whom we have discussed the matter would welcome the chance to refuse payment for poor-quality reviews. And while some reviews are more useful than others, paying more for very high-quality reviews would require paying less for reviews that are merely good or adequate quality, since otherwise the system would be flooded with PubCreds (for more on this point, go here). We doubt reviewers would welcome a system in which they might not be paid for a good-quality review because another reviewer provided an even better-quality review! To ensure fairness and uniformity in payments for reviews, there will need to be clear guidelines about what makes a review useful or not useful. There are many guidelines out there already. Standardizing and specifying the expectations of a peer review would be a valuable contribution in itself.

4) The market of review credits could interact with the real economy if, for instance, scientist X pays money to graduate students or developing-world scientists to perform reviews which are then submitted in the name of scientist X.

It seems unlikely that someone would go to these lengths, but there are enough people, and in some disciplines large enough rewards, for anything to happen. Note that the scenario described would involved the already prohibited practices of passing on confidential information (submitted mss) without appropriate permission, and signing one's name to something one did not write.

5) PubCreds is only a minor reform. A much more radical and complete overhaul of how science is communicated is at least desirable and perhaps necessary. For example, ‘crowdsourcing’, the PLoS ONE solution, the ArXiV solution, and/or something like a Slashdot rating/karma system.

NCEAS and the ESA Theory Section tried to implement an ArXiV-type preprint server for theoretical ecology back in the 1990s; it was little-used. Early experiments at Nature and PLoS ONE allowing online comments on papers haven't been promising. Hardly anyone comments, and hardly any papers attract any comments. Indeed, it's been proposed that, for such systems to work, authors must also be required to comment--i.e. a "CommentCreds" system precisely analogous to PubCreds!  We would be interested to see a detailed, workable vision of an alternative scientific publication system. To be compelling, such a vision would need to explain how to transition from the current system, given the incentives and constraints faced by individual scientists (particularly the incentives and constraints imposed by granting agencies and employers, which have their own ways of evaluating their applicants and employees and won't easily be convinced to change them). We are confident that implementation of PubCreds would not interfere with--and indeed, might be an essential part of--a more radical reform of scientific publishing in the future.

6) Economists have studied the kind of artificial market proposed. What do these studies suggest about the PubCred Banking system? Other somewhat similar schemes have had successes and failures (e.g., Capitol Hill Baby-sitting Co-op).

We are going to carry out some market modeling, in collaboration with Stefano Allesina, in order to figure out how stable a PubCred system would be, calibrate the charge and payment amounts, tweak other operational details, and hopefully identify unexpected problems. Elsewhere we discuss the lessons of the Capitol Hill Baby-sitting Co-op for the PubCred system.

7) The PubCred system does not address the wastage in the current system, i.e., rejection, reformatting, revision. PLoS, for example, reduces this wastage.

It does not address wastage directly. However, it will encourage authors to think harder before submitting an ms to a journal, which could lead to greater chance of acceptance.

8) The current system is not broken, and will not break.

Many people, including the EiCs of numerous leading journals, have published editorials over the last several years noting or predicting the breakdown of the current system. Why would they do this if there was no problem? Note that this evidence of breakdown is not purely anecdotal; all EiCs have extensive data on the submission and review process at their journals. Further, even those who feel the current system is not broken agree that individuals face strong incentives to submit papers, and weak incentives to review. In light of those incentives, it is hard to imagine that the current system will not break down at some point. Having said all that, the period of data collection we suggest prior to the PubCred system going 'live' would go a long way towards quantifying precisely how broken the system is and in what ways.   

8) What about including submission and reviewing of grant proposals?

It does seem possible, as we mentioned in our article, that the system we proposed could be used by funding agencies. In order to explore this further, we have approached the NERC (UK). At least one incoming NSF Program Officer has informally expressed interest in the system, in part because NSF apparently is experiencing some difficulty obtaining reviews of proposals. We have not yet considered whether PubCreds should be transferable between the journal system and the granting body system, or whether granting bodies (either individually, or as a group) should have a separate "GrantCreds" system. We would welcome comments on this point.

9) The details that were not covered in the paper, and the details that were, are probably going to make the system very difficult or impossible to implement.

This is rather a fatalistic point of view, with little evidence in support. Our view, and that of our many supporters, is that the potential importance of our core proposal merits some considerable effort to work out the details.

10) Heterogeneity of reviewing jobs, particularly differences in time and experience required, make inappropriate one level of payment for all reviews.

Right now, it seems clear that the tragedy of the reviewer commons arises primarily from inequities in the numbers of (conscientious) reviews different people do, relative to the number of papers they submit. In comparison, inequities arising from the fact that some papers are 'easier' to review than others certainly are rare and small in magnitude, since there seems to be little reason to think that certain reviewers are only asked to review 'difficult' papers while others are only asked to review 'easy' papers. Further, we believe the community would be in a much better position to try to account for the 'difficulty' of a review after addressing much larger inequities in the number of reviews people do relative to how much they submit. There is no reason to think that our proposal would make the issue of heterogeneity of reviewing tasks more difficult to address, or that failure to address this issue would in any way interfere with the implementation of our proposal.

11) You suggested that handling editors be credited with 0.5 PubCreds per manuscript handled, since their job is usually somewhat easier than that of a reviewer, at least when reviews are of good quality. But the job of a handling editor is actually harder, so they should be paid more. And how much should Editors-in-Chief be paid?

We have no strong views on how much handling editors should be paid, relative to reviewers. Indeed, we've also heard arguments for paying handling editors less than reviewers! The key thing is that the price of submission be equal to how much it costs to review and handle it. We have not yet considered how much EiCs should be paid, and would welcome suggestions. Clearly the job of EiC is an onerous one. Perhaps EiCs should be paid a small amount for every submission to their journals? Or perhaps EiCs should be allowed some number of free submissions per year?

12) There will be a lot of sensitive information on one central database.

Exactly what information is needed is not yet specified, but probably need not allow tracing of reviewers to manuscripts. Trust through security and minimization of stored data should be a goal.

13) Reviews should not carry over from one journal to another.

Carrying over of reviews is not a part of the PubCred Banking system, but could be integrated with it. There seem to be both benefits and costs to carrying over of reviews. These should be discussed independently of our proposal.

15) There are informal feedbacks operating already, such as a regular reviewer being more likely to have their ms sent out for external review.

It seems that these informal feedbacks are insufficient to encourage reviewing, otherwise there would not be the perception of a system in trouble. If informal feedbacks are good, then that's a reason to formalize and strengthen them.

16) The system might ‘discriminate’ against those with a less good grasp of the English language.

By making reviews much easier to obtain, the system will encourage journals to send papers by non-native English speakers out for review, which can only help those authors. Regarding reviewers whose native language isn't English, we imagine that as reviews become more important to conduct, there will be an increase in the amount of training available for all researchers, teaching them how to prepare constructive and useful reviews.

17) Smaller journals would lose out. They don't have the editorial staffs and online submission systems to participate, but will lose reviewers and authors if they don't.

Any implementation of the system would be designed to require as little effort as possible from any journals. Journals that wish to participate, but that do not use currently use online submission and reviewing systems, might need to adopt those systems in order to send the required information to the PubCred Bank in an efficient manner. Journals not in the system would not necessarily lose out, however. So long as they have a loyal reviewer, author, and reader base, they will carry on as usual.

18) Don't you need a way to uniquely identify authors and reviewers for your proposal to be feasible?

No. Like the real money banking system, the PubCred Bank would only need to uniquely identify accounts, not individuals. For instance, the system might work as follows:

a. Authors sign up for an account at the Pub Cred Bank, and receive a unique account number.

b. In order to submit an article, the submitting person must enter the Pub Cred Bank account number of all authors. At the same time, the submitting person indicates which authors will bear the cost of submission, including who will bear the cost of handling, and the reviews.

c. This information is passed to the Pub Cred Bank, which then emails all authors, and also requires that authors whose account will be debited click on a link to authorize this debit. One the PubCred Bank has sufficient authorization for debits, a communication is sent to the journal submission web site and the ms submission is completed.

d. If the ms is rejected without review, PubCreds are debited for handling. If it is reviewed, PubCreds are debited according to the distribution of debits already authorized.

e. Reviewers will also need a Pub Cred account. Their unique account number will be submitted with reviews. If the handling editor agrees that the review is useful, they authorize credit to the reviewers account.

f. It’s not even a problem if individuals have multiple accounts at the PubCred Bank. After all, lots of people have multiple real-money accounts (checking accounts, savings accounts, credit cards…) which they use to take in and pay out funds. Authors with multiple accounts would simply need to specify which of their accounts they wish to use to pay for a particular submission, and reviewers with multiple accounts would simply need to specify which of their accounts they wanted their PubCreds deposited into. 

19) Your proposal fails to address problem X, where X is some problem with the peer review system other than the 'tragedy of the reviewer commons'.

By design, our proposal is a focused attempt to solve one important, specific problem. This focus makes the proposal more practical, easier to implement, and less likely to have perverse side effects than a less-focused proposal. There may be other problems worth addressing, but we believe that attempting to address them at this time is impractical (it's going to be hard enough just to build agreement around PubCreds!) Failure to address other problems in the peer review system is not a failing or drawback of our proposal, unless it can be shown that our proposal would make those other problems substantially worse.

20) You should not have to review in order to submit. Those who do good science should be allowed to submit it, period. Nothing should prevent the publication of worthy science, because anything that does slows scientific progress.

This is a selfish, parasitic point of view. If everyone adopted it, no science at all would be peer reviewed, and scientific progress as we know it would cease. We want to see science as a whole progress at the maximum sustainable rate. We believe that is the rate at which science would advance under the PubCred system. We believe the current system is not sustainable. The current system gives authors a strong incentive to write short, shallow papers, and to quickly resubmit rejected papers without revising. These behaviors merely give the appearance of rapidly advancing science. The current system also leads to rejection of many papers without review, which increases the stochasticity of decision-making and actually slows the publication of some good papers while hastening the publication of some poor (or less-good) papers.  Under the PubCred system authors will have an incentive to publish fewer but more substantive papers (unless they want to do an enormous amount of reviewing), and those papers will be sent out for thoughtful reviews. That enhances rather than inhibits the progress of science.