Single vs double-blind peer review: an experiment

From the 5^th of September 2019, Functional Ecology will be running an experiment looking at the effects of single-blind vs double-blind review In this post, Chuck Fox, Executive Editor for Functional Ecology, explains why.

by Chuck Fox, Executive Editor of Functional Ecology

There is a widespread perception among academic researchers that systemic biases influence the outcomes of scholarly peer review. In particular, there are concerns that editorial decisions may be influenced by author gender, nationality, location and/or reputation, rather than just the quality of a manuscript. We do not know whether, and to what extent, the outcomes of peer review of papers to Functional Ecology are subject to the unconscious biases of reviewers. However, the perception that such biases are present undermines confidence in the objectivity and fairness of peer review.

To reduce these potential biases, some scholarly journals have adopted new models of peer review. The traditional review model is single-blind – author identities are known to reviewers but reviewers are anonymous to authors. Double-blind peer review extends anonymity to authors – both reviewers and authors are anonymous to each other. Some journals review exclusively single-blind, some review exclusively double-blind (mandatory double-blind peer review), and some allow authors to choose if their identity is disclosed to reviewers (optional double-blind peer review). Surveys show that researchers generally prefer double-blind review over single-blind review (Ware 2008; Mulligen at el. 2013; Taylor & Francis Group 2016), with this preference especially strong for early career researchers (Moylan et al. 2014; Rodriguez-Bravo et al. 2016). However, the effectiveness of double-blind peer review in eliminating reviewer biases is uncertain. Blinding may also be ineffective at masking author identities, especially in fields such as ecology where researchers are often associated with specific study organisms/locations and/or questions.

At Functional Ecology, we believe that decisions regarding the peer review model we use should be based on evidence. We have therefore decided to run a controlled trial of double-blind peer review that will allow us to quantify the consequences (both costs and benefits) of double-blind versus single-blind peer review. During this trial, half of all research submissions, chosen randomly, will be reviewed under our current single-blind peer review model and half will be reviewed double-blind. (The trial will not include invited and other non-standard papers).

The goal of this trial is to understand how a change in peer review model will affect the quality, timeliness and fairness of the peer review process, and to quantify the costs experienced by the journal. In particular, this study will ask (among other questions):

(a) Whether reviews are easier/harder to obtain when author identities are masked,

(b) Whether the quality, constructiveness and/or criticalness of peer reviews changes when reviewers are unaware of the identity of the authors,

(c) Whether these variables are influenced by characteristics of the authors, such as their gender or geographic location, when author identities are known to reviewers but not when authors are anonymous, and

(d) Whether blinding is effective at masking author identities.

The journal will also monitor workload of editorial staff (e.g., the time necessary to ensure manuscripts are properly anonymized) to predict the financial costs of double-blind review.

What this trial means for authors

The process of submitting your manuscript to the journal will change very little. Beginning 5^th September 2019, and continuing for the next two years, we will require that allpapers submitted to Functional Ecology be submitted as if they will be reviewed via a double-blind process. We will require that all submissions to the journal include a title page that is detached from the main body of the manuscript, which will include:

Manuscript title
Author details (names and addresses of all authors, and the identity of the corresponding author)
Author contribution statements, and
Acknowledgements.

We will also require that no author identifying information be present in the manuscript file; this includes avoiding unintentional self-identification, such as by using the third person when referencing prior research (e.g., say “previous research has shown” rather than “in my previous research …”). Details of these requirements are available in the author guidelines.

As papers are submitted, they will be assigned to one of the two peer review treatments by ScholarOne Manuscripts. Neither the authors or editors will control this assignment – it will be random with respect to all features of the manuscript and authors. By submitting to the journal authors acknowledge and accept that the peer review process applied to their paper will be determined by this random assignment, and submission of your paper indicates consent to be part of the trial. Authors that wish to opt-out of the peer review trial are invited to submit their paper to one of our sister journals.

The peer review process undergone by papers in both treatments will be identical in all regards except that authors will be identified to reviewers in the single-blind treatment but will be anonymous to reviewers in the double-blind treatment.

What this trial means for reviewers

We expect the quality and diversity of papers received by the journal, and thus sent for peer review, to remain as high and exciting as ever. The only effect of this peer review trial that will be evident to reviewers is that they will be asked (for half of submissions) to review papers for which the author identities are blinded to them. After peer review (and after the final decision on a manuscript has been made by the editors!) we will invite reviewers to complete a brief survey in which we ask them how well the paper was blinded, and to identify the authors of the manuscript if they can. Reviewer responses can then be compared to actual authorship lists to quantify the frequency at which authors successfully identify reviewers.

Data driven decisions and the future of scholarly publishing

The editors of Functional Ecology are excited to have the support of the British Ecological Society as we attempt to tackle one of the key issues in scientific publishing – understanding the biases that undermine the fairness of the peer review. This project will be a large and comprehensive analysis of biases in the peer review process, results from which will facilitate data-driven decisions about peer review models by this journal, it’s owner society, and the publishing community more broadly. We thank all of our authors and reviewers for your support during this project, and encourage you to be part of this important effort by submitting your papers to Functional Ecology.

References

Moylan, E. C., Harold, S., O’Neill, C., & Kowalczuk, M. K. (2014). Open, single-blind, double-blind: which peer review process do you prefer? BMC Pharmacology and Toxicology 15: 55.

Mulligan, A., Hall, L., & Raphael, E. (2013). Peer review in a changing world: An international study measuring the attitudes of researchers. Journal of the American Society for Information Science and Technology, 64(1), 132-161.

Rodríguez‐Bravo, B., Nicholas, D., Herman, E., Boukacem‐Zeghmouri, C., Watkinson, A., Xu, J., … & Świgoń, M. (2017). Peer review: The experience and views of early career researchers. Learned Publishing, 30(4), 269-277.

Taylor & Francis Group (2016) Peer review: A global view: Author Services Motivations, training and support in peer review.

Ware, M. (2008). Peer review in scholarly journals: Perspective of the scholarly community–Results from an international study. Information Services & Use 28: 109-112.