To identify or not to identify: a choice in chemistry education research and practice

Chemistry Education Research and Practice is pleased to announce that authors will have the option to select whether their manuscript will undergo a single-anonymous or double-anonymous peer review process starting concurrent with this editorial. Single-anonymous has been the process to date, where the identities of the referees are not disclosed (is masked) to the author; however, the identity of the author is made known to the referees. In double-anonymous, the identities of the referees are masked to the author, and the identities of the authors are masked to the referees. Research literature from other disciplines found that single-anonymous review has been associated with higher referee ratings of manuscripts but also the potential for referee bias, while double-anonymous reviews can mitigate biases, such as the halo effect, but can also lead to lower referee ratings overall. It is unclear if these findings extend into chemistry education research. Allowing authors the choice of a single- or double-anonymous review process will also allow the Journal to learn about the relative benefits of each approach when applied within the chemistry education community.

A search through the typical publishers in science education research revealed that among the high-impact journals, such as Science Education, Journal of Research in Science Teaching (Wiley), the International Journal of Science Education and Studies in Science Education (Taylor & Francis), as well as the International Journal of STEM Education and Research in Science Education (Springer), a double-anonymous review process is prevalent. Meanwhile, journals with a specific focus on chemistry education research, such as the Journal of Chemical Education (ACS) or Chemistry Teacher International (DeGruyter), use single-anonymous processes for peer review. None of these science education or chemistry education research journals provide authors with a choice on the peer review process.

Findings from single- versus double-anonymous review studies

The advantages and disadvantages of the single- or double-anonymous peer review process have been discussed for decades. As the quality and importance of scholarly research largely relies on the peer review process, it is not uncommon for researchers to hold perceptions of systematic biases that influence outcomes. It has long been suggested that an author's status plays a role in the outcome (Merton, 1968), where more established researchers receive more credit for their contributions than their lesser-known colleagues in the field. Additionally, gender bias has been noted (Knobloch-Westerwick et al., 2013), where submissions from male first authors are rated more highly than those from female first authors, particularly in male-dominated fields. It has been suggested that the double-anonymous review process could ameliorate the impact of these biases. However, the success of double-anonymous reviews is questioned, as often referees might be able to identify authors through self-citation, writing style, and research subject matter. This can undermine the idea of a double-anonymous review to achieve an increase in diversity among accepted publications and reduce bias towards certain authors or institutions.

Studies have been conducted to elucidate the effects of the single- versus double-anonymous review process. Investigations have included the examination of differences in referee ratings and acceptance rates overall, as well as regarding author gender or status biases. In an investigation of submissions to The American Economic Review, using a randomized trial design, Blank (1991) found that under double-anonymous conditions, manuscripts had lower overall acceptance rates and received more critical reviews than those in the single-anonymous condition. However, no effects by author gender were detected in the study. A more recent randomized trial investigation of submissions to the journal Functional Ecology (Fox et al., 2002) examined impacts on manuscript reviews based on author gender and both the economic development status and English language proficiency of the country of submission. Similar to Blank's study (1991), no bias by author gender was detected and under double-anonymous conditions, manuscripts on average received lower referee ratings and were less likely to be invited for revision or resubmission. Additionally, the single-anonymous review conditions favored submissions from countries with higher economic development status and English language proficiency. A study conducted by Tomkins et al. (2017), on peer reviewed conference proceedings in the field of computer science, used a research design where a total of four referees were randomly assigned to each manuscript – two referees received the manuscript with the author identified (single-anonymous) and two referees received the manuscript with the author identity masked (double-anonymous). While no bias by author gender was detected, under single-anonymous conditions the referees were more likely to submit positive reviews for manuscripts from famous authors and from top universities or companies. This has been referred to as a “halo effect”, where work is rated more favorably based on the status of the author or the author's background. Similarly, outcomes favoring more prominent authors were found in a study from the field of finance. In a study by Huber et al. (2022), a single manuscript that was co-authored by a prominent author (a Nobel laureate in the field) and a relatively unknown early-career author was sent for review under one of three conditions: double-anonymous, single-anonymous naming only the prominent author, or single-anonymous naming only the relatively unknown author. Comparisons revealed that it was more likely that the manuscript was rejected when only the relatively unknown author was named than under the double-anonymous condition or when only the prominent author was named. While these studies looked at the effects of single- and double-anonymous reviews in business, computer science, or ecology, these biases might apply to science education journals as well.

The literature suggests that double-anonymous reviews can mitigate biases, such as the halo effect, but can also lead to lower referee ratings overall. Single-anonymous review has been associated with higher ratings, but also the potential for referee bias. It is of note that the literature on studies that examined the relative effects of each approach is sparse and does not include studies conducted within educational research fields. Thus, the impact of either approach within chemistry education can be considered reasonably unknown at this point. This uncertainty informed the decision to provide the option for authors to elect either approach for their manuscripts.

Submitting for double-anonymous or single-anonymous review

Peer review is meant to ensure a fair assessment of the quality of research and the relevance of the work to a journal's audience. However, factors such as a manuscript author's gender, status in a field, and country of origin have been known to affect referee behavior (Wenneras and Wold, 1997; Link, 1998). If an author submitting to CERP feels that the perceived merit of their research may be affected by factors other than scientific quality (Lortie, et al., 2007), such as those mentioned in this editorial, then they may choose double-anonymous review.

When submitting to CERP for double-anonymous review, the submission system will generate a title page without the authors’ names or institutions included. Authors will need to ensure that their manuscript is anonymized. The following are offered as guidance for anonymizing a manuscript.

(a) Ensure author names are not listed at the beginning of the manuscript.

(b) Replace all self-citations within the body of the manuscript, including when you are not the first author, with (Author, Year).

(c) Replace bibliographic references of any self-cited work, with the phrase “Author, (Year)”.

(d) Omit the acknowledgement section that identifies contributors and/or funders. The acknowledgement and funding information can be added after the peer review process and before the manuscript is published.

(e) If one is publishing a series of studies on a single topic, care should be taken to not give away the identity of an author.

(f) Pre-publication of a study in sites such as ChemRxiv may make it possible for reviewers to identify authorship. Consider not pre-publishing when pursuing a double-anonymous review.

Within CERP, authors may still elect a single-anonymous review process, as they may be building on past work and want referees to evaluate the current work within this context. For example, authors who are evaluating a particular pedagogy may want referees to be aware of the their relative experience with this pedagogy based on their past published work. Additionally, research literature from other disciplines found that manuscripts reviewed under single-anonymous conditions were rated higher than under double-anonymous conditions, although it is unclear if that finding extends into chemistry education research.

The process for submitting to CERP with single-anonymous review is unchanged. Reference information throughout the manuscript should be completed following the Journal format. If the authors elect, an acknowledgement of the authors’ relative contributions can be included in the manuscript.

Future directions

The decision to switch to author choice on the review process was made with acknowledgement of the uncertainty surrounding the advantages of each approach. The enactment of author choice offers an opportunity to generate knowledge that can inform future processes. Over time, we hope to learn and share what we learn on the relative impact of each review process type. In particular, submission rates for each type of review process can detail the community's interest in each approach. Additionally, manuscript outcomes such as acceptance rates or number of times revisions were requested can be compared between the two approaches. Chemistry Education Research and Practice serves a diverse community (Graulich et al., 2023) and any metrics can also be checked for variations among authors’ disclosed sex, country or origin, or status as a first-time contributor to the journal. Ultimately, the information learned may support a decision to adopt one of the review process types or maintain authors’ choice.

References

1. Blank R. M., (1991), The effects of double-blind versus single-blind reviewing: Experimental evidence from The American Economic Review, Am. Econ. Rev., 81, 1041–1067.
2. Fox C. W., Meyer J. and Aimé E., (2002), Double-blind peer review affects reviewer ratings and editor decisions at an ecology journal, Funct. Ecol., 37, 1144–1157.
3. Graulich N., Kahveci A., Lawrie G. A., Lewis S. E. and Nyachwaya J., (2023), Laying the foundations to build on: exploring diversity in our Chemistry Education Research & Practice community, Chem. Educ. Res. Pract., 24, 8–11.
4. Huber J., Inoua S., Kerschbamer R., König-Kersting C., Palan S. and Smith V. L., (2022), Nobel and novice: Author prominence affects peer review, Proc. Natl. Acad. Sci. U. S. A., 119, 1–7.
5. Knobloch-Westerwick S., Glynn C. J. and Huge M., (2013), The Matilda effect in science communication, Sci. Commun., 35, 603–625.
6. Link A. M., (1998), US and non-US submissions – an analysis of reviewer bias, J. Am. Med. Assoc., 280, 246–247.
7. Lortie C. J., Aarssen L. W., Budden A. E., Koricheva J. K., Leimu R. and Tregenza T., (2007), Publication bias and merit in ecology, Oikos, 116, 1247–1253.
8. Merton R. K., (1968), The Matthew effect in science: The reward and communication systems of science are considered, Science, 159, 56–63.
9. Tomkins A., Zhang M. and Heavlin W. D., (2017), Reviewer bias in single-versus double-blind peer review, Proc. Natl. Acad. Sci. U. S. A., 114, 12708–12713.
10. Wenneras C. and Wold A. (1997), Nepotism and sexism in peer-review, Nature, 387, 341–343.

---