Peer review and quality assurance

Topic lead(s): Clarissa F. D. Carneiro, Elle Malcolmson, Alex Mendonça
Last updated: 12 August 2024
Page: https://www.oajournals-toolkit.org/running-a-journal/peer-review-and-quality-assurance

Peer review has been applied by journals for centuries, but formats vary widely. It is considered the best system available for assessing the quality of manuscripts submitted for publication, although it has limitations. A key challenge for journals is to identify available peer-reviewers, including because the number of articles published globally is increasing at a fast pace.

Today, peer review is considered an indispensable requirement for formal publication. As a result, journals need to establish appropriate peer review policies as well as approaches to recruit peer reviewers.

Peer review aims to assess the suitability of a submission for publication. It is delivered by experts in the field of the submission, who must be identified by editorial staff. Different journals and editors have different expectations around the role of reviewers, so clear guidance is essential. Notably, authors tend to prefer rapid peer review, as this means that their work is published quickly. However, time pressure on peer reviewers may compromise the quality of their input. As a result, journals have a responsibility to keep speed and quality of peer review in balance, to ensure that published articles are of high standard.

Whether a submission is deemed to be suitable for publication depends on the journal’s article selection criteria  as well as on the quality and integrity of the research being described. For example, peer reviewers may be asked to focus on methodological rigour, novelty, engagement with open scholarship practices or any other criteria set by editorial staff.

Approaches to peer review

Peer review can be delivered in a broad range of ways, all of which have their respective merits. It is the editorial board’s responsibility to choose a suitable approach, considering the following dimensions:

Dimension Key considerations
Number of reviewers
  • Multiple reviewers are preferable (at least two per submission are highly recommended), as consensus between them is typically limited
  • Recruiting multiple reviewers may be difficult
Interaction between reviewers
  • Consensus from reviewers can more easily help authors improve their submission
  • Collaborative comments by reviewers can help editors make decisions
Identification of authors and reviewers
  • Multiple variations are possible, including single- (names of reviewers hidden from the author; name of the author shared with reviewers), double- (no names shared with either authors or reviewers) and triple-anonymous peer review (no names shared with either authors or reviewers, plus the editor remains anonymous, too)
  • Identification may lead to biased comments, which can be mitigated via levels of anonymity in peer review
  • Identification may be partly inevitable if a preprint version has been posted (i.e. authors can be identified by default)
Publication of peer review reports (also known as transparent or open peer review)
  • Improves transparency and allows research on peer review itself, with the goal of continuously improving the system
  • May discourage some authors and reviewers from submitting

The use of automated review tools is also possible, although journals should note that these are only able to detect basic issues (e.g. plagiarism, data sharing statements, reagent identifiers). By using artificial intelligence, these tools can help reduce the editors’ and reviewers’ workload, although ongoing research has highlighted potential concerns around computational bias. As a result, automated review tools should not be used to automatically make decisions on submissions received, but only to inform the peer review process.

How to choose peer reviewers?

Identifying reviewers is a complex challenge, as researchers are typically very busy and the number of articles that need reviewing annually grows more quickly than the pool of available reviewers. The following strategies may help in identifying peer reviewers:

  • Asking authors to provide suggestions
  • Checking the references in the submission or automated tools to identify researchers in similar areas
  • Using personal networks, including from editors, the editorial boards and previous authors and guest editors
  • Inviting previous peer-reviewers
  • Asking declining peer-reviewers to provide suggestions for alternative candidates

Today, there are imbalances in peer review, which contribute to difficulties in finding reviewers and to the overburdening of a small sample of reviewers (Kovanis et al., 2016). In addition, diversifying reviewers in terms of gender, region or other personal characteristics can have a positive impact (Murray et al., 2019).

Limitations of peer review

Currently, there is no reliable evidence that peer review significantly contributes to the overall quality of scientific literature. This lack of evidence, however, does not indicate that peer review is harmful or should be avoided, but just points to the need for more research about established and innovative peer review systems.

Important concerns about peer review include the low agreement between reviewers, the fact that they are subjected to biases and that peer review can be very time-consuming.

Finally, peer reviewers can only address the quality of a manuscript – not of the underlying research. Other methods of quality assurance, such as assessing reproducibility or replicability, are being considered by some journals alongside peer review (e.g. American Economic Association), although this remains a limited practice. The requirement for a data availability statement and data sharing as part of editorial policies is a useful way to enable activities such as reproducibility or replicability checks.

Manipulation of the publication process

The Committee on Publication ethics defines ‘paper mills’ as individuals, groups or organisations that aim to manipulate the publication process for financial gain. These actors pursue the fraudulent submission, peer review and publication of articles that are, in most cases, incorrect and not arising from genuine research endeavours. Identifying paper mill activity is complex, as these articles are designed to deceive all stakeholders involved in the publication process as well as readers. Often, fraudulent articles are only spotted after they are published, as concerns across several articles may start appearing as a coherent pattern. In these cases, editors should collect information, documentation and data from authors to inform next steps.

Dealing with paper mills creates a significant administrative burden. Journals are therefore advised to have clear guidance and processes in place, as well as to operate transparently and share information with other publishers (and, potentially, the author’s institution) as appropriate. Any retraction notices applied to articles identified as arising from a paper mill should be transparent and clear, too. Throughout the process of investigating paper mill activity, journals and editors should respect confidentiality, as there is a high risk of unwillingly damaging an author’s reputation even when claims or concerns may eventually be resolved or unfunded.

Share this article

Download this article