A randomized controlled trial presented at the 10th Peer Review Congress in September 2025 investigated whether anonymizing reviewers to each other during online discussions affects the quality and fairness of peer review. The study, recorded in a video published by the Peer Review Congress YouTube channel, explored seven research questions.
Key findings include:
- Discussion volume: Researchers examined whether reviewers engage in more discussion when they are anonymous versus when their identities are revealed.
- Decision influence: The study looked at whether final decisions align more closely with senior or junior reviewers' opinions under different anonymity conditions.
- Politeness: The trial measured whether reviewers are more polite when not anonymous.
- Reviewer experience: Self-reported experiences were compared between the two conditions.
- Preference: Reviewers were asked which condition they preferred.
- Important factors: The study identified what factors reviewers consider crucial in the policy decision on anonymity.
- Dishonest behavior: Finally, it investigated whether reviewers had experienced dishonest behavior when their identity was revealed to other reviewers.
The peer review process in computer science often involves individual reviews followed by a typed forum discussion. This research provides evidence to inform whether journals and conferences should keep reviewers anonymous to each other during that discussion phase.