Chen, Shiping;
Brumby, Duncan;
Cox, Anna;
(2025)
Envisioning the Future of Peer Review: Investigating LLM-Assisted Reviewing Using ChatGPT as a Case Study.
In:
Proceedings of the 4th Annual Symposium on Human-Computer Interaction for Work (CHIWORK ’25).
(pp. pp. 1-18).
ACM
Preview |
Text
Chen_3729176.3729196.pdf Download (2MB) | Preview |
Abstract
Peer review ensures research quality but faces growing challenges. Large Language Models (LLMs) offer potential solutions, yet most research has focused on independent AI-generated reviews rather than human-AI collaboration. We conducted a within-subject experiment with 24 HCI reviewers to evaluate LLM-assisted peer review’s impact on review efficiency, quality, workload, and user perceptions. Analysing subjective ratings, objective completion data, qualitative interviews, and chat history, we found that ChatGPT significantly reduces workload but does not substantially shorten review time or improve review quality. Reviewers valued its support for summarisation, information retrieval, idea generation, and confidence-building but noted challenges in content verification, interaction inefficiencies, and the need for human oversight. Our findings contribute to the broader conversation on the role of technology in scholarly publishing, offering design insights for supportive review systems that complement rather than replace human expertise, and highlighting the importance of responsible integration in academic peer review.
| Type: | Proceedings paper |
|---|---|
| Title: | Envisioning the Future of Peer Review: Investigating LLM-Assisted Reviewing Using ChatGPT as a Case Study |
| Event: | Proceedings of the 4th Annual Symposium on Human-Computer Interaction for Work (CHIWORK ’25) |
| Location: | Amsterdam, Netherlands |
| ISBN-13: | 979-8-4007-1384-2 |
| Open access status: | An open access version is available from UCL Discovery |
| DOI: | 10.1145/3729176.3729196 |
| Publisher version: | https://doi.org/10.1145/3729176.3729196 |
| Language: | English |
| Additional information: | This work is licensed under a Creative Commons Attribution International 4.0 License. |
| Keywords: | Peer Review; LLM; AI; Human-AI Collaboration; Future Work; Usercentred Design; Mixed-method Study; Empirical Study |
| UCL classification: | UCL UCL > Provost and Vice Provost Offices > School of Life and Medical Sciences UCL > Provost and Vice Provost Offices > School of Life and Medical Sciences > Faculty of Brain Sciences UCL > Provost and Vice Provost Offices > School of Life and Medical Sciences > Faculty of Brain Sciences > Div of Psychology and Lang Sciences |
| URI: | https://discovery.ucl.ac.uk/id/eprint/10208390 |
Archive Staff Only
 |
View Item |
