Lam, Yu Yan;
Chu, Samuel Kai Wah;
Ong, Elsie Li Chen;
Suen, Winnie Wing Lam;
Xu, Lingran;
Lam, Lavender Chin Lui;
Wong, Scarlett Man Yu;
(2024)
Comparative Study of GenAI (ChatGPT) vs. Human in Generating Multiple Choice Questions Based on the PIRLS Reading Assessment Framework.
Proceedings of the Association for Information Science and Technology
, 61
(1)
pp. 537-540.
10.1002/pra2.1054.
Preview |
Text
Comparative Study of GenAI (ChatGPT) vs. Human in Generating MCQ Based on the PIRLS Reading Assessment Framework.pdf - Accepted Version Download (245kB) | Preview |
Abstract
Human-generated multiple-choice questions (MCQs) are commonly used to ensure objective evaluation in education. However, generating high-quality questions is difficult and time-consuming. Generative artificial intelligence (GenAI) has emerged as an automated approach for question generation, but challenges remain in terms of biases and diversity in training data. This study aims to compare the quality of GenAI-generated MCQs with humans-created ones. In Part 1 of this study, 16 MCQs were created by humans and GenAI individually with alignment to the Progress in International Reading Literacy Study (PIRLS) assessment framework. In Part 2, the quality of MCQs generated was assessed based on the clarity, appropriateness, suitability, and alignment to PIRLS by four assessors. Wilcoxon rank sum tests were conducted to compare GenAI versus humans generated MCQs. The findings highlight GenAI's potential as it was difficult to differentiate from human created questions and offer recommendations for integrating AI technology for the future.
| Type: | Article |
|---|---|
| Title: | Comparative Study of GenAI (ChatGPT) vs. Human in Generating Multiple Choice Questions Based on the PIRLS Reading Assessment Framework |
| Open access status: | An open access version is available from UCL Discovery |
| DOI: | 10.1002/pra2.1054 |
| Publisher version: | http://dx.doi.org/10.1002/pra2.1054 |
| Language: | English |
| Additional information: | This version is the author accepted manuscript. For information on re-use, please refer to the publisher’s terms and conditions. |
| Keywords: | GenAI; PIRLS; question assessment; question creation; Reading |
| UCL classification: | UCL UCL > Provost and Vice Provost Offices > School of Education UCL > Provost and Vice Provost Offices > School of Education > UCL Institute of Education UCL > Provost and Vice Provost Offices > School of Education > UCL Institute of Education > IOE - Psychology and Human Development |
| URI: | https://discovery.ucl.ac.uk/id/eprint/10200190 |
Archive Staff Only
 |
View Item |
