Wong, Kester;
Bulathwela, Sahan;
Cukurova, Mutlu;
(2025)
Exploring Human-AI Complementarity in CPS Diagnosis Using Unimodal and Multimodal BERT Models.
In:
Proceedings of HEXED 2025: 2nd Human-Centric eXplainable AI in Education (HEXED).
HEXED Workshop: Palermo, Italy.
Preview |
Text
2507.14579v1.pdf - Accepted Version Download (678kB) | Preview |
Abstract
Detecting collaborative problem solving (CPS) indicators from dialogue using machine learning techniques is a significant challenge for the field of AI in Education. Recent studies have explored the use of Bidirectional Encoder Representations from Transformers (BERT) models on transcription data to reliably detect meaningful CPS indicators. A notable advancement involved the multimodal BERT variant, AudiBERT, which integrates speech and acoustic-prosodic audio features to enhance CPS diagnosis. Although initial results demonstrated multimodal improvements, the statistical significance of these enhancements remained unclear, and there was insufficient guidance on leveraging human-AI complementarity for CPS diagnosis tasks. This workshop paper extends the previous research by highlighting that the AudiBERT model not only improved the classification of classes that were sparse in the dataset, but it also had statistically significant class-wise improvements over the BERT model for classifications in the social-cognitive dimension. However, similar significant class-wise improvements over the BERT model were not observed for classifications in the affective dimension. A correlation analysis highlighted that larger training data was significantly associated with higher recall performance for both the AudiBERT and BERT models. Additionally, the precision of the BERT model was significantly associated with high inter-rater agreement among human coders. When employing the BERT model to diagnose indicators within these subskills that were well-detected by the AudiBERT model, the performance across all indicators was inconsistent. We conclude the paper by outlining a structured approach towards achieving human-AI complementarity for CPS diagnosis, highlighting the crucial inclusion of model explainability to support human agency and engagement in the reflective coding process.
| Type: | Proceedings paper |
|---|---|
| Title: | Exploring Human-AI Complementarity in CPS Diagnosis Using Unimodal and Multimodal BERT Models |
| Event: | HEXED 2025: 2nd Human-Centric eXplainable AI in Education (HEXED) Workshop |
| Location: | Palermo, Italy |
| Open access status: | An open access version is available from UCL Discovery |
| Publisher version: | https://hexed-workshop.github.io/ |
| Language: | English |
| Additional information: | Copyright © 2025. Copyright for this paper by its authors. Use permitted under Creative Commons License Attribution 4.0 International (CC BY 4.0) (https://creativecommons.org/licenses/by/4.0/deed.en). |
| Keywords: | multimodal audio data, human-AI complementarity, transformer-based models, collaborative problem solving |
| UCL classification: | UCL UCL > Provost and Vice Provost Offices > School of Education UCL > Provost and Vice Provost Offices > School of Education > UCL Institute of Education UCL > Provost and Vice Provost Offices > School of Education > UCL Institute of Education > IOE - Culture, Communication and Media |
| URI: | https://discovery.ucl.ac.uk/id/eprint/10212934 |
Archive Staff Only
 |
View Item |
