Fairbairn, Judith;
(2025)
Raters’ perceptions of topic relevance
at high intermediate (CEFR B2) level in Aptis speaking performances.
Doctoral thesis (Ph.D), UCL (University College London).
Preview |
Text
Fairbairn_10214172_thesis.pdf Download (4MB) | Preview |
Abstract
Deciding if a test taker’s spoken response to a task is on topic is not straightforward, with likely variation in raters’ perceptions (Knoch et al., 2021). There has been no dedicated focus on topic relevance in language assessment research, although this content-based criterion is often found to be a consideration. To address this gap, this study employs a mixed methods design to examine how and why raters make ontopic decisions. The research questions are twofold: to examine (1) the internal criteria that raters report using to decide if a speaking response is on- or off-topic and (2) the relationship between these internal criteria and raters’ Aptis scoring and on-topic decisions with respect to rating severity, reliability, consistency and construct relevance. Thirty Aptis raters marked 45 recorded B2-level speaking responses chosen from the British Council corpus of 9000 Aptis General speaking tests. Raters provided on-topic decisions, rationales for their decisions and overall Aptis scores, and participated in semi-structured email exchanges. Many-Facet Rasch Measurement of on-topic decisions and Aptis scores investigated rater severity, leniency, fit and bias. Comparative thematic analysis of raters’ written perceptions identified themes and severe, consistent and construct-relevant perceptions. Findings identified a wide spectrum of perceptions by raters of acceptable topic relevance performance and these decisions affect scoring. The findings identify areas to consider when developing topic relevance constructs and widens the socio-cognitive framework for test validation (Weir, 2005). Implications include the development of a topic relevance construct for Aptis and improving training for British Council raters. Adjustments have already been made to test specifications for item writers based on these findings.
| Type: | Thesis (Doctoral) |
|---|---|
| Qualification: | Ph.D |
| Title: | Raters’ perceptions of topic relevance at high intermediate (CEFR B2) level in Aptis speaking performances |
| Open access status: | An open access version is available from UCL Discovery |
| Language: | English |
| Additional information: | Copyright © The Author 2025. Original content in this thesis is licensed under the terms of the Creative Commons Attribution-NonCommercial 4.0 International (CC BY-NC 4.0) Licence (https://creativecommons.org/licenses/by-nc/4.0/). Any third-party copyright material present remains the property of its respective owner(s) and is licensed under its existing terms. Access may initially be restricted at the author’s request. |
| Keywords: | Language testing, Speaking on-topic, Rating |
| UCL classification: | UCL UCL > Provost and Vice Provost Offices > School of Education UCL > Provost and Vice Provost Offices > School of Education > UCL Institute of Education UCL > Provost and Vice Provost Offices > School of Education > UCL Institute of Education > IOE - Culture, Communication and Media |
| URI: | https://discovery.ucl.ac.uk/id/eprint/10214172 |
Archive Staff Only
 |
View Item |
