Kelly, Katherine Mary;
(2025)
Practitioners' understanding of validity in the context of comparative judgement: Implications for practice in standard setting for general qualifications in England.
Doctoral thesis (Ph.D), UCL (University College London).
Preview |
Text
Kelly_10204326_thesis.pdf Download (1MB) | Preview |
Abstract
Grading standards for high-stakes national assessments in England are set using statistics and examiner judgement. Research highlighting the limitations of examiner judgement has eroded trust in this evidence, and statistical evidence has come to predominate. Nevertheless, there is a compelling case for retaining a judgemental element. The industry is thus exploring alternative methods of eliciting judgements. One option is comparative judgement. However, comparative judgement lacks a coherent and well-established theoretical basis that sets out “what good looks like” – both in terms of how validity is understood, and what that means for designing and conducting comparative judgement exercises. This study therefore has two research questions. First, how do comparative judgement practitioners conceptualise validity? Second, how do practitioners use this conceptualisation to inform their design choices for comparative judgement exercises? I conducted semi-structured interviews with 13 comparative judgement practitioners and analysed the interviews using constructivist grounded theory. There were four key findings: • Best practice is highly contextualised, but is underpinned by a common process; • The assessment construct defines and is defined by expert judges; • Practitioners indirectly exert power over the assessment construct; • Practitioners operationalise validity as the fidelity with which the construct is translated into practice. Like judgemental evidence in standard setting, comparative judgement has tended to be used when conventional methods have failed. As such, comparative judgement is unlikely to resolve the tensions around using examiner judgement in standard setting. I conclude that the examinations industry should not focus on how to improve examiner judgement, but instead consider under what conditions it would trust in examiner judgement.
| Type: | Thesis (Doctoral) |
|---|---|
| Qualification: | Ph.D |
| Title: | Practitioners' understanding of validity in the context of comparative judgement: Implications for practice in standard setting for general qualifications in England |
| Open access status: | An open access version is available from UCL Discovery |
| Language: | English |
| Additional information: | Copyright © The Author 2025. Original content in this thesis is licensed under the terms of the Creative Commons Attribution-NonCommercial 4.0 International (CC BY-NC 4.0) Licence (https://creativecommons.org/licenses/by-nc/4.0/). Any third-party copyright material present remains the property of its respective owner(s) and is licensed under its existing terms. Access may initially be restricted at the author’s request. |
| UCL classification: | UCL UCL > Provost and Vice Provost Offices > School of Education UCL > Provost and Vice Provost Offices > School of Education > UCL Institute of Education UCL > Provost and Vice Provost Offices > School of Education > UCL Institute of Education > IOE - Curriculum, Pedagogy and Assessment |
| URI: | https://discovery.ucl.ac.uk/id/eprint/10204326 |
Loading...
Loading...Loading...Loading...Archive Staff Only
 |
View Item |
