eprintid: 10204326 rev_number: 14 eprint_status: archive userid: 699 dir: disk0/10/20/43/26 datestamp: 2025-03-07 09:58:06 lastmod: 2025-03-07 09:58:06 status_changed: 2025-03-07 09:58:06 type: thesis metadata_visibility: show sword_depositor: 699 creators_name: Kelly, Katherine Mary title: Practitioners' understanding of validity in the context of comparative judgement: Implications for practice in standard setting for general qualifications in England ispublished: unpub divisions: UCL divisions: B16 divisions: B14 divisions: J78 note: Copyright © The Author 2025. Original content in this thesis is licensed under the terms of the Creative Commons Attribution-NonCommercial 4.0 International (CC BY-NC 4.0) Licence (https://creativecommons.org/licenses/by-nc/4.0/). Any third-party copyright material present remains the property of its respective owner(s) and is licensed under its existing terms. Access may initially be restricted at the author’s request. abstract: Grading standards for high-stakes national assessments in England are set using statistics and examiner judgement. Research highlighting the limitations of examiner judgement has eroded trust in this evidence, and statistical evidence has come to predominate. Nevertheless, there is a compelling case for retaining a judgemental element. The industry is thus exploring alternative methods of eliciting judgements. One option is comparative judgement. However, comparative judgement lacks a coherent and well-established theoretical basis that sets out “what good looks like” – both in terms of how validity is understood, and what that means for designing and conducting comparative judgement exercises. This study therefore has two research questions. First, how do comparative judgement practitioners conceptualise validity? Second, how do practitioners use this conceptualisation to inform their design choices for comparative judgement exercises? I conducted semi-structured interviews with 13 comparative judgement practitioners and analysed the interviews using constructivist grounded theory. There were four key findings: • Best practice is highly contextualised, but is underpinned by a common process; • The assessment construct defines and is defined by expert judges; • Practitioners indirectly exert power over the assessment construct; • Practitioners operationalise validity as the fidelity with which the construct is translated into practice. Like judgemental evidence in standard setting, comparative judgement has tended to be used when conventional methods have failed. As such, comparative judgement is unlikely to resolve the tensions around using examiner judgement in standard setting. I conclude that the examinations industry should not focus on how to improve examiner judgement, but instead consider under what conditions it would trust in examiner judgement. date: 2025-02-28 date_type: published oa_status: green full_text_type: other thesis_class: doctoral_open thesis_award: Ph.D language: eng primo: open primo_central: open_green verified: verified_manual elements_id: 2358111 lyricists_name: Kelly, Katherine lyricists_id: KMTKE44 actors_name: Kelly, Katherine actors_id: KMTKE44 actors_role: owner full_text_status: public pages: 208 institution: UCL (University College London) department: Curriculum, Pedagogy & Assessment, UCL Institute of Education thesis_type: Doctoral citation: Kelly, Katherine Mary; (2025) Practitioners' understanding of validity in the context of comparative judgement: Implications for practice in standard setting for general qualifications in England. Doctoral thesis (Ph.D), UCL (University College London). Green open access document_url: https://discovery.ucl.ac.uk/id/eprint/10204326/13/Kelly_10204326_thesis.pdf