Zieger, Laura Raffaella; (2022) Essays on the psychometric and statistical properties of the Programme for International Student Assessment. Doctoral thesis (Ph.D), UCL (University College London).

Preview

Text Thesis_Zieger_final.pdf - Accepted Version Download (4MB) | Preview

Abstract

The Organisation for Economic Co-operation and Development’s (OECD) Programme for International Student Assessment (PISA) has become one of the key studies for evidence-based education policymaking across the globe. Yet its psychometric and statistical properties – and therefore its limitations and implications – are often poorly understood by researchers and stakeholders. This thesis aims to shed new light on the impact that psychometric and statistical issues can have on key statistics and cross-country comparisons. In the first study, I investigate the ‘conditioning model’, where background variables are used in the derivation of student achievement scores. Thereby, I systematically vary the background variables used within the conditioning model and analyse the impact upon country rankings. My key finding is that the exact specification of the conditioning model matters; cross-country comparisons of PISA scores can change substantially depending upon the statistical methodology used. The second study is an extension of the first, as I conduct a simulation study to further investigate research questions and additionally examine other key aspects influencing the achievement score computation, more specifically the preparation of variables and the test and background questionnaire design. This study confirms that the specification of the conditioning model matters, though bias in the results can be reduced by asking students questions in all subjects. In contrast, variable preparation and the background questionnaire design have negligible impact. The third study aims to provide a broader and more comprehensive review of different psychometric and statistical properties in PISA. By using a case study, six different properties that can potentially affect the validity of parental education group comparisons in Germany are evaluated. This study highlights how diverse sources of bias can be in PISA, and that they indeed impact the results. While I did not find that all the investigated issues introduce bias, several properties substantially impacted the results.

Download activity - last month

Export as CSVXMLJSON

Download activity - last 12 months

Export as CSVJSONXML

Downloads by country - last 12 months

Export as CSVXMLJSON

Archive Staff Only

View Item