Mignemi, Giuseppe;
Calcagni, Antonio;
Spoto, Andrea;
Manolopoulou, Ioanna;
(2024)
Mixture polarization in inter-rater agreement analysis: a Bayesian nonparametric index.
Statistical Methods and Applications
10.1007/s10260-023-00741-x.
(In press).
Preview |
Text
Manolopoulou_s10260-023-00741-x.pdf Download (2MB) | Preview |
Abstract
In several observational contexts where different raters evaluate a set of items, it is common to assume that all raters draw their scores from the same underlying distribution. However, a plenty of scientific works have evidenced the relevance of individual variability in different type of rating tasks. To address this issue the intra-class correlation coefficient (ICC) has been used as a measure of variability among raters within the Hierarchical Linear Models approach. A common distributional assumption in this setting is to specify hierarchical effects as independent and identically distributed from a normal with the mean parameter fixed to zero and unknown variance. The present work aims to overcome this strong assumption in the inter-rater agreement estimation by placing a Dirichlet Process Mixture over the hierarchical effects’ prior distribution. A new nonparametric index λ is proposed to quantify raters polarization in presence of group heterogeneity. The model is applied on a set of simulated experiments and real world data. Possible future directions are discussed.
| Type: | Article |
|---|---|
| Title: | Mixture polarization in inter-rater agreement analysis: a Bayesian nonparametric index |
| Open access status: | An open access version is available from UCL Discovery |
| DOI: | 10.1007/s10260-023-00741-x |
| Publisher version: | http://dx.doi.org/10.1007/s10260-023-00741-x |
| Language: | English |
| Additional information: | Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article's Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article's Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/. |
| Keywords: | Science & Technology, Physical Sciences, Statistics & Probability, Mathematics, Bayesian nonparametrics, Inter-rater agreement, Dirichlet process mixture, Hierarchical Bayesian models, RANDOM-EFFECTS MODELS, LINEAR MIXED MODELS, TEACHER LIKE ME, CULTURAL CONSENSUS, RELIABILITY, HETEROGENEITY |
| UCL classification: | UCL UCL > Provost and Vice Provost Offices > UCL BEAMS UCL > Provost and Vice Provost Offices > UCL BEAMS > Faculty of Maths and Physical Sciences UCL > Provost and Vice Provost Offices > UCL BEAMS > Faculty of Maths and Physical Sciences > Dept of Statistical Science |
| URI: | https://discovery.ucl.ac.uk/id/eprint/10187207 |
Archive Staff Only
 |
View Item |
