Singmann, Henrik;
Heck, Daniel W;
Barth, Marius;
Erdfelder, Edgar;
Arnold, Nina R;
Aust, Frederik;
Calanchini, Jimmy;
... Groß, Julia; + view all
(2024)
Evaluating the Robustness of Parameter Estimates in Cognitive Models: A Meta-Analytic Review of Multinomial Processing Tree Models Across the Multiverse of Estimation Methods.
Psychological Bulletin
10.1037/bul0000434.
(In press).
![[thumbnail of Singmann et al. - 2024 - Evaluating the robustness of parameter estimates i.pdf]](https://discovery.ucl.ac.uk/style/images/fileicons/application_pdf.png) |
PDF
Singmann et al. - 2024 - Evaluating the robustness of parameter estimates i.pdf - Published Version Download (5MB) |
Abstract
Researchers have become increasingly aware that data-analysis decisions affect results. Here, we examine this issue systematically for multinomial processing tree (MPT) models, a popular class of cognitive models for categorical data. Specifically, we examine the robustness of MPT model parameter estimates that arise from two important decisions: the level of data aggregation (complete-pooling, no-pooling, or partial-pooling) and the statistical framework (frequentist or Bayesian). These decisions span a multiverse of estimation methods. We synthesized the data from 13,956 participants (164 published data sets) with a meta-analytic strategy and analyzed the magnitude of divergence between estimation methods for the parameters of nine popular MPT models in psychology (e.g., process-dissociation, source monitoring). We further examined moderators as potential sources of divergence. We found that the absolute divergence between estimation methods was small on average (<.04; with MPT parameters ranging between 0 and 1); in some cases, however, divergence amounted to nearly the maximum possible range (.97). Divergence was partly explained by few moderators (e.g., the specific MPT model parameter, uncertainty in parameter estimation), but not by other plausible candidate moderators (e.g., parameter trade-offs, parameter correlations) or their interactions. Partial-pooling methods showed the smallest divergence within and across levels of pooling and thus seem to be an appropriate default method. Using MPT models as an example, we show how transparency and robustness can be increased in the field of cognitive modeling.
| Type: | Article |
|---|---|
| Title: | Evaluating the Robustness of Parameter Estimates in Cognitive Models: A Meta-Analytic Review of Multinomial Processing Tree Models Across the Multiverse of Estimation Methods |
| Open access status: | An open access version is available from UCL Discovery |
| DOI: | 10.1037/bul0000434 |
| Publisher version: | http://dx.doi.org/10.1037/bul0000434 |
| Language: | English |
| Additional information: | This work is licensed under a Creative Commons Attribution 4.0 International License (CC BY 4.0; http://creativecommons.org/licenses/by/4.0). This license permits copying and redistributing the work in any medium or format, as well as adapting the material for any purpose, even commercially. |
| Keywords: | multiverse analysis, parameter estimation, transparency, cognitive modeling, multinomial processing tree models |
| UCL classification: | UCL UCL > Provost and Vice Provost Offices > School of Life and Medical Sciences UCL > Provost and Vice Provost Offices > School of Life and Medical Sciences > Faculty of Brain Sciences UCL > Provost and Vice Provost Offices > School of Life and Medical Sciences > Faculty of Brain Sciences > Div of Psychology and Lang Sciences UCL > Provost and Vice Provost Offices > School of Life and Medical Sciences > Faculty of Brain Sciences > Div of Psychology and Lang Sciences > Experimental Psychology |
| URI: | https://discovery.ucl.ac.uk/id/eprint/10194057 |
Archive Staff Only
 |
View Item |
