Siepe, Björn S;
Bartoš, František;
Morris, Tim P;
Boulesteix, Anne-Laure;
Heck, Daniel W;
Pawel, Samuel;
(2024)
Simulation studies for methodological research in psychology: A standardized template for planning, preregistration, and reporting.
Psychological Methods
10.1037/met0000695.
(In press).
Preview |
Text
Simulation_Studies_for_Methodological_Research_in_Psychology_ACCEPTED.pdf - Accepted Version Download (1MB) | Preview |
Abstract
Simulation studies are widely used for evaluating the performance of statistical methods in psychology. However, the quality of simulation studies can vary widely in terms of their design, execution, and reporting. In order to assess the quality of typical simulation studies in psychology, we reviewed 321 articles published in Psychological Methods, Behavior Research Methods, and Multivariate Behavioral Research in 2021 and 2022, among which 100/321 = 31.2% report a simulation study. We find that many articles do not provide complete and transparent information about key aspects of the study, such as justifications for the number of simulation repetitions, Monte Carlo uncertainty estimates, or code and data to reproduce the simulation studies. To address this problem, we provide a summary of the ADEMP (aims, data-generating mechanism, estimands and other targets, methods, performance measures) design and reporting framework from Morris et al. (2019) adapted to simulation studies in psychology. Based on this framework, we provide ADEMP-PreReg, a step-by-step template for researchers to use when designing, potentially preregistering, and reporting their simulation studies. We give formulae for estimating common performance measures, their Monte Carlo standard errors, and for calculating the number of simulation repetitions to achieve a desired Monte Carlo standard error. Finally, we give a detailed tutorial on how to apply the ADEMP framework in practice using an example simulation study on the evaluation of methods for the analysis of pre–post measurement experiments.
| Type: | Article |
|---|---|
| Title: | Simulation studies for methodological research in psychology: A standardized template for planning, preregistration, and reporting |
| Open access status: | An open access version is available from UCL Discovery |
| DOI: | 10.1037/met0000695 |
| Publisher version: | http://dx.doi.org/10.1037/met0000695 |
| Language: | English |
| Additional information: | This version is the author accepted manuscript. For information on re-use, please refer to the publisher's terms and conditions. |
| UCL classification: | UCL UCL > Provost and Vice Provost Offices > School of Life and Medical Sciences UCL > Provost and Vice Provost Offices > School of Life and Medical Sciences > Faculty of Population Health Sciences > Inst of Clinical Trials and Methodology UCL > Provost and Vice Provost Offices > School of Life and Medical Sciences > Faculty of Population Health Sciences > Inst of Clinical Trials and Methodology > MRC Clinical Trials Unit at UCL |
| URI: | https://discovery.ucl.ac.uk/id/eprint/10200328 |
Archive Staff Only
 |
View Item |
