Singmann, Henrik;
Kellen, David;
Cox, Gregory E;
Chandramouli, Suyog H;
Davis-Stober, Clintin P;
Dunn, John C;
Gronau, Quentin F;
... Shiffrin, Richard M; + view all
(2022)
Statistics in the Service of Science: Don't Let the Tail Wag the Dog.
Computational Brain & Behavior
10.1007/s42113-022-00129-2.
(In press).
Preview |
Text
Singmann_s42113-022-00129-2.pdf Download (934kB) | Preview |
Abstract
Statistical modeling is generally meant to describe patterns in data in service of the broader scientific goal of developing theories to explain those patterns. Statistical models support meaningful inferences when models are built so as to align parameters of the model with potential causal mechanisms and how they manifest in data. When statistical models are instead based on assumptions chosen by default, attempts to draw inferences can be uninformative or even paradoxical—in essence, the tail is trying to wag the dog. These issues are illustrated by van Doorn et al. (this issue) in the context of using Bayes Factors to identify effects and interactions in linear mixed models. We show that the problems identified in their applications (along with other problems identified here) can be circumvented by using priors over inherently meaningful units instead of default priors on standardized scales. This case study illustrates how researchers must directly engage with a number of substantive issues in order to support meaningful inferences, of which we highlight two: The first is the problem of coordination, which requires a researcher to specify how the theoretical constructs postulated by a model are functionally related to observable variables. The second is the problem of generalization, which requires a researcher to consider how a model may represent theoretical constructs shared across similar but non-identical situations, along with the fact that model comparison metrics like Bayes Factors do not directly address this form of generalization. For statistical modeling to serve the goals of science, models cannot be based on default assumptions, but should instead be based on an understanding of their coordination function and on how they represent causal mechanisms that may be expected to generalize to other related scenarios.
| Type: | Article |
|---|---|
| Title: | Statistics in the Service of Science: Don't Let the Tail Wag the Dog |
| Open access status: | An open access version is available from UCL Discovery |
| DOI: | 10.1007/s42113-022-00129-2 |
| Publisher version: | https://doi.org/10.1007/s42113-022-00129-2 |
| Language: | English |
| Additional information: | This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article's Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article's Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/. |
| Keywords: | Mixed models, Default Bayes Factors, Standardized effect sizes, Coordination function, Generalizability, Statistics |
| UCL classification: | UCL > Provost and Vice Provost Offices > School of Life and Medical Sciences > Faculty of Brain Sciences UCL > Provost and Vice Provost Offices > School of Life and Medical Sciences > Faculty of Brain Sciences > Div of Psychology and Lang Sciences > Experimental Psychology UCL > Provost and Vice Provost Offices > School of Life and Medical Sciences UCL UCL > Provost and Vice Provost Offices > School of Life and Medical Sciences > Faculty of Brain Sciences > Div of Psychology and Lang Sciences |
| URI: | https://discovery.ucl.ac.uk/id/eprint/10146347 |
Archive Staff Only
 |
View Item |
