Abstract

Evaluating proposals to improve scientific research can be challenging, partly because individual actions can combine and propagate downstream in complex and unexpected ways. In this symposium, we use simulations to delve deeper into how the effects of questionable research practices and p-hacking propagate into the research literature. In particular, we present SAM, an extensible and modular framework for simulating and studying the effects of various forms of QRPs on scientific results, as well as their propagative influence on meta-analytic metrics and publication bias detection/correction methods. In addition, we use data from Registered Replication Reports to emulate and show how researcher degrees of freedom and selective reporting in individual studies propagate to meta-analysis.

First, Anton Olsson-Collentine will present the results from applying multiverse analysis to 236 labs across 10 Registered Replication Reports, demonstrating what meta-analytic data can look like under the hood.

Second, Amir Abdol will describe the underlying design principles of SAM, its main features, and how it can be used to model the scientific process. Jelte Wicherts will then discuss the results of a study replicated and extended in SAM, in which the effect of various strategies on effect size bias and chances of finding significant results were evaluated, as well as the utility of lowering the alpha threshold from .05 to .005, or .0005, across different strategies.

Finally, Esther Maassen will present the results of a simulation study performed in SAM, in which the effects of popular p-hacking strategies and publication bias on meta-analytic results are examined.