Dr Jessica Flake (University of British Columbia)

Title: Methodological Research Needs Methodological Reform Too

Abstract:

Psychology is in a period of methodological reform. Researchers are rethinking their practices, sharing their data, and trying out registered reports. In this open science era, my work has focused on the role of measurement practices. I’ll provide some background as to how that previous metascience and psychometric research in the context of replication studies led me to discover two related problems. First, even as registration becomes common, we lack practices to analysis plan for complex models and to ensure their transparent reporting and reproducibility. Second, current methodological research does not address this because it does not focus on how to navigate the garden of forking paths and quantify the uncertainty in results that comes from reasonable analytical flexibility. In fact, methodological research seeks to increase the size of the garden of forking paths! These problems prevent the uptake of open science practices and threaten the validity of research results. I’ll preview my on-going work to develop research synthesis methods for methodologies and ways of integrating multiverse analysis with metascience to quantify analytical uncertainty. I’ll discuss how applied and meta researchers can consider decision making in analysis pipelines as a unique contributor to the uncertainty of results and ideas for correcting for it.

Bio:

Dr. Flake developed an interest in psychology and statistics as an undergraduate at Northern Kentucky University and went on to complete a PhD in Measurement, Evaluation, and Assessment from the University of Connecticut in 2015. She recently joined the faculty in the Quantitative Methods Area of the Department of Psychology at University of British Columbia, after having been an Associate Professor at McGill University. Her lab works on applied problems in psychological measurement and psychometrics with an emphasis on reproducibility and replicability. She was a founding member of the Psychological Science Accelerator, a distributed laboratory network of over 1,000 researchers, serving as the Assistant Director of the Data and Methods Committee from 2018 to 2023. She continues to work on methodological challenges for big team science and psychometric methods for complex data structures. Currently she is an Associate Editor at Advances in Methods and Practices in Psychological Science and Psychological Methods. Her recent and future work is considering how open science practices like registered reports and multiverse analysis can be further developed for latent variable models.