Jenna Harder writes:
When analyzing data, researchers may have multiple reasonable options for the many decisions they must make about the data—for example, how to code a variable or which participants to exclude. Therefore, there exists a multiverse of possible data sets. A classic multiverse analysis involves performing a given analysis on every potential data set in this multiverse to examine how each data decision affects the results. However, a limitation of the multiverse analysis is that it addresses only data cleaning and analytic decisions, yet researcher decisions that affect results also happen at the data-collection stage. I propose an adaptation of the multiverse method in which the multiverse of data sets is composed of real data sets from studies varying in data-collection methods of interest. I walk through an example analysis applying the approach to 19 studies on shooting decisions to demonstrate the usefulness of this approach and conclude with a further discussion of the limitations and applications of this method.
I like this because of the term “classic multiverse analysis.” It’s fun to be a classic!
In all seriousness, I like the multiverse idea, and ideally it should be thought of as a step toward a multilevel model, in the same way that the secret weapon is both a visualization tool and an implicit multilevel model.
I’m also glad that Perspectives on Psychological Science has decided to publish research again, and not just publish lies about people. That’s cool. I guess an apology will never be coming, but at least they’ve moved on.