Skip to content

The turtles stop here. Why we meta-science: a meta-meta-science manifesto

All those postscripts in the previous post . . . this sort of explanation of why I’m writing about the scientific process, it comes up a lot.

I spend a lot of time thinking and writing about the research process, rather than just doing research.

And all too often I often find myself taking time out to explain why I’m spending time on meta-science that I could be spending on science instead.

My answer is that meta-science discussions are, like it or not, necessary conditions for more technical work. Without meta-science, we keep getting caught in deference traps. Remember when we and others criticized silly regression discontinuity analyses? The response in some quarters was a reflexive deference to standard methods in econometrics, without reflection on the applicability of these methods to the problems at hand. Remember when we and others criticized silly psychology experiments? The response in some quarters was a reflexive deference to standard practices in that field (“You have no choice but to accept,” etc.). Remember when we and others criticized ridiculously large estimated causal effects in policy analysis? The response in some quarters was to not respond at all, perhaps based on a savvy judgment that institutional reputations would outlast technical criticism. Remember pizzagate? The response was to duck and weave, which probably would’ve worked had there not been so so so many data anomalies, and also that particular researcher had no powerful friends to go on the counterattack against his critics. Remember that gremlins research? Again, the researcher didn’t give an inch; he relied on the deference given to the economics profession, and the economics profession didn’t seem to care that its reputation was being used in this way. Remember beauty and sex ratio, divorce predictions, etc.? Technical criticism means nothing at all to the Freakonomicses, Gladwells, NPRs, and Teds of the world. Remember how methods of modern survey adjusted were blasted by authority figures such as the president of the American Association for Public Opinion Research and the world’s most famous statistical analyst? Again, our technical arguments didn’t matter one bit to these people. Technical reasoning didn’t come into play at all. It was just one deference trap after another. So, yes, I spend hours dismantling these deference traps so we can get to our real work. Perhaps not the best use of my time, given all my technical training, but somebody’s gotta do it. I’m sick and tired of feeling like I have to explain myself, but the alternative, where people don’t understand why I’m doing this, seems worse. In the words of Auden, “To-morrow, perhaps the future.”

Indeed, not only do we have to spend valuable time on meta-science so we can get to our science, but sometimes we need to spend time on meta-meta-science as in the above paragraph and the postscripts to the previous post. We need the meta-meta-science to explain to people who would otherwise ask why we are wasting our time on meta-science.



  1. Tom Passin says:

    I seem to remember many years ago reading a paper that looked at the effectiveness of a number of meta-analyses. It concluded, IIRC, that they didn’t do very well (in view of later information, I suppose).

    Anyone else remember about this or similar studies?

  2. Nicely put, but you know turtles never stop…

  3. Mark Palko says:

    A bit surprised you didn’t go with Carroll’s Achilles and the Tortoise

  4. Jonathan says:

    Thanks again for the link about moral economy. It may sound funny to this audience but Taylor Swift perfectly encapsulates Trump in those deference terms: saying he has the nerve to feign moral superiorty to threaten violence. When you feign moral superiority, you’re asserting your story version is correct, is the winner, is structurally the outcome. A dominance assertion very much like you often describe in refusals to share data.

  5. Carlo Mengoli says:

    Meta-science and epistemology are the same thing.
    Epistmology is philosophy.
    Philosophy is necessary to human life.

  6. jim says:

    “I spend a lot of time thinking and writing about the research process, rather than just doing research.”

    Thank you, Andrew. Your technical knowledge and credibility is spent wisely on the issues you raise here. Your work and that of the many professionals who share your views is changing the world for the better. It’s a noble service to science, to humanity and to your country.

  7. Megan Higgs says:

    Scientists have a responsibility to think and talk about how we are doing science, and to count it as “real work.” Without that, there is an implicit, unquestioning acceptance of methodology and current norms — which of course sounds ironically unscientific. Thinking and writing about the research process is real work — it is doing research — even if it’s not valued within the current culture as much as highly technical work. The people doing the technical work must to be part of the conversation to avoid a huge gap between philosophical discussions and what’s actually happening in practice. I think that’s one big reason people love this blog. It’s hard to imagine any endeavor where work on how and why we’re doing it wouldn’t be crucial to the success of the endeavor.

  8. Andre Ariew says:

    If only more of us philosophers of science would perform the meta science (i.e., our job) then you folks could go back to doing your research. On the other hand, maybe the extant philosophy of science is unrecognized or under appreciated. I don’t know which is the case. Perhaps the research issues that require meta science are too specific, out of the purview of the philosophy of science cannon. It would be nice if there was a better connection between science and philosophy of science. But it is hard enough to keep up with our own literatures.

  9. yyw says:

    Based on my observation, the problem is that we have a substantial portion (large majority?) of social scientists and even many statisticians that are not adequately trained. They are not equipped to make informed choice between a reputable statistician like Andrew and standard practice. NPR and Gladwells of the world are even more hopeless.

  10. turtle says:

    people should read up on magnitude based inference and how its been applied by a generation of sports scientists. For another example.

  11. Manuel says:

    Calling “thinking and writing about the research process” meta-science is a terrible error. Call it just science, that’s what it is.

    Putting “just doing research” on a pedestal is silly. “Doing” something without “thinking about it” is the antithesis of science. And this applies also to “research”.

    • Michael Nelson says:

      Good point. The scientific method includes a stage that is “report” or “communicate,” which implies a stage in which the work of others is reviewed. It’s usually not explicitly stated because we all like that circle of arrows that illustrates the iterative nature of science. I think that leads a lot of scientists to consider activities outside that closed loop as extra-scientific or para-scientific–also, we tend not to be paid for it, so things like peer review may seem more like a virtue than an obligation. Maybe we should modify the circle diagram to make it two interlocking rings, which cross at the “report” stage and again at the “formulate a question” stage, indicating that we critique others’ work, in part, so that we can continue to build upon it.

Leave a Reply