This came up in discussion the other day. In statistics and medicine, we’re generally told to rely when possible on the statistically significance (or lack of statistical significance) of results from randomized trials. But, as we know, statistical significance has all sorts of problems, most notably that it ignores questions of cost and benefit, and […]
Archive of posts filed under the Causal Inference category.
Please socially distance me from this regression model!
A biostatistician writes: The BMJ just published a paper using regression discontinuity to estimate the effect of social distancing. But they have terrible models. As I am from Canada, I had particular interest in the model for Canada, which is on their supplemental material, page 84 [reproduced above]. I could not believe this was published. […]
Association Between Universal Curve Fitting in a Health Care Journal and Journal Acceptance Among Health Care Researchers
Matt Folz points us to this recent JAMA article that features this amazing graph: Beautiful. Just beautiful. I say this ironically.
Further debate over mindset interventions
Warne Following up on this post, “Study finds ‘Growth Mindset’ intervention taking less than an hour raises grades for ninth graders,” commenter D points us to this post by Russell Warne that’s critical of research on growth mindset. Here’s Warne: Do you believe that how hard you work to learn something is more important than […]
“To Change the World, Behavioral Intervention Research Will Need to Get Serious About Heterogeneity”
Beth Tipton, Chris Bryan, and David Yeager write: The increasing influence of behavioral science in policy has been a hallmark of the past decade, but so has a crisis of confidence in the replicability of behavioral science findings. In this essay, we describe a nascent paradigm shift in behavioral intervention research—a heterogeneity revolution—that we believe […]
Adjusting for Type M error
Erik Drysdale discusses and gives some formulas, demonstrating on an example that will be familiar to regular readers of this blog.
Coronavirus jailbreak
Emma Pierson writes: My two sisters and I, with my friend Jacob Steinhardt, spent the last several days looking at the statistical methodology in a paper which has achieved a lot of press – Incarceration and Its Disseminations: COVID-19 Pandemic Lessons From Chicago’s Cook County Jail (results in supplement), published in Health Affairs. (Here’s the […]
Regression and Other Stories is available!
This will be, without a doubt, the most fun you’ll have ever had reading a statistics book. Also I think you’ll learn a few things reading it. I know that we learned a lot writing it. Regression and Other Stories started out as the first half of Data Analysis Using Regression and Multilevel/Hierarchical Models, but […]
No, I don’t believe that claim based on regression discontinuity analysis that . . .
tl;dr. See point 4 below. Despite the p-less-than-0.05 statistical significance of the discontinuity in the above graph, no, I do not believe that losing a close election causes U.S. governors to die 5-10 years longer, as was claimed in this recently published article. Or, to put it another way: Despite the p-less-than-0.05 statistical significance of […]
The value of thinking about varying treatment effects: coronavirus example
Yesterday we discussed difficulties with the concept of average treatment effect. Part of designing a study is accounting for uncertainty in effect sizes. Unfortunately there is a tradition in clinical trials of making optimistic assumptions in order to claim high power. Here is an example that came up in March, 2020. A doctor was designing […]
Understanding the “average treatment effect” number
In statistics and econometrics there’s lots of talk about the average treatment effect. I’ve often been skeptical of the focus on the average treatment effect, for the simple reason that, if you’re talking about an average effect, then you’re recognizing the possibility of variation; and if there’s important variation (enough so that we’re talking about […]
The point here is not the face masks; it’s the impossibility of assumption-free causal inference when the different treatments are entangled in this way.
Adam Pearce writers: When I read your Another Regression Discontinuity Disaster post last year, I was curious how much shifting the breakpoint would change the fit lines. A covid paper making the rounds this weekend used a similar technique so I hooked it up to an interactive widget that lets you tweak the start and […]
Challenges to the Reproducibility of Machine Learning Models in Health Care; also a brief discussion about not overrating randomized clinical trials
Mark Tuttle pointed me to this article by Andrew Beam, Arjun Manrai, and Marzyeh Ghassemi, Challenges to the Reproducibility of Machine Learning Models in Health Care, which appeared in the Journal of the American Medical Association. Beam et al. write: Reproducibility has been an important and intensely debated topic in science and medicine for the […]
How should those Lancet/Surgisphere/Harvard data have been analyzed?
As you will recall, the original criticism of the recent Lancet/Surgisphere/Harvard paper on hydro-oxy-whatever was not that the data came from a Theranos-like company that employs more adult-content models than statisticians, but rather that the data, being observational, required some adjustment to yield strong causal conclusions—and the causal adjustment reported in that article did not […]
Sequential Bayesian Designs for Rapid Learning in COVID-19 Clinical Trials
This from Frank Harrell looks important: This trial will adopt a Bayesian framework. Continuous learning from data and computation of probabilities that are directly applicable to decision making in the face of uncertainty are hallmarks of the Bayesian approach. Bayesian sequential designs are the simplest of flexible designs, and continuous learning capitalizes on their efficiency, […]
This one’s for the Lancet editorial board: A trolley problem for our times (involving a plate of delicious cookies and a steaming pile of poop)
A trolley problem for our times OK, I couldn’t quite frame this one as a trolley problem—maybe those of you who are more philosophically adept than I am can do this—so I set it up as a cookie problem? Here it is: Suppose someone was to knock on your office door and use some mix […]
“The good news about this episode is that it’s kinda shut up those people who were criticizing that Stanford antibody study because it was an un-peer-reviewed preprint. . . .” and a P.P.P.S. with Paul Alper’s line about the dead horse
People keep emailing me about this recently published paper, but I already said I’m not going to write about it. So I’ll mask the details. Philippe Lemoine writes: So far it seems you haven’t taken a close look at the paper yourself and I’m hoping that you will, because I’m curious to know what you […]
This is not a post about remdesivir.
Someone pointed me to this post by a doctor named Daniel Hopkins on a site called KevinMD.com, expressing skepticism about a new study of remdesivir. I guess some work has been done following up on that trial on 18 monkeys. From the KevinMD post: On April 29th Anthony Fauci announced the National Institute of Allergy […]
Alexey Guzey’s sleep deprivation self-experiment
Alexey “Matthew Walker’s ‘Why We Sleep’ Is Riddled with Scientific and Factual Errors” Guzey writes: I [Guzey] recently finished my 14-day sleep deprivation self experiment and I ended up analyzing the data I have only in the standard p < 0.05 way and then interpreting it by writing explicitly about how much I believe I […]
Be careful when estimating years of life lost: quick-and-dirty estimates of attributable risk are, well, quick and dirty.
Peter Morfeld writes: Global burden of disease (GBD) studies and environmental burden of disease (EBD) studies are supported by hundreds of scientifically well-respected co-authors, are published in high level journals, are cited world wide and have a large impact on health institutions‘ reports and related political discussions. The main metrics used to calculate the impact […]