A colleague writes:
This is a very provocative claim about the Sturgis rally—can you do a stats “fact check”? I’m curious if this has been subjected to statistical scrutiny.
I replied that I’m curious why he said this study is provocative: It makes sense that when people get together and connect nodes in the social network, that they’ll spread disease, right?? I don’t know about the exact numbers, but the general idea seems reasonable. The only part of the abstract that really rubs me the wrong way is the last sentence, “We conclude that the Sturgis Motorcycle Rally generated public health costs of approximately $12.2 billion.” That sounds like B.S. double-counting (not that I’ve read the paper in detail; I’m just generally suspicious of this sort of claim).
But, the $12 billion dollar thing aside, what is it about the paper that you would consider controversial? Indeed, why would it get a lot of attention? I feel like I’m missing something.
My colleague responded:
I’m not into converting everything into money-equivalents so I’m not concerned with that part.
But just to emphasize that your view isn’t widely held, see here.
I think my point is that I find it plausible so sure in some sense it isn’t an outrageous claim, but not sure the evidence really shifts my prior that much. Someone sent me some data on mask rules claiming that there was observational evidence that masking reduces transmission. I strongly disagreed based on the data he sent – I just felt we couldn’t learn anything. So my priors are still strongly that masking helps. But looking at the timing of community masking rules in that other study didn’t really change that view.
I replied: So you’re saying the result is believable but that this is just one (noisily measured) data point so it doesn’t carry much information? That makes sense to me. But that’s different from the link in that twitter thread, which calls the claim “ridiculous.”
More generally, this reflects a problem with scientific communication: individual studies are supposed to be definitive, so what do you do with an analysis that is consistent with a generally plausible claim but would not represent strong evidence on its own? My colleague referred to the claim in that paper as being “provocative,” but I think the provocative part is not the substantive claim but rather the meta-claim that the paper represents strong evidence in favor of the substantive claim, rather thank weak evidence that is consistent with the substantive claim.
P.S. Thanks to Zad for the most adorable version of quarantining with your family in a small apartment.