Eliot Johnson writes:
You’ve devoted quite a few blog posts to challenging orthodox views regarding statistical significance. If there’s been discussion of this as it relates to the 5-sigma rule in physics, then I’ve missed that thread. If not, why not open up a critical discussion about it?
Here’s a link to one blog post about 5-sigma.
My reply: Physics is an interesting realm to consider for hypothesis testing, because it’s one area where the null hypothesis really might be true, or at least true to several decimal places. On the other hand, with experimental data there will always be measurement error, and your measurement error model will be imperfect.
It’s hard for me to imagine a world in which it makes sense to identify 5 sigma as a “discovery”—but maybe that just indicates the poverty of my imagination!
In all seriousness, I guess I’d have to look more carefully at some particular example. Maybe some physicist could help on this one. My intuition would be that in any problem for which we might want to use such a threshold, we’d be better off fitting a hierarchical model.