I see five problems here that together form a feedback loop with bad consequences. Here are the problems:
1. Irrelevant or misunderstood statistical or econometric theory
2. Poorly-executed research
3. Other people in the field being loath to criticize, taking published or even preprinted claims as correct until proved otherwise
4. Journalists taking published or even preprinted claims as correct until proved otherwise
5. Journalists following the scientist-as-hero template.
There are also related issues such as fads in different academic fields, etc.
When I write about regression discontinuity, we often focus on item #2 above, because it helps to be specific. But the point of my post was that maybe we could work on #3 and #4. If economics could withdraw from their defensive position by which something written by a credentialed economist is considered to be good work until proved otherwise, and if policy journalists could withdraw from their default deference to anything that has an identification strategy and statistical significance, then maybe we could break that feedback loop.
Economists: You know better! Think about your own applied work. You’ve done bad analyses yourself, even published some bad analyses, right? I know I have. Given that you’ve done it, why assume by default that other people haven’t made serious mistakes in understanding.
Policy journalists: You can know better! You already have a default skepticism. If someone presented you with a pure observational comparison,