Jonathan Falk writes:
Many years ago, when I was a baby economist, a fight broke out in my firm between two economists. There was a question as to whether a particular change in the telecommunications laws had spurred productivity improvements or not. There a trend of x% per year in productivity improvements that had gone on for five years or so, and we were looking three years after the change, and the productivity improvements were still around 2% per year. Economist 1 said: “Nothing to see here. The productivity trend is unaffected.” Economist 2 said: “WTF? You think productivity changes come from nowhere? It’s the law change that allowed the last three years to happen.” The conversation got heated (WTF was the least of it) and Economist 2, and his whole group, were fired. It was really ugly.
I timidly approached the firm President and said: ‘Y’know, there is really no way to tell which one was right. There just isn’t enough data to tell, and so essentially it comes down to your priors about the causes of the trend.” He agreed, but said Economist 2 and his group had acted with disrespect toward Economist 1, and that’s why they were fired. He had no idea who was right.
The stuff you posted yesterday is a perfect example of that fight all over again. Looking at the estimated R panels by state, you could certainly not spot any actual impact of shelter-in-place orders. Everywhere R was falling before well before any stay-at-home mandate, often to 1 or below, as in NY, and the continued drops to 1 in places like MA don’t look any steeper than places before the orders. And just look at Florida and try and fit that in your model of governmental action.
At that point, all the commenters look at the same data and draw inferences based on nothing but their priors, as far as I can tell. What was the demonstration effect of Italy? Of Washington? Of NYC? Did schools closing before stay-at-home orders function as partial stay-at-home orders? Do people require stay-at-home orders at all? The answers to these questions will directly determine the effect of removing stay-at-home orders (really? you sure about that?) but even with multiple jurisdictions to help us tease out the effects, I suspect there just isn’t close to enough data to do so with any confidence. And that’s even assuming that the R-estimation model, which is absolutely necessary to move from uninterpretable disease and death curves to how-are-we-doing-against-the-disease curves, are perfect!
So sure, this is Stan multilevel modeling, and that’s great, but is there any way to tease out a stay-in-place order effect? And even if you could, do you have any confidence that the effect of relaxing it is symmetrically opposite? I love to answer questions with models, but this just looks too tough. Fortunately, everyone at the blog seems to be treating everyone else with respect, for some value of respect. So nobody has to get fired—yet.
I asked Falk if I could post this and he said, sure, but:
The 2% should be changed to x% to be parallel. (I think it was 2, but productivity changes are always around 2%.)
That’s news you can use!