Skip to content

The problem with p-hacking is not the “hacking,” it’s the “p”

Clifford Anderson-Bergman writes:

On CrossValidated, a discussion came up I thought you may be interested in. The quick summary of it is that a poster posed the question that isn’t Fisher’s advice to go get more data when results are statistical insignificant essentially endorsing p-hacking.

After a bit of a discussion that spanned an answer and another question/answer pair, a rather interesting concept came up. That is, at many times in real world analysis of data, an analyst who uses statistical significance is forced to make a choice between preserving type I error rates and minimizing type S/M errors. Unfortunately, many researchers will dogmatically think it is better to preserve type I error rates, despite the fact that type 1 errors are typically impossible and type S/M errors are very likely!

Anyways, I thought you might find the conflict of these two approaches interesting. If you’re curious about the discussion, here are the two posts:

I clicked the first link above, which started with this question:

Allegedly, a researcher once approached Fisher with ‘non-significant’ results, asking him what he should do, and Fisher said, ‘go get more data’.
From a Neyman-Pearson perspective, this is blatant p-hacking, but is there a use case where Fisher’s go-get-more-data approach makes sense?

My quick answer, and it’s not a joke, is that it’s not “p-hacking” if you never try to make a claim of statistical significance.

To put it another way: If your only goal as a scientist is to reject the null hypothesis of zero effect and zero systematic error, then, sure, you can just gather more data until you reject that hypothesis. But so what? There was never a good reason to take that hypothesis seriously in the first place.

The problem with p-hacking is not the “hacking,” it’s the “p.” Or, more precisely, the problem is null hypothesis significance testing, the practice of finding data which reject straw-man hypothesis B, and taking this as evidence in support of preferred model A.

The point I just made is also expressed at the second link above. I’m glad to see these ideas being spread in this way.

Leave a Reply