Gabor Simonovits writes:
I thought you might be interested in this paper with Gabor Kezdi of U Michigan and Peter Kardos of Bloomfield College, about an online intervention reducing anti-Roma prejudice and far-right voting in Hungary through a role-playing game.
The paper is similar to some existing social psychology studies on perspective taking but we made an effort to improve on the credibility of the analysis by (1) using a relatively large sample (2) registering and following a pre-analysis plan (3) using pre-treatment measures to explore differential attrition and (4) estimating long term effects of the treatment. It got desk-rejected from PNAS and Psych Science but was just accepted for publication in APSR.
I have not had a chance to read the paper carefully. But, just speaking generally, I agree with Simonovits that: (1) a large sample can’t hurt, (2) preregistration makes this sort of result much more believable, (3) using pre-treatment variables can be crucial in getting enough precision to estimate what you care about, and (4) richer outcome measures can help a lot.
Also, whassup. No graphs??
Much to compliment here – the authors appear to have gone to great lengths to set out a plan, report what they found, and not engage in the numerous bad practices we frequently see. I have a basic question – not necessarily a criticism of this paper, as I’ve seen this in many other papers. For the main regression model results, the R-squares are all less than 0.1 (some even considerably lower). While I usually don’t spend much time looking at these, I am always uncomfortable when I see numbers that low – it suggests to me that there are so many omitted variables that almost anything could be happening. It would seem to take a very strong assumption that these myriad factors (which account for over 90% of the variation)are random with respect to the treatment being studied.
Should this be a concern? Like I’ve said, I’ve seen this before and authors never seem to even comment on it. In my own work, I am uncomfortable when I am able to account for so little variation in my response variables. Should I stop worrying about that?
I think in the case of experiments, goodness of fit (GOF) is less of a concern. Your argument, which is broadly fair, is that there is more room for unmeasured variables to confound the estimate of the coefficient of interest. But in the case of an experiment, the treatment is randomly assigned and therefore is orthogonal to all confounders in the limit. In particular, there is no reason to believe potential confounders will be more or less correlated in practice with the randomly assigned treatment depending on the GOF.
In terms of the size of GOF, would we really expect a one-off experimental treatment to explain more than 8% of the variance in standardized prejudice scores. If it explained 80% of the variance I would worry that their prejudice scores were not really measuring anything of interest but were just getting respondents to repeat the treatment back to the researcher.
If the sample is large enough and assignment truly random, then this statement is true. Otherwise it is an example typical of the often observed overconfidence in randomization as a solution to all confounds even in small samples.
From the paper:
“We registered our design in the EGAP repository (http://egap.org/design-registrations, ID: 20161128AA) before we started the data collection. All analyses specified in the pre-analysis plan (PAP) were implemented and are reported in the paper. We provide the full list of departures from the PAP in the Supporting Information (SI).”
Yes, this indeed seems to deserve use of the term “pre-registered”
http://egap.org/registration/2281
1) a clear link to the pre-registration information in the actual paper
2) the pre-registration information can be read by the reader
+ 2
I am puzzled as to what reason there could be to desk-reject this? Twice?
If I were an editor, then the abstract would certainly give me pause, especially this: “Hungary’s overtly
racist, far right party.” This does not seem an objective description of Jobbik, a party which just won 20% of the vote in Hungary. How can a party that won 20% of the vote be fairly described as “far right?”
It’s further than the center-right party. So, equivalently, the Green party might be described as far-left (as long as it’s not the largest left-wing party).
> It’s further than the center-right party.
So call it “right wing.” I don’t see how you can fairly describe something as “far right” if it wins 20% of the vote.
> the Green party might be described as far-left
I have never seen the Green Party described, in an academic article, as “far left.” Have you?
You know what would be a really good way to lessen anti-Roma prejudice? Somebody coming up with an intervention that would persuade Roma to behave better toward non-Roma.