A few months ago we wrote about the following project for prepublication review, as described by Ruben Arslan:
A colleague recently asked me to be a neutral arbiter on his Red Team challenge. He picked me because I was skeptical of his research plans at a conference and because I recently put out a bug bounty program for my blog, preprints, and publications (where people get paid if they find programming errors in my scientific code). . . . The Red Team approach is a bit different to my bounty program. Their challenge recruits five people who are given a $200 stipend to examine data, code, and manuscript. Each critical error they find yields a donation to charity, but it’s restricted to about a month of investigation. I [Arslan] have to arbitrate what is and isn’t critical (we set out some guidelines beforehand). . . . My own interest in this comes from my work as a reviewer and supervisor, where I often find errors, especially if people share their data cleaning scripts and not just their modelling scripts, but also from my own work. When I write software, I have some best practices to rely on and still make tons of mistakes. I’m trying to import these best practices to my scientific code. I’ve especially tried to come up with ways to improve after I recently corrected a published paper twice after someone found coding errors during a reanalysis . . .
They did the Red Team Review, and Arslan says:
It went well. 107 issues and some major problems among them (some nitpicking too).
Some details are here:
The Red Team Challenge (Part 1): Why I placed a bounty on my own research, by Nicholas Coles
The Red Team Challenge (Part 2): The Arbiter’s View, by Ruben Arslan
The Red Team Challenge (Part 3): Is it Feasible in Practice?, by Daniël Lakens and Leonid Tiokhin
This all looks good to me. I hope other people do this red team thing. I have sometimes paid people to find mistakes in my books, but it makes sense to think about these ideas more generally. If the authors are paying the reviewers, there’s a pleasant alignment of incentives, much different than the usual scientific publication process which has so many problems.
$200 isn’t much for a month of work. But I guess it beats the $0 per month plus benefits currently offered for per review.
It isn’t much for a day of work. The real reward is the privelege of reading the paper. Yes it is.
There’s a big potential for reputational gains in the professional network, too.
Cool. I’ll let my landlord share in the privile of reading the paper and tell the grocery store they can have some potential for reputational gains.
We have formed a company since I sent that email and are offering red teams as a service. We’re looking for clients of course, but also red teamers. Please fill out the form if that interests you.
Reminds me of the final level of the TOP guidelines from Nosek et al. (https://www.cos.io/initiatives/top-guidelines), which seems to imply that checking other people’s research will someday be an industry.
Meta-psychology follows the level 3 TOP guideline for analyses (i.e., full reproduction). They have designated editors who reproduce the analyses and note that it worked in the header of the article. Check the top right of any article: https://open.lnu.se/index.php/metapsychology/issue/view/164