Susan Gruber, Geneviève Lefebvre, Tibor Schuster, and Alexandre Piché write:
The ACIC 2019 Data Challenge is Live!
Datasets are available for download (no registration required) at https://sites.google.com/view/ACIC2019DataChallenge/data-challenge (bottom of the page).
Check out the FAQ at https://sites.google.com/view/ACIC2019DataChallenge/faq
The deadline for submitting results is April 15, 2019.The fourth Causal Inference Data Challenge is taking place as part of the 2019 Atlantic Causal Inference Conference (ACIC) to be held in Montreal, Canada
(https://www.mcgill.ca/epi-biostat-occh/news-events/atlantic-causal-inference-conference-2019). The data challenge focuses on computational methods of inferring causal effects from quasi-real world data. This year there are two tracks: low dimensional and high dimensional data. Participants will analyze 3200 datasets in either Track 1 or Track 2 to estimate marginal additive treatment effects and associated 95% confidence intervals. Entries will be evaluated with respect to bias, variance, mean squared error, and confidence interval coverage across a variety of data generating processes.
I’m not a big fan of 95% intervals, and I am aware of the general problems arising from this sort of competition: the problems in the contest are not necessarily similar to the problems to which a particular method might be applied. That said, Jennifer has assured me that she and others learned a lot from the results of previous competitions in this series, so on that basis I encourage all of you to take a look and check out this one.
What is the gold standard model they are comparing to in order to judge performance? Is this all done on computer generated data or something?
Yes,they state “Within each track 100 datasets have been drawn from 32 unique data generating processes (DPG)”. Andrew will like that they include varying levels of effect heterogeneity, but probably be less happy that the target is ATE regardless of effect heterogeneity. They also simulate under strong ignorability, and in real life the bias from the unmeasured is a much more serious problem than variations on how to condition on the covariates. Nevertheless, it’ll be a good resource for testing methods and software.