Pavel Atanasov writes:
I noticed your 2016 post on belief updating. Here is the key bit:
From the perspective of the judgment and decision making literature, the challenge is integrating new information at the appropriate rate: not so fast that your predictions jump up and down like a yo-yo (the fate of naive poll-watchers) and not so slow that you’re glued to your prior information (as happened with the prediction markets leading up to Brexit).
I wanted to share a manuscript, co-authored with Jens Witkowski, Lyle Ungar, Barbara Mellers, and Philip Tetlock, that addresses a closely related question: How do accurate forecasters update their predictions, given the twin threats of over- and under-reaction? Here’s the abstract:
We study the belief updating patterns of real-world forecasters and relate those patterns to forecaster accuracy. We distinguish three aspects of belief updating: frequency of updates, magnitude of updates, and each forecaster’s confirmation propensity (i.e., a forecaster’s tendency to restate her preceding forecast). Drawing on data from a four-year forecasting tournament that elicited over 400,000 probabilistic predictions on almost 500 geopolitical questions, we find that the most accurate forecasters make frequent, small updates, while low-skill forecasters are prone to make infrequent, large revisions or to confirm their initial judgments. Relating these findings to behavioral and psychometric measures, we find that high-frequency updaters tend to demonstrate deeper subject-matter knowledge and more open-mindedness, access more information, and improve their accuracy over time. Small-increment updaters tend to score higher in fluid intelligence and obtain their advantage from superior accuracy in their initial forecasts. Hence, frequent, small forecast revisions offer reliable signals of skill.
Slowness to update . . . that’s one of my favorite topics! Good to see work in this area.