https://twitter.com/realDonaldTrump/status/1326673298692972544

We can all stop thinking about this now, it’s been figured out!!!

]]>Zhou, Carlos

If you look at Zhou’s code, I think we see the same dynamic that I am talking about, just in a different place. The final vote shares (panel c) systematically pools towards 0.5 compared to the vote share at any given point in time. As volatility increases, it does more relative pooling, along with greater time until election day.

Excellent! I feel like this has been very productive actually. Thanks for sharing code Zhou.

> Look at the nopaste I did at Chris more recently. That does the logit transform.

Thanks. My first guess was wrong and it’s still the case that changing the “volatility” parameter leaves winProb unchanged. The volatility of voteShare increases (but not as much, it just gets compressed into the extremes). The residual variance at some point (the uncertainty) is not just a function of the time remaining, it also depends on the value of voteShare. Higher residual variance (uncertainty about the outcome) corresponds to winProb values closer to 50%. In any case I won’t say that Taleb is not wrong, I don’t know exactly what does he claim.

]]>FWIW, I like the replicate() formulation over pnorm since it lets you drop in other specifications of the volatility if you wish and see if it makes any difference.

]]>Carlos:

Look at the nopaste I did at Chris more recently. That does the logit transform.

]]>Carlos, as of right now it is not explicit, at least not the way I’m interpreting it. Rather there’s a process that delivers win probabilities that exhibits volatility. I think we could specify a model for vote share and then add epistemic uncertainty – to mimic the real world situation with inference and forecasting- and get to the same endpoint. But I’m not convinced either way yet! What I am convinced is that this is very close to what Taleb is arguing we do somehow…

]]>Chris, is the vote share included in your model (or can it be derived somehow)?

I would expect the forecast to be 0 or 1 on the last day (when there is no uncertainty left), but it doesn’t seem to be the case.

]]>> That’s just an artifact of the random walk specification. It’s a sufficient rebuttal because it’s the same example Taleb uses

Your example may be relevant to Taleb’s example. I’ve never understood what Fig. 3 was supossed to show.

But the discussion was about what happens when the volatility of the vote share is high. Using Taleb’s terminology:

Y0 the observed estimated proportion of votes expressed in [0,1] at time t0.

s annualized volatility of Y, or uncertainty attending outcomes for Y in the remaining time until expiration.

Your simulation only makes sense if s*sqrt(T) is small (much less than 1).

> I think using the logit transform solves the problem though.

I see Chris Wilson has shared some code, I’ll look at it. My guess is that when a non-linear transformation is applied the generated winProb sequences are no longer independent of the parameter “volatility”. Changing the parameter in your simulation (keeping it low enough for the simulation to remain valid!) scales “dailyJumps” and leaves “winProb” unchanged. But your simulation is not applicable when the variance of the vote share is high.

One suggestion about the simulation code: doing

winProb = c(winProb, pnorm(-sum(dailyjumps), sd=volatility*sqrt(electioncycle-i), lower.tail=FALSE))

instead of

winProb = c(winProb, mean(sum(dailyjumps)- replicate(reruns,sum(rnorm(electioncycle – i)*volatility)) > 0))

is much faster, generates a precise forecast and makes more clear what’s going on.

]]>Hi Zhou, I think the blog ate my comment so re-posting. I think you are highlighting a crucial difference in what we are modeling here. My stochastic model is representing a process for delivering win probabilities, so think of this as vote share plus uncertainty in that vote share. Imagine a sequence of posterior distributions over vote share, such that for each day you can compute the probability that vote share > 0.5 (= win). Volatility in this process subsumes both volatility in the underlying quantity as well as epistemic uncertainty. So I’m not using the logit just to constrain vote share as a deterministic quantity to (0,1).

And I believe this is the key to what Taleb is trying to point out. What should the impact of any source of uncertainty be on our updating of win probabilities? Whether this is the right way of thinking about the problem I am not at all sure.

Zhou, another quick thought. With your fix to that line of code – so that it now does what I was trying to do lol – I think we can highlight the subtle distinction you are drawing with vote shares.

My approach is to suppose there is a stochastic process for delivering win probabilities. So this would represent movement in underlying vote intention AND uncertainty in that quantity. So yes, “if the win probability is more volatile, it’s more volatile”. I’m imagining a sequence of probabilistic predictions for vote share, each of which has a posterior distribution, and hence the ability to define the probability of exceeding 0.5.

This is what my logit_support process is doing basically. It is saying, use the inverse_logit to generate a probability for the result *if election were held on that day*. What does the future volatility in this quantity mean for how to update win probabilities?

This is subtly different than using the logit transform as you had originally suggested, simply as a way to constrain vote share without uncertainty. So you are totally correct there.

And I believe this difference explains what Taleb is trying to do, somehow. Whether it is the most useful way of looking at the problem is totally up for debate.

]]>Obama similarly generated large crowds with a religious fervor of “hope and change”. Intensity of support might augment median support to predict votes.

]]>Hi Zhou,

Aha thanks for this correction #logit_support[i] + sum(rnorm(electioncycle-i))*volatility#

The perils of late night coding after an exhausting week :)

I will look through more carefully later, but this does seem to change things…

Chris:

There’s an error in your code:

logit_support[i] + sum(rnorm(1)*(electioncycle-i)*volatility))

is incorrect.

I mean the sum basically does nothing, while the “electioncycle-i” needs to be square rooted.

logit_support[i] + sum(rnorm(electioncycle-i))*volatility

will be basically exactly the same as my code.

The second problem with your code is that you aren’t calculating win probability, you are modelling vote share. This is because your model directly uses the inv_logit(logit_support) as win probability, instead of checking whether this is > 1.

I mean, the point of the volatility in this discussion is that we are talking about how volatility in polling should feed through to the win probability, so in your formulation you’ve actually forced the win probability to be more volatile as a function of volatility. As a result you’re basically saying “if the win probability is more volatile, it’s more volatile”.

I consider this a fixed version:

Try changing the volatility for a fixed RNG seed. The vote share curves change but the win probabilities not at all.

]]>How to the big grocery chains make it? I used hear people say the had a 2-3 % margin.

]]>“It’s much better to have multiple polls …”

Thanks!

]]>Zhou, others

Here’s a simulation of a stochastic process on the logit scale and then we get probabilities with the usual inverse logit function. There are two probabilities here, one is the “day of” election forecast, and the other discounts based on future unrealized probability. The latter behaves as Taleb suggests, and as I agreed above – stronger pooling towards 0.5 with increased volatility or time to forecasted event. I believe this same mechanism is present in Andrew and Nate’s models with a drift term, although Andrew is obviously in best position to clarify that.

I am open to corrections on where this simulation is going wrong (if it is)

]]>@somebody :

>A statement like “the forecast is at 10% for Trump right now, but remember that Trump is an unpredictable candidate and Hilary is under investigation so that could change drastically” *is wrong*, and anticipating the investigation to have a large and unpredictable effect on public opinion *should nudge your forecast closer to 50%*.

Enh, this is perhaps pedantic but I think there’s justifiable reasons to make statements like this. What is going on in these situations is that the modeled volatility does not fully represent the modeller’s personal estimate of the volatility.

For instance, the volatility estimate might come from an aggregation of multiple elections in the past. Then, what happens when you are confronted with an election with (apparently) substantially higher or lower volatility? Well, you can throw out all the historical data, but that’s a Lot of your sample size and you could find yourself chasing a false signal into the weeds. You can go in with a plug in value of the volatility that represents your own personal posterior assessment, but that’ll attract the ire of people with very different priors. And so on.

So you can see that you can easily come to the situation where you end up with a model you don’t 100% believe in, and end up making statements like that. Heck, Andrew did the same in 2020.

]]>Carlos: That’s just an artifact of the random walk specification. It’s a sufficient rebuttal because it’s the same example Taleb uses – I think Taleb says that this is the only situation you need to look at, and I’m inclined to agree. But if you want to go more generally you would have to have a different definition of “volatility”. You wouldn’t be able to define volatility in terms of a fixed constant number, because as you identify, if the vote share is close to 0% or 100%, the volatility necessarily reduces. I think using the logit transform solves the problem though.

Chris:

>Zhou, I’d have to think about this some more, but here’s what I think you are perhaps missing. The relevant forecast is NOT *what would happen if election held today?*, but *what will happen on singular election day?* ahead. For the latter, high volatility spreads out where things “land” on that future date, and the amount of uncertainty should increase proportional to time remaining.

No, I think you misunderstand. The simulation code stores “what would happen if the election is held today” as the electoral state. It then generates and aggregates what will happen on a singular day ahead by monte-carlo, producing the win probability. That’s the sum(rnorm(electioncycle – i)*volatility) term that computes the remaining volatility in the days that remain.

]]>> What I’m simulating is the difference in votes between Biden and Trump, i.e. (vote_share-0.5)*some large constant. […] The simulation simulates this as the cumulative sum of random gaussian daily changes in voting intention (that’s the dailyjumps), which have the specified degree of volatility.

That makes sense only if that specified degree of volatility is low relative to that large constant (i.e. if the volatility of vote_share(t) that we’re talking about is low). Otherwise, there is a non-negilible possibility that the cumulative sum of random gaussian daily changes is big enough to reach vote_shares below 0% or over 100%.

In summary, that simulation doesn’t tell us anything about the case under discussion where the volatility of the vote share is high.

]]>Why not sort after the count instead of checking them twice? Anyway, still waiting for the explanation for this.

]]>somebody, yea I see that. I think that’s a good way to explain it. However, I guess what I’m saying is the relevant forecasting task is the future unrealized volatility. In that specific sense, I am agreeing with Taleb. It should tamper your probabilities toward 0.5 by some amount, but I prefer explicit model-based ways of specifying this rather than bluster. My understanding is that both Gelman and Silver’s models have drift terms to accomplish this, but this obviously misses the possibility of sharper swings driven by e.g. feedback cycles where GOP voters are blasted with media and social media coverage of the scary Democrats and their crushing mail-in vote advantage and are energized to turnout on election day, etc.

]]>Chris Wilson wrote

> Zhou, I’d have to think about this some more, but here’s what I think you are perhaps missing. The relevant forecast is NOT *what would happen if election held today?*, but *what will happen on singular election day?* ahead. For the latter, high volatility spreads out where things “land” on that future date, and the amount of uncertainty should increase proportional to time remaining.

Zhou isn’t missing that, these are correctly simulated probabilities for the final outcome conditional on intermediate values of the process. The key distinction between this and Taleb’s example is that Taleb is talking about “future unrealized volatility”. Conditional on public opinion being frozen at a current level, if you take future volatility in public opinion to infinity probability goes to 0.5. But before looking at any public opinion, you expect higher volatility to also push your intermediate surveys to more extreme values, so the volatilities cancel out and the conditional probabilities you expect to see are independent of volatility. Or, to put it another way, volatility makes a signal of a given magnitude less informative about the final outcome, but also causes you to see stronger signals.

]]>So I’m sticking with my assessment above and elsewhere :) High uncertainty –> 0.5, and time remaining should increase uncertainty. However, there is not way to judge a series of forecast probabilities and determine whether it is a martingale (an ill-posed question IMO), or whether intertemporal arbitrage is possible, etc. All that matters is whether they followed the Bayesian mechanics correctly. If you have a better model, or better data, or both, you can expect to make money betting on final outcome. Additionally, if you knew, for instance, the inner guts of the model and had a better way to forecast the covariates they are conditioning on, you might be able to game them in a series of bets assuming they are willing to trade in and out of their positions with you, *based on your superior knowledge of the future data*. You might call this “arbitrage” but that would be a fast and loose sense of the term.

]]>Zhou, I’d have to think about this some more, but here’s what I think you are perhaps missing. The relevant forecast is NOT *what would happen if election held today?*, but *what will happen on singular election day?* ahead. For the latter, high volatility spreads out where things “land” on that future date, and the amount of uncertainty should increase proportional to time remaining.

]]>Scientist:

I don’t know about Predictit, but according to Josh Miller, who knows about these things, it really was possible to make money on that California bet on Betfair, as they take the vig out of your winnings, not out of the total bet. So, yeah, it’s stupid money. Or money that people were throwing around for entertainment money that was sitting on the floor and nobody bothered to pick it up, or something like that.

]]>Predictit at least has a 5% withdrawal fee, so once costs are accounted for along with bid/ask spread, 3% on predictit at least is 0% for Trump. So what is going on here isn’t so much stupid money as a reflection of a market structure that makes simple interpretation of probabilities fail at the level of around 5%. Insomuch as it does reflect stupid money, it is people who think Trump has a 0% chance and don’t understand their trading costs. I expect similar structural reasons on other sites.

]]>> What does that mean?

What I mean is that if Biden is 5% up on Trump in the polls at a given moment in time, then under high volatility it’s true that a well calibrated win-probability forecast will give a number closer to 0.5 than otherwise.

OTOH, if the volatility is high, Biden is more likely to be 5, 10, 15% up (or behind) on Trump at any moment in time. This variability in the state of the election cancels out the variability in the future-event forecast, which means that just by looking at a graph of projected win-probabilities over time, that graph is not (to some degree of approximation) actually affected by the degree of variability.

>

> Anyway, I think that the volatility under discussion is the volatility of vote_share(t) and the binary event to forecast is vote_share(T)>0.5

>

> I don’t see how your simulation relates to that.

What I’m simulating is the difference in votes between Biden and Trump, i.e. (vote_share-0.5)*some large constant.

Regard the state of the election (interpreted as the vote shares if the election happens immediately) as this value. The simulation simulates this as the cumulative sum of random gaussian daily changes in voting intention (that’s the dailyjumps), which have the specified degree of volatility.

Winprob runs the simulation forward from each point in time to see what proportion of future instances does the sum exceed 0.

You can transform this simulation into a vote_share > 0.5 formulation quite easily. Do e.g. a logit transform.

]]>>> I don’t think anyone disagrees that as uncertainty/volatility goes to infinity, a forecast of a binary stabilizes exactly at 0.5,

> I disagree. It stabilises at exactly 0.5 *for a fixed ‘state’ of the election*.

What does that mean?

Anyway, I think that the volatility under discussion is the volatility of vote_share(t) and the binary event to forecast is vote_share(T)>0.5

I don’t see how your simulation relates to that.

]]>@Zhou Fang

Since we’re following Taleb’s argument, we’re using his terminology, in which he uses “volatility” to mean “future unrealized volatility” or “volatility in the remaining time to expiration.” So his statement is, as you say, only about a fixed state of the election and I think deliberately so. That may seem like a vacuous case, but I think it’s because he’s refuting a hypothetical person who thinks that high volatility justifies large and frequent movements in probabilities. He states this in the intro

> We observe stark errors among political scientists and forecasters, for instance with 1) assessors giving the candidate D. Trump between 0.1% and 3% chances of success , 2) jumps in the revisions of forecasts from 48% to 15%, both made while invoking uncertainty.

both of whom would be wrong, if they existed. A statement like “the forecast is at 10% for Trump right now, but remember that Trump is an unpredictable candidate and Hilary is under investigation so that could change drastically” *is wrong*, and anticipating the investigation to have a large and unpredictable effect on public opinion *should nudge your forecast closer to 50%*. But by not stating who he’s talking about, or including specific quotations, or describing an incorrect methodology for contrast makes it really hard to follow what he’s actually trying to say.

For the record, I do think people who think like the above exist because I know personally some people who have said those things. Not that any of them would actually read this paper, or like Taleb makes an argument that’d be plausibly convincing to them. I do get the impression that he prefers that people continue to be wrong so he can keep insulting them.

I have no clue what’s going on with that Mathematica graph, or what it’s trying to tell me.

]]>Zhou, great point! I was thinking too much about the static properties.

]]>> somebody, I don’t think anyone disagrees that as uncertainty/volatility goes to infinity, a forecast of a binary stabilizes exactly at 0.5,

I disagree. It stabilises at exactly 0.5 *for a fixed ‘state’ of the election*. However if we are making predictions *during* such an election cycle, high volatility implies also more extreme states of the electoral process. The overall effect is that theoretically there *is* no impact from volatility on the genuine win probabilities.

You can confirm this by a simple simulation model in R.

Taleb is flat out wrong.

]]>somebody:

I’ve looked into this before. Taleb is flat out wrong. The graphs in that paper are the results of a coding error.

I analysed this in this quora answer:

]]>> Why don’t pollster just release their raw data,

Why would they?

]]>Probably there’s confidentiality issues. If you have individual level ethnicity, income, age, and gender, rough geographic location, education level… that’s potentially enough to identify individuals in some cases.

]]>> (Actually, I was wondering about this. During the run up to this election, I noticed that the number of samples in the polls were really small: a few hundred or so. Was that a large enough sample, I wondered? My memory has it that the polls I read about in the Japanese press have much larger sample, multiple thousands. My hypothesis is that with the demand for frequent poling, the amount of money available to spend on each poll goes way down.)

It’s much better to have multiple polls with a few hundred or so samples, especially from different people, than one big one with several thousand. Actual sample size based uncertainty is much less important than bias in sampling and you only can get an assessment of that by repeating with different people/methodologies.

]]>David,

“People” are going to keep doing and talking about exit polls just like they’re going to keep trying to aggregate pre-election telephone polls and pretend they can tell us something. There is a huge viewership for all that sort of forecasting and “people” will continue with it until they are finally laughed out of the room after some future election.

]]>Its certainly not very informative. But then a huge tranche of people are operating on the basis that reality doesn’t really exist and whatever story is told loudly enough is the one that matters.

]]>+1

]]>Ah. I see. It’s like the banking crisis: you pool a really large number of really bad (largely fraudulent loans by local banks (note: these were NOT the “sub-prime” loans)) and tell investors (and yourselves) that the pooling bit magically makes the conglomerate a good investment and then are surprised when the scheme implodes.

(Actually, I was wondering about this. During the run up to this election, I noticed that the number of samples in the polls were really small: a few hundred or so. Was that a large enough sample, I wondered? My memory has it that the polls I read about in the Japanese press have much larger sample, multiple thousands. My hypothesis is that with the demand for frequent poling, the amount of money available to spend on each poll goes way down.)

OK, bashing people after the fact isn’t nice. Sorry.

Serious question: I’ve been noticing recently that people are trying to make conclusions from exit poling data. This seems seriously nuts, since we know that anyone who understands that Covid-19 isn’t a liberal scam voted early, and exit polls are meaningless this year.

But people are still talking seriously about exit polls. Is that as nuts as it seems?

]]>I’m blowing in from epidemic forecasting, where we have our own problems (you may have noticed a few).

Why don’t pollster just release their raw data, and let the modellers/statisticians have a go? In epidemic modelling there is *nothing* more annoying than serological surveys where they are reporting a “corrected” figure, reweighted by biases in the serological sample population relative to some background demographics.

The reason this is vexing is because infection risk is itself heavily biased, and part of the epi modelling job is to try and figure that out vis-a-vis real data not the headline figures of data where some medical statistician, who hasn’t read a stats paper since 1995, has already done their dummies guide to bias correction overlay which you then need to try and reverse unpick.

]]>Perhaps in our new post-truth society, the probability should eventually converge to the probability that a random person believes their candidate actually won the election.

]]>Congressional Republicans didn’t do great in 2018, but did much better in 2016 and 2020. In light of this, it seems to me that the Republican overperformance is best explained by Trump and not vice versa.

]]>somebody, I don’t think anyone disagrees that as uncertainty/volatility goes to infinity, a forecast of a binary stabilizes exactly at 0.5, and also that more time remaining until the forecasted event means more things can happen hence greater uncertainty. Taleb’s whole project here starts and ends with those observations, and takes a fairly long, confusing and mostly irrelevant detour through “intertemporal arbitrage”, martingality, and his option pricing model (that has yet to be applied in a case study or real world application AFAICT). After a long conversation with Aubrey Clayton and Dhruv Madeka, all participants agreed you cannot look at a time series of forecasts and determine *is this a martingale?* Aubrey even provided a super volatile simulated forecast series from one of Taleb’s option pricing models to prove the point. So, basically, this all comes down to the idea that with better information/model you can make money by making clever bets, but it is not guaranteed in an arbitrage sense in most any real world context, and you can only Dutch Book an incoherent bookie. Nothing much has been gained here :)

]]>That is what is great about science. Incompetence and fraud are treated the same if its done properly.

]]>I feel like we’ve gotten away from the original point, which is what is Taleb trying to say.

Taleb claims to be responding to the idea that high levels of future uncertainty mean the probability *should* jump around a lot, citing

> jumps in the revisions of forecasts from 48% to 15%…made while invoking uncertainty.

So I believe he’s responding specifically to some hypothetical person who is identifying uncertainty in future public opinion with the probability that some candidate wins the election. Identifying probabilities by intermediate polls + measurement error *would be incoherent*, and Taleb believed people were doing that. His paper shows that under a simple model mapping volatile public opinion to election outcomes, coherently updated probabilities go to stability at 0.5 as volatility goes to infinity, rather than causing each intermediate probability to be sampled uniformly from [0, 1]. He also strongly implies that Nate Silver is one of these hypothetical people who don’t understand the distinction.

It’s certainly possible to have a bad model and lose money, or have a coherent overconfident model that nonetheless has a low likelihood on historical data, and I agree that it’s impossible in general distinguish between a bad model and incoherence just by looking at forecast timeseries.

]]>> I know their model isn’t a coherent model of the election outcome

But you don’t know that.

The following model is coherent: “public opinion is very stable, each day stays the same with probability 99.99999% and moves with probability 0.000001% with the change given by some probability distribution.”

It’s not a very bad model but it’s not incoherent. The optimal forecast with that model is to align with the polls.

> What we want to do is avoid admitting the possibility of an expected-profitable system of betting.

This has nothing to do with Dutch books or intertemporal arbitrage. If some bookie offer bets for binary events always with even odds nobody would be able to make a Dutch book against them, intertemporal or not. But it would create expected-profitable systems of betting if you know anything at all about the probability of those events.

]]>> But there is no way to say for sure that any sequence of forecasts in (0 1) is not a martingale, it could really be the perfect forecast obtained with the true model.

I agree. A true model with correct updating can still produce a sinusoid or any other shape with non-zero probability. But it can certainly be suspicious, and very large periodic swings should be very unlikely.

]]>> And in general it will not produce a guaranteed profit (which is what the standard definitions of Dutch book and arbitrage are about).

I agree that the definitions of Dutch book and arbitrage here do not accord with the standard ones, but it is what it is. What we want to do is avoid admitting the possibility of an expected-profitable system of betting.

> Your argument seems to be about knowing that they have a bad model (and having a better model) to get an advantage in expectation.

It’s more that I know their model isn’t a coherent model of the election outcome at all, but rather a model of current public opinion. They aren’t reporting the probability that candidate A wins the election, but rather that candidate A would win under a hypothetical election today, and I wouldn’t expect that reported probability to update with coherent martingality like a proper bayesian process, because it isn’t. Hence Taleb’s claim that “real probabilities don’t move like that.” Nathaniel definitely already knew that current average + polling error wasn’t the true election outcome probability, hence the “nowcast” feature. He was already using a cubic time-to-election uncertainty term to heuristically pull probabilities towards 0.5. Was it not enough, or did it behave weirdly? I don’t know.

]]>By the way, I think we also agree that the probabilistic model used for forecasting should be consistent and a sequence of forecasts should be a martingale. But there is no way to say for sure that any sequence of forecasts in (0 1) is not a martingale, it could really be the perfect forecast obtained with the true model. And how could inconsistencies be exploited if they can only be detected (statistically) ex-post? You need additional assumptions to postulate that the exploits that would have worked in the past will work (in expectation, at least) in the future.

Of course if you have your own forecasts which you know are good (or at least better) you don’t have to find a way to exploit inconsistencies, should they exist. You directly profit from your superior knowledge (and it’s irrelevant whether the model used by your counterparty is consistent and the forecasts a martingale under that model).

]]>