This is reasonable, but a strawman.

Yes, 99% 100 days out under the election circumstances we’ve observed would surely look arbitrageable. Now if we’re talking Putin or Hussein’s election, not so much (if anything, it would’ve been arbitrageable in the other direction).

But 538 never predicted 99% at 100 days out. In fact, they never went above 85% for Clinton. And they never even put it as high as 80% at 100+ days out.

Does 2016’s periodicity seem unlikely for a non-arbitrageable model? Yes. Does it seem impossible? No, of course not. I don’t have the statistical tools in my toolbox to test the likelihood that a non-arbitrageable model could have produced 538’s prediction set — in fact, I don’t even know enough stats to say whether such a tool exists.

Taleb makes the similar strawman argument in this video: https://youtu.be/YRvPF__du9w , at one point discussing how a 0% prediction for one side would be clearly arbitrageable. Well, no kidding — but no one is making that sort of prediction.

]]>The forecasting problem: it’s the 8:00 on January 1st. I give you a widget with a LED, which is off. Every midnight, the state of the LED may change. I ask you to give a probabilistic forecast of the state of the light on January 31st at noon. Every morning, you will have an opportunity to check the current state and update your forecast.

On the last day the prediction will be easy, if it’s on (off) in the morning the probability that it’s on at noon is 100% (0%). In previous days you need some kind of model

One simple model is to assume that there is correlation in the sequence of states and it flips each night with probability x. If x is less than 0.5 we will forecast that it’s more likely than not to end in the current state and our confidence gets higher as we approach the final day.

The charts in https://imgur.com/a/4UbZWHN show the sequence of assigned probabilities produced by forecasters that use the model above with x=1%, 2% and 3% in five different scenarios. The predictions flip from one side to the other each time the state of the light changes. They end at 0 or 1, as there is no uncertainty left on the final day.

The models do not seem very good, as they would expect on average one flip or less over the month, not several. But even if we don’t know anything about the model used, from the sequence of predictions we can sometimes say that the predictions seem to move too much. Whatever the model, the sequence of predictions should be a martingale. The expected value of the sum of the square of daily increments does equal the difference between the initial and final values of p*(1-p). The final value is 0 here, the last prediction in the sequence is either 0 o 1.

The expected value of sum(delta^2) is 0.18 for the forecaster with the 1% model who gives the more confident forecasts. There is a clear bound on the maximum value that can happen with more than 5% probability (0.18*20). If we observe a value over 3.6 we can calculate a p-value<0.05, etc. (That sum is called "movement" in the Augenblick and Rabin paper mentioned in another comment, they mention that tighter bounds can be obtained but I've not really looked beyond the introduction yet.)

The charts were generated from sequences of states that changed at midnight with probability 25%. I took five realizations at random, didn't The dashed lines show the predictions of a forecaster that had used the right model.

This kind of test doesn't detect all the "bad" models but it detects "seemingly inconsistent" predictions (sequences with an unlikely high movement). A forecaster that gave a constant prediction of 50% (except on the final day) may not be good if the sequence of observed states is really informative but wouldn't be inconsistent. The movement would be 0.5^2, just as expected, whatever is the true data generating process.

]]>+ 2

]]>Exactly, guaranteed, as in it’s logically impossible for it to be otherwise, probability = 1. That seems like a much stronger statement than is actually warranted.

If he just said “by incorporating the patterns in variation in price I can come up with a trading strategy such that if my description of the pattern is reasonably good, and it appears to be, I will have a high probability of making money” then I think we could just all move on, because there’s nothing particularly controversial about that. More good/real information about the future = making money is widely understood.

]]>One last question, maybe… When you say

“Essentially, if the price of the binary option varies too much, a simple trading strategy of buying low and selling high is guaranteed to produce a profit”

do you really mean that when you buy “low” (whatever it means) you are *guaranteed* to have the opportunity to sell higher in the future?

]]>Please note – Ive stopped actively checking this page, unlikely to see responses

]]>Ahh I started this thread when a coworker sent it to me (with your tweets) with such hope of a truce. But, seems like its a false hand extended – specially after reading comments like “Quant finance apparatus isnt useful” (er, how would someone even validate something like this? Isnt the point here to bridge the two? Isnt the apparatus of martingales best developed in QF?)

And seeing your tweets Aubrey. Like I said, let me know via email once you’ve read and understood (google Dambis Dubins Schwartz etc) and are willing to engage productively. Im not sure you’re ready to reduce the temperature and actually try to have a conversation – willing to engage when you are!

]]>+1 to Carlos’ comments, and thanks all for the discussion. Every time we follow a point through, this whole thing starts to really seem like a Motte and Bailey situation. The Motte is the intuition that high uncertainty -> 50% (in a binary setting) and a couple other uncontroversial assertions; the Bailey is the ‘no-arbitrage pricing’ machinery. The unconvincing link is how their argument doesn’t just come down to a variant of, ‘an agent with better information/models will be able to generate positive returns against a forecaster with worse information/models’. Everyone agrees that correct Bayesian updating is necessary, and thus that a forecaster’s forecasts must trivially be martingales. To my understanding, that is actually just De Finetti’s derivation for subjective probability – the axioms are the only way to not get Dutch-booked!

]]>I think the harder case is something like my asteroid hitting the earth… Conditional on the data today we can project forward the trajectory and see which states will be wiped out… but there are many bits of mass we can’t see that perturb the orbit so tomorrow the trajectory will be different… on the other hand we may not have a model for the perturbations… someone with better information may have such a model and they can make money by betting against us, but they can’t make *guaranteed* money.

in the context of Andrews model the state level polls play the role of the asteroid. Sure someone may say “hey the polls are highly variable, this person is always updating on them without projecting their future variability… I can make money off this… Perhaps true. But you can’t make GUARANTEED money unless you can guarantee something about the future of the polls.

not all money making on the basis of better information is arbitrage

]]>> framing things in terms of option prices and applying the machinery of no-arbitrage pricing really adds nothing of substance to forecasting

I agree.

“Thus we are able to show that when there is a high uncertainty about the final outcome, 1) indeed, the arbitrage value of the forecast (as a binary option) gets closer to 50% and 2) the estimate should not undergo large changes even if polls or other bases show significant variation.”

Sure, if there is high uncertainty about the final outcome the probability is close to 50% (because the uncertainty is high) and changes are not large (because the probability is close to 50%).

“a binary option reveals more about uncertainty than about the true estimation, a result well known to traders”

I think non-traders also understand that when there is high uncertainty about a binary outcome the probability is close to 50%. (Disclaimer: I took a course on stochastic processes with applications to finance in grad school.)

]]>About assuming the existence of a probability measure: yeah, just wanted to include that for completeness. Taleb talks about it as though it’s something new, but the argument goes back to de Finetti. I think it just underscores my point above that framing things in terms of option prices and applying the machinery of no-arbitrage pricing really adds nothing of substance to forecasting, since probabilistic forecasts already satisfy all the desired properties those prices are supposed to have anyway. The hard direction is entirely the other way (that is, starting with prices and seeing them as probabilities): if there’s no arbitrage there must be an equivalent martingale measure, intertemporal arbitrage in particular implies prices satisfy Bayes’ rule, etc. But conditional expectation with respect to a probability measure automatically produces martingales and satisfies Bayes’ rule. So we can dismiss the arbitrage concerns out of hand and get back to the real questions of how to come up with probability models based on data and how to evaluate them against outcomes.

]]>> It’s possible to show that any combination of prices for the three contracts — event A, event B given A, and event A and B — has to satisfy Bayes’ rule in order not to admit arbitrage.

Sure, that makes sense. I guess I was already assuming a Bayesian probability measure.

Thanks for the clarifications.

As far as your “Obviously, all this talk about true probabilities and frequencies runs afoul of subjective probability assignments based on information, but the usual setup in this domain is that there is some true underlying “physical” probability measure, maybe just representing the consensus of the market.”

I think there is no problem with Bayesian states of information. It’s always the case that a person with *better* predictions about the future can make money from someone with worse predictions, but it’s not arbitrageable so long as the Bayesian isn’t assigning positive probability to things that logically can’t happen. If that occurs, then the counter-party can know for sure they will make money by betting the thing *wont* happen.

The worst case of course is where the outcomes are perfectly predictable (ie. a completely rigged game). When that’s the case then the rigger knows that certain outcomes have zero probability, and can arbitrage the Bayesian.

]]>The processes being considered here are all bounded on a finite interval, so no need for the no-free-lunch version. The ordinary kind of arbitrage and Fundamental Theorem applies.

I should have mentioned, intertemporal arbitrage is definitely possible, if the forecaster is also asked for conditional probability forecasts (e.g. “What will your ultimate win probability be if you win in the next spin?) It’s possible to show that any combination of prices for the three contracts — event A, event B given A, and event A and B — has to satisfy Bayes’ rule in order not to admit arbitrage.

In practice, though, we never get those forecasts from say 538. Instead we just get all the conditional probability updates based on whatever happened, without the missing pieces to see if it was consistent with some Bayesian updating for some probability measure.

]]>A useful nontechnical description of some of the math, specifically the no-free-lunch notion:

]]>This is helpful. Clearly, if someone assigns nonzero probability to an impossible state of affairs (say, somehow “winning twice on one spin” as you mention, or let’s say trading in a stock below a price that the rules of the market prohibit for example) then it’s risk free to bet against that and true arbitrage becomes possible.

in any other situation, the question of “who is right” is not decidable ahead of time, and so we always have a risk of loss, and hence no arbitrage.

However I wonder if the notion of “arbitrage” being used by Dhruv is a “free lunch with vanishing risk” (https://en.wikipedia.org/wiki/No_free_lunch_with_vanishing_risk)

I don’t understand the technical definition given there exactly.

]]>I said “a wrong model (which produces a series of predictions which is not a true martingale)” but it’s not a necessary condition. A sequence of constant predictions would be a martingale and that doesn’t make a model right. Also, the idea that “not being a true martingale” indicates that a model is wrong has to be understood in a statistical sense. The model can only be proved wrong when things that are supposed to be impossible are observed.

]]>Thanks for your reply. I think I agree: a wrong model (which produces a series of predictions which is not a true martingale) is not guaranteed to lose in general when confronted with a good model (based on the true probabilities) but only when it’s so bad that it bets on impossible outcomes. (I’m not sure I agree with the last paragraph but anyway it seems unrelated to the preceding discussion.)

]]>(Sorry, formatting got messed up here, I put brackets around B to represent the quadratic variation but it turned the text bold. Hopefully it’s clear what I meant.)

]]>Carlos:

Thank you for this comment. I hope the new additions in the thread above are helpful. Per your (and others’) request, here’s another simple example to illustrate some of the key concepts here as I understand them:

Suppose someone is going to play 3 spins of roulette and bet on Red each time for 1 unit per spin. Before each spin he’s also going to forecast the probability of finishing the game above where he started. Initially he computes the probability to be p^3 + 3*p^2*(1-p), where p is the chance of winning each spin. Call this the forecast probability at time 0, F_0.

Then, supposing he wins the first spin, his new forecast probability is p^2 + 2*p*(1-p), since he needs to win at least 1 out of the following 2. Call this the forecast at time 1, F_1(W). If he loses the first spin his updated forecast is p^2, because he needs to win both of the remaining times. Call this F_1(L).

It happens then that F_0 = p*F_1(W) + (1-p)*F_1(L), and this pattern holds in general, meaning the forecast probabilities are a martingale. However, this is true no matter what probability p he uses. According to his probability measure, the forecast is always a martingale, by the tower property of conditional expectation.

Now, imagine an investor sees this and wants to bet on whether the forecasts are going to go up or down. Maybe the investor knows the real probability of winning roulette. So maybe the gambler/forecaster thinks his forecasts are going to behave like a martingale, but the investor knows otherwise. They can earn a *likely* profit by betting the right direction — say the gambler thought the game was 50/50, so his realized forecast win probabilities are going to steadily drift downwards. The investor would sell the initial forecast short and buy it later when it’s low. So, what I said above is that strategy may earn a likely profit but it’s not arbitrage in the strict sense of a guaranteed chance of profit with no possibility of loss. If the forecaster’s probabilities are equivalent to the investor’s, meaning they agree on what events are possible just with different probabilities, there’s always some probability the investor will lose money. The Fundamental Theorem of Asset Pricing guarantees this under general conditions.

But if, say, the player thought there was some probability of winning 2 units on the final spin, say, when that really is impossible, the forecasts he came up with would be a martingale but with respect to a probability measure that’s not equivalent to the “true” measure. This is why I said it’s not actually a requirement that the forecasts “actually are” martingales to be arbitrage-free. However, they should be martingales if the probabilities they embed are properly calibrated, meaning they agree with the frequencies with which the events occur.

Obviously, all this talk about true probabilities and frequencies runs afoul of subjective probability assignments based on information, but the usual setup in this domain is that there is some true underlying “physical” probability measure, maybe just representing the consensus of the market. There’s also an assumed filtration that represents the “information” available to all market participants, so they’re all able, say, to assign probabilities to the same events at the the same time.

Hope that helps.

]]>This is actually, despite all the venom, surprisingly helpful. Believe it or not, I think we’re making progress toward uncovering the real issue here — which is that you and I use the word “arbitrage” to mean different things — and I’m beginning to understand your argument. Honestly, thank you for the explanations. Maybe we can call a truce and discuss further with the temperature lowered a bit?

Here is your “volatility test” argument as I now understand it. Tell me if I’ve got this right: Treating the forecast probabilities as prices, the portfolio that holds 2(B_0 – B_t) units of the “asset,” continuously rebalanced, has zero initial cost and final value given by **_T – (B_T – B_0)^2, where **** is the quadratic variation (that’s just Ito). And since the process is bounded in [0,1] this means the total quadratic variation ****_T can’t be greater than 1 with probability 1, otherwise this portfolio certainly has a positive value and earns a guaranteed profit. So in the end it’s not actually about the process being a martingale, it is about it being arbitrage-free. Is that about it?**

The problem I have then remains that it requires you to know ahead of time that the forecast probabilities were going to be that variable. In order to be strict arbitrage, you’d need to know the QV would cross the required threshold with probability 1. For an arithmetic Brownian motion as in your paper, that’s maybe ok (although I think you run into problems with the convergence of the QV process and the assumption that B is bounded — basically you’re saying a BM can’t be a.s. bounded, which is obvious — probably you want B_t to be a doubly reflected BM instead or something), but in general the QV_T is going to have some probability distribution that includes some probability of being greater than the threshold and some below. Observing a single sample from the process isn’t going to establish whether it was “arbitrageable,” even if the path is so variable that the particular realized QV_T is > 1.

The data I provided is not insulting, I promise. I think we can all benefit from a concrete worked example, as mentioned below in reply to Carlos as well. In that spirit I’ve calculated the portfolio values using your strategy, and I’ll also tell you where the forecasts came from: this is a sample path from Taleb’s binomial option pricing model, and not at all an atypical one. As you’ll see it has a final QV around 0.89, but many of the random samples have QV this high. The 538 forecasts in 2016 were nowhere near this variable, so I don’t know how can you look at one and say it’s evidence of arbitrage while the other is a rigorously updated arbitrage-free model. As I say in my note, even for a simple random walk model it’s easy to produce forecast probabilities that varied more than 538’s, and by construction they’re necessarily martingales.

**But anyway, you’re not actually talking about arbitrage in that strict sense. You’re defining arbitrage in some looser sense of earning higher average returns than should be reasonably attainable, like “alpha.” So this is like saying “this fund strategy outpaced the market by a lot this year, therefore this is evidence of some inefficiency in the market.” The Foster-Stine-Young argument only appears to apply using this definition as well (again, this is not referenced in your paper, so I’m playing catch-up here). With the particular strategy above I’m not sure how you’d use their test here, since the initial portfolio value is 0, how do compute the return? And what’s the comparable market index? Maybe you can fill in the rest, if this is now the real test you want to use. What’s the result of applying the same test to the 538 data, and how do you assess that their forecasts were wrong?**

+1. This whole conversation would be easier to follow with a fully worked example – from Dhruv, or Dhruv + Aubrey – starting from unambiguous definitions of the relevant terms and demonstrating careful workflow.

]]>Thanks Carlos. this was a very useful illustration of some of the issues with the whole conversation.

To me intertemporal arbitrage strictly speaking is not possible. At any moment an asteroid could strike your house killing you before you close out your position causing you to lose money, so the risk of loss is never zero intertemporal my.

true arbitrage must always be between two different people at essentially the same time (buy low from Joe, sell high to Fred).

however statistical intertemporal moneymaking is a real thing… there are strategies we can discuss where having better information results in a very high probability of making money. are quants calling this arbitrage?

]]>> Just to be clear, here is the complete description of the “volatility test” from Taleb and Madeka (2019):

> “Essentially, if the price of the binary option varies too much, a simple trading strategy of buying low and selling high is guaranteed to produce a profit.”

I also find a bit confusing what’s being claimed. The term “guaranteed” in that quote suggests it’s about arbitrage in the sense you mention of “zero initial investment, no probability of loss, and positive probability of profit” (as do the references to “arbitrage-free” and “arbitrage pricing” in that paper or the original one from Taleb).

However, the “simple trading strategy of buying low and selling high” is not about replicating portfolios if it means buying now and selling later. This interpretation is unambiguous in the original paper:

intertemporal arbitrage is created, by “buying” and “selling” from the assessor

someone can “buy” from the forecaster then “sell” back to him, generating a positive expected “return”

A “high” price can be “shorted” by the arbitrageur, a “low” price can be “bought”, and so on repeatedly.

If it’s about trading with the forecaster at different times to generate a positive expected return it’s not guaranteed. And what’s the relevance in that context of arbitrage-free pricing using a risk-neutral measure?

If it’s about true arbitrage, where the arbitrageur can trade with the forecaster and with other market participants zero-investment portfolio with a guaranteed non-negative payoff, this is unrelated to the forecasts being a martingale or the forecaster being somehow wrong.

Imagine the forecaster has a perfect insight and produces a sequence of forecasts that passes all the tests for real-world martingality. He thinks, and he’s right, that the true price of the binary option is 0.52. However, the market/risk-neutral price is 0.48. The market-implied probability is lower than the real-world probability. The forecast is right.

Arbitrage is possible even thought the forecaster is accurate. The forecaster will be happy to buy at 0.50 (for a expected profit of 0.02) and the arbitrageur will be happy to sell at 0.50 (for a sure profit of 0.02, as he can just buy it from someone else at 0.48). The arbitrageur is not making his profit at the expense of the forecaster.

Conversely, if this was a bad forecaster with a bad model and a sequence of predictions failing the martingality tests who valued the option at 0.48, in line with the market price, arbitrage wouldn’t be possible. (But someone with a better prediction may make money, on expectation, trading with the forecaster.)

]]>Aubrey, thanks for your explanation, and for the definition of Arbitrage that you are using. this is helpful. I have only one question. if there should be zero probability of loss and positive probability of profit… I only have the question of according to whose Bayesian probability measure?

When a trader gets an inside tip that a certain event seriously affecting the value of the stock will be announced tomorrow for sure. He assigns 100% probability to a lower close tomorrow and shorts the stock today… he believes an arbitrage opportunity occurred.

when tomorrow the news is announced and also the government announces a big bailout of the company… the stock price rises slightly…

it’s an admissable state of Bayesian information to believe a crash is a sure thing… but the real world doesn’t have to play along. a-priori there is no way to disprove the incorrectness of the sure thing, a-posteriori it is too late.

For a Bayesian, it seems that we can only ever talk about whether our information and assumptions lead us to believe arbitrage is possible, not whether it “truly in fact is possible”

]]>Actually, lets tie this out since you seem to be more interested in twitter style dialogue than in understanding – check the P&L on the trade described above and in our response (you’ll have to read further than where you seem to have stopped given your previous incorrect response – hold proportional to deviation!) and apply Foster and Stine to the resulting P&L. r_f would be 0 here to adopt the notation of their paper!

Let me know if you actually decide to read! You don’t seem to have scrolled past page 1 of our response and given your lack of engagement with the other papers, Im guessing you haven’t opened (or perhaps understood) them yet.

Good luck!

Dhruv

]]>No I’m not willing to engage what seems like an insulting link – I haven’t really even looked at it (seeing your words and the responses to it).

I described the trade above – feel free to run it if the link wasn’t a joke. tell me the alpha it generates, and we can go from there.

For context, the trades are given *literally* a sentence after that. (in scrolling order): “The stochastic integral

…. can be replicated at zero cost, indicating that the value of … bounded by the maximum value of the square difference on the right hand side of the equation”

The trade is obvious from that, its given by the stochastic integral! What is missing here for you! I described it above too! Its a mean reversion trade that “replicates” variance (also holds if vol isnt necessarily constant – it replicates the total QV \int^T_t \sigma^2_u du), we even describe it in the paper!

Thanks for the description – but when I say that the SVI smile is not arbitrage free, I think you and I mean different things. This is what Im willing to acknowledge is the language difference. It means *if* you know that the trader is using an SVI model then it would fit into your description. This is not how quants use it when they say, “the model is not arbitrage free.”

To the second point, the outputs must be a martingale. So yes, tomorrow you or Nate could adopt the betting market and according to Aug and Rabin – it wouldn’t work. Thats not the point we’re making – and the language difference hits here again. I was willing to acknowledge that – but that doesn’t diffuse the point that bounded martingales cannot vary excessively.

I was willing to make the concession that this is a different language – and thats the disconnect. But now youre talking about words “you dont use the word quadratic variation” and you have now said what amounts to once again, not reading any of the papers mentioned above – do you agree Foster and Stine is a single path result? Do you also think A and R’s result is “magical”??

]]>So I take it you’re still not willing/able to show how your test works with actual data? Hint: you may recognize the forecasts I provided if you look at them the right way.

Just to be clear, here is the complete description of the “volatility test” from Taleb and Madeka (2019):

“Essentially, if the price of the binary option varies too much, a simple trading strategy of buying low and selling high is guaranteed to produce a profit.”

(for which a reference is given to Dupire’s notes for the final exam in his class at NYU in spring 2019. Such a handy reference!)

If you really do want to have a productive discussion, I’d like you to please specify exactly what constitutes varying “too much,” what the trading strategy is to take advantage, and how you know that it is guaranteed to produce a profit. Then, please apply that test to the sample data I provided and/or some historical forecast data of your choosing. Until you do that, I agree this discussion is mostly useless and I don’t think we’re going to make any progress here.

I think it *is* entirely possible that you and I are using the word “arbitrage” differently. I mean arbitrage in the usual quant finance sense of, e.g., Shreve (2004): an arbitrage is an admissible portfolio with zero initial investment, no probability of loss, and positive probability of profit. This does not, in particular, mean that forecasts have to be martingales to be arbitrage-free (nor does it preclude mean-reversion), despite what you and Taleb seem to be hung up on. The Fundamental Theorem of Asset Pricing says that arbitrage is impossible iff there’s an *equivalent probability measure* under which (discounted) prices are martingales. My point (1) in the initial comment above is that if a set of forecasts is performed using Bayesian updating, meaning that they represent the conditional probability of some future outcome at time T with respect to a filtration (meaning some E[I(T)|F_t] where I(T) is the indicator function of the event at time T), then those forecasts are automatically martingales with respect to the probability measure used to construct the forecasts, i.e., E[E[I(T) | F_t] | F_s] = E[I(T) | F_s] by the tower property. So, if you want to claim that some forecasts violated the no-arbitrage condition, you’d need to show they were constructed using a probability measure that’s not equivalent to yours/the market’s. Meaning this whole test of martingality is an unnecessary tangent to begin with, but go ahead and prove me wrong.

By the way, for Daniel, Chris and others feeling confused, this is, I think, a partial resolution to the thought-experiments provided upthread here. There are certainly arbitrage opportunities if one party has information making an outcome certain when the other does not; this means their probability measures are not equivalent (differ on null sets). But unless the profit is certain, there’s no arbitrage. The probabilities used to construct forecasts can be completely personal to the forecaster, and whether this produces forecasts that “actually are” martingales is immaterial. For example, the fortunes of a person betting on a casino game with negative expected value don’t represent arbitrage, even though it seems like “surely” you could profit by betting against them. In any finite time, your profit is highly probable (according to realized frequencies/the “physical measure”) but not guaranteed.

]]>> you can’t test a sequence of predictions to see if they are a martingale

However, can do a test to see if these predictions come from a model of the world that looks compatible with observed reality, in the same way that you can test a set of predictions to see how accurate they are in aggregate. For independent forecasts, there are statistical measures that can be used to check if the model is well calibrated.

For a sequence of predictions, we know it is expected to behave in some way if the model is right. We can do a statistical test to check how well the sequence of predictions behaves. For example, if my prediction jumps every few days from 1% to 99% and back it may be a martingale when expectations are calculated using my model but my model looks wrong and doesn’t seem to reflect real-world expectations.

]]>Thanks everyone – Ill try one more post. I was willing to acknowledge that how quants use the word “arbitrage-free” is different from how statisticians use it (happy to go into this but this is getting tiring). The difference in language was the benefit of the doubt I was willing to give Clayton – but he doesnt seem to have really understood that. I tried to explain this as a ex-post test here, including references for other *single-path* tests (Foster and Stine is *explicitly* a single path test for drift). Its *tougher* to reject on a single path (for ex. Foster and Stine need *twenty fold* beat of the market on a single managers returns), but not “magical”. Im beginning to think Clayton has largely missed the point – and wont engage because of that. Thats fine! But Im with Andrew that twitter is largely useless, and Claytons responses have resembled that more than usefulness – I would *strongly* encourage you to read the Rabin paper, Foster and Stine and our response to Clayton.

The filtration does not change the measure. The triple is defined (\omega, F, P) for a reason. The output of the model has to be a martingale – this can be achieved in many ways (Nassims, Nate seems to have cubic stochastic vol). This is not a question of the measure you’re generating from.

Clayton incorrectly seems to think a single path does not have a test. He may not like our volatility bound, maybe he prefers Foster and Stine. Corollary here on Page 7: https://core.ac.uk/download/pdf/132271344.pdf

“Corollary. A manager’s performance over a given period of time allows us to infer that

he can beat the market with 95% confidence provided his portfolio grows by a factor of at

least twenty‐fold relative to the total growth of the market portfolio over the same period. ”

This is a *single path* test for drift. There are tests for autocorrelation, quadratic var/variance. (The test is in this paper, run a trade proportional to deviation from the initial point (or the market) and see the money it generates. Thats a test. Do you get significant alpha from this trade? But again this is getting absurd)

If you prefer the Bayesian interpretation of uncertainty reduction: http://faculty.haas.berkeley.edu/ned/AugenblickRabin_MovementUncertainty.pdf

yes, an unbounded martingale can have an arbitrary path with an arbitrary probability. A bounded martingale cannot.

@Carlos – yes you might be unlucky. This is what p-values and rejection regions are for.

]]>I’ve not followed the discussion closely but the all the talk about arbitraging seems to bring more confusion than insight.

The martingale property comes from the basic fact that when you apply your probabilistic model today’s prediction for some future event is the average of tomorrow’s predictions over all the possible scenarios weighted according to their probability according to the model.

You can calculate the distribution of some statistics (like the volatility in the sequence of predictions) according to the model and use a statistical test to see if it looks as expected. If it doesn’t it may suggest that your model is wrong but you may also be unlucky.

Say for example that we predict the number of heads at the end of a sequence of ten flips. If our model is that the coin is fair, we would start predicting 5. If we get four heads in a row, our prediction would climb to 7 at that point. Such a big change in our prediction was unexpected. If we get four tails in a row then the prediction will drop again to 5. The was also unexpected. But it’s not a proof that our model is wrong (but it can strongly suggest it). It doesn’t necessarily create an arbitraging opportunity, no matters how we define it. Even statistical arbitrage depends on our model being wrong.

By the way, having the right model doesn’t mean we don’t open arbitrage opportunities for others. I could have a perfect model predicting exchange rates and someone else could make consistent profits taking my bets and hedging them on the market.

]]>Dhruv, I think the contentious part at a Bayesian blog is the clear underlying assumption that there is only one “true” model and everyone knows what it is.

the model y[i+1] = y[i] + Normal(0,1) is clearly a Martingale, yet if someone has information about what specific draw will come out of the Normal(0,1) process, they will arbitrage the heck out of you.

The concept “martingality = non arbitrageable” can only hold if you assume something like “all participants have the same state of information about the future”

Bayesians don’t believe that the world really actually is a RNG, the Martingality is located *inside their head* as a description of how much they know, not *in the world* as a description of the fundamental unpredictability of a process.

]]>Dhruv, I think it’s great that you are willing to participate on this board. One of the things that I most appreciate about Gelman’s blog is that I get to learn from some really smart people, especially when they disagree!

Anyhow, from my point of view, the two most important objections to your paper are this:

1. The point that Lakeland, myself and others have been making that the property of martingality/arbitrage-ability, in this context, is totally conditional on your model (including the goal of the model), and your set of information. Daniel has provided several colorful scenarios whereby a trivially martingale set of forecasts could be arbitraged by an agent with superior information. The property does not inhere only in a realized set of forecast probabilities!

2. With that in mind, you and Aubrey seem to be talking past each other a bit about the issue of bounds. Ignoring the philosophical objection in #1, is there or is there not a specific quantitative test for martingality/arbitrage-ability for a realized time series of forecast probabilities in your view? (This is different than the general assertion that such bounds must exist). If yes, could you walk us through its application in Aubrey’s simulated fake time series of forecast probabilities? If you are only arguing that such bounds must exist in general, but there is no specific test, then how does this not just come down to eye-balling such a time series and going on intuition whether or not it has the properties you would like to see?

Ahh, it’s rare for me to say this – but it’s a deep shame you seemed to have largely missed the argument. I had no idea that saying that there are events that occur with diminishing probability for martingales was so contentious. Good luck! I hope we can have a more productive discussion when you actually read the paper and the bounds.

]]>Yes, the famous “arbitrage by eyeball” test :). Surely there’s more to the method than just that, though, right? Surely these assessments are based on some precise criteria and not just on Taleb’s gut feel about what a martingale “looks like.”

]]>Unrigorous updating, clearly deviating too much from 0.5 ;)

]]>Love this

]]>But that sinusoidal pattern happened one time in one election. It did not happen in say, 2008:

https://revolution-computing.typepad.com/.a/6a010534b1db25970b017ee4d81fe1970d-600wi

or 2012

https://delong.typepad.com/.a/6a00e551f080038834017c32eedc04970b-pi

And it isn’t happening this year.

It seems an extremely tenuous argument to claim that the swinginess of 2016 was a predictable thing, especially when making forecasts early on.

]]>Ok, I concede. You and your co-author have a secret test for martingality (which according to you is equivalent to being arbitrage-free), involving the quadratic variation in some yet-unspecified way, that you can use to determine if a set of forecasts crossed the “arbitrage boundary,” despite the fact that you can’t reveal where that boundary is. Can you help me out, though? I have this set of forecast probabilities and I can’t determine whether they’re arbitrageable. Is this a martingale?

https://docs.google.com/spreadsheets/d/1u8tn_PULyZ9UotNPmGluzLTFj8sf1ZB0TnVQdXBmoDg/edit?usp=sharing

]]>To further add to the evidence that it isnt wise to dismiss the martingality analysis (despite Aubreys er, “claims”). Andrew et al’s paper cited this work – which frames the qv bounds for a [0,1] prediction as uncertainty reduction.

I enjoyed this one – different language, same result

http://faculty.haas.berkeley.edu/ned/AugenblickRabin_MovementUncertainty.pdf

]]>After more analyses, I think the stated probability of Trump winning would be changed by using longer tails to calibrate state predictions. However, the analysis methods that I used are hacky, so maybe better work from your team would be somewhat different.

Here’s what I did:

I logit-transformed all state result simulations to work on a scale where they are normal.

For each combination of state and previous election year, I replaced the state’s normal distribution of predictions with a shifted-and-scaled t distribution of the same mean and sd, as well as high degrees of freedom (DoF). Since the t approaches the normal distribution for high DoF, this left model predictions unchanged. I then reduced DoF until state tail predictions appeared generally calibrated. That happened for about 3.5 DoF.

For 2020, I took the simulated outcomes matrix (electoral_college_simulations.csv: 40 000 rows of draws x 51 columns of states). For each entry in the matrix, I computed its CDF value from the state’s normal distribution of outcomes, then replaced it with the same percentile from the shifted-and-scaled t distribution of 3.5 DoF. For example, if the matrix entry was 1.96 and the state’s 2020 simulations had a standard normal distribution, then I replaced the matrix entry with the 97.5th percentile of a t distribution with 3.5 DoF, which is 2.94. This procedure preserves mean predictions for each state and the Spearman correlations between states, but not Pearson correlations between states.

Altogether, these steps increased Trump’s 2020 win probability by 50% — from 8% to 12%.

My methods are hacks and will be different from the correct approach of working t distributions directly into the stan model, which I still can’t get running. But I think my results should be similar to the correct approach because non-tail parts of predictions shouldn’t change much between normal and t distributed predictions. The end message is that a DoF low enough to calibrate state results is likely to substantially increase Trump’s stated win probability.

I also learned a useful lesson: As Trump’s chances reduce, the assigned Trump win probabilities become increasingly different for narrow- vs. wide-tailed models. This may explain why you did not see changes when trying wide tails in the past (Trump’s win probability was higher then). Moreover, if Trump’s chances continue to reduce, then using normal vs. wide tails will produce larger differences in Trump’s win probability in the future. So wide tails may become necessary even if you don’t think they matter now.

I’m happy to answer any questions here or at the submitted email. Thanks.

]]>Quite possibly none yet, though I’m not keeping close track of results outside the US. Though *if* you believe the 20-25% estimates, possibly NYC, and maybe Stockholm? (Do we actually have any decent seroprevalence from Stockholm, or only estimates?)

My point is exactly that the data isn’t sufficient to make useful statements one way or another.

Also, I think one would need a fairly strong expectation about what “post-pandemic/endemic” COVID would look like, to make any testable statements, anyway. What kind of pattern of hospitalizations and deaths would be compatible with an “endemic” state, and what kind of pattern would disprove it?

Clearly continued exponential growth, or anything stressing hospital capacity, would disprove it. But what about short of that?

]]>What places have 1) and 2) ? Then we could discuss about the “lack of resurgence”.

]]>Oh, I didn’t mean that *you* thought it was impossible. But I’ve seen that claim repeatedly – and the original claim in the thread by jim seemed to at least be saying that no place had achieved herd immunity (I don’t think we can be *sure* of any place since not enough time has passed – but I would be *very* surprised if the hard-hit parts of Latin America with 60%+ seroprevalence haven’t, for example, and I wouldn’t rule out even some parts of the US).

And yeah I am fairly optimistic about vaccines at this point.

]]>@confused: yes of course herd immunity does occur naturally, I don’t deny that. But most likely it will require upwards of 50% of people to have had the virus within a 6 month period, and possibly upwards of 60 or 70%.

But yes I was making a statement about both practically and ethically, we shouldn’t pursue it. We have strong evidence based on many many vaccines in trials now that there will be an at least moderately effective vaccine within a few more months (say 3 to 8). So pursuing a herd immunity strategy prior to that means maybe up to tripling the number of people who’ve had the virus in the US over the next 3 to 6 months, rather than waiting until that vaccine is available and we can use it instead. A terrible strategy.

]]>oops, quotes should be @ Daniel Lakeland… sorry

]]>@ Carlos Ungil:

Well, I’m mostly saying that I don’t think anyone knows enough to make firm statements about herd immunity for COVID, due to the lack of places that have all of:

1) decent data showing a high % infected

2) significant time since end of first wave/surge

3) low enough restrictions that lack of resurgence can’t be attributed to restrictions alone

>>The idea that there are always multiple waves is just wrong.

Yep.

>> The fundamental issue with “Herd Immunity” ideas is that if you induce the “Herd Immunity” by giving people the virus… you’ve just capitulated completely.

Thing is, though, there are two different statements here. If this is an ethical statement that we *shouldn’t* pursue herd immunity by natural infection, fine.

What I object to is statements/implications that herd immunity by natural infection *can’t* happen (or *won’t* happen in the absence of measures or vaccination). If this were true, humanity wouldn’t have survived before the invention of vaccines.

]]>The idea that there are always multiple waves is just wrong. I mean, if you run a standard SEIR model it has *one* wave, everyone gets infected until herd immunity.

Of course in real epidemics, where the disease is something more than a simple sniffle/cold, people take steps to protect themselves, and maybe immunity wears off, and there are mutations, and we get a complex feedback between cases and actions, and you see a certain kind of chaotic dynamics resulting in “waves”. But those waves aren’t just “natural” they’re a result of the choices and actions of the people.

The fundamental issue with “Herd Immunity” ideas is that if you induce the “Herd Immunity” by giving people the virus… you’ve just capitulated completely. It’s like here comes the Panzer division, let’s all lie down in front of the tanks and make it easy for them to roll over us.

All the theoretical “we could get to herd immunity with only 26% of people having the virus if the connectivity graph just happens to have this particular form and we just give it to this particular subset” stuff is total garbage. It’s like “you could hit the golf ball into the hole in one shot”…yes you could… but almost never does that happen.

Herd Immunity is only an idea that makes sense with respect to vaccines, or an endemic virus where everyone already had something like it in the past.

]]>I guess those South American cities are not yet out of the first wave so they could hardly see anything like a true “second wave” yet.

In summary, the places with a second wave are places that shouldn’t be expected to have herd immunity while the places without a second wave are places that shouldn’t be expected to have herd immunity. Makes sense.

]]>For example, suppose there’s some “theorem” about martingales being non-arbitrageable …

Now we take my scheme where someone uses a crypto RNG and outputs a new number every day for 100 days… you can place bets on the final price.

Except this time, someone first sucked down the Crypto RNG data and wrote it to a disk… and a hacker has infiltrated this computer and read that file… The hacker now has 100% certainty about the entire path that will be output in the future. His probability measure is very different from everyone elses.

So we have “everyone else” and “the hacker”.

The hacker can clearly buy contracts when they are under the final price and sell them later when they are above the final price. The hacker has ZERO risk, and makes a guaranteed profit.

And yet everyone agreed that the crypto RNG output was a martingale and couldn’t be arbitraged… So whatever that proof is it either assumes something like “there is fundamentally only a single ‘real’ source of randomness that is shared by everyone and no-one has knowledge of the internals of it” which is just *clearly wrong* in elections and in the crypto-rng with hacker case… or the theorem is wrong.

]]>Depends what you think the herd immunity threshold is.

Some cities in South America seem to have hit 60%+ though (and there are probably a lot more that haven’t had serology studies done…)

But some of the “network” models have suggested herd immunity thresholds can be much lower in less homogeneous social networks.

But I think serology studies suggest Madrid was only a bit over 10%, so no way – but it had a disproportionately high death rate due to lack of understanding of treatment back in March + age of population.

Again, though, I think there is a lot of confusion as to what people mean by “herd immunity”. When pandemics end naturally (e.g. 1918-19, before vaccines) the virus doesn’t disappear, it just stops being “at epidemic levels” and becomes part of the normal background.

]]>