Skip to content

No, this senatorial stock-picking study does not address concerns about insider trading:

Jonathan Falk writes:

As you have tirelessly promoted, a huge problem with NHST is that “insignificant” effects on average can mask, via attenuation bias, important changes in subgroups. Further, as you have somewhat less tirelessly pointed out, you need much bigger samples to reliably see anything in subgroups, particularly when (ok.. you’re back to your tireless status) the groups themselves are within the researcher’s ability to cherrypick.

Here’s what I’m missing: Is there any possible solution to this? I mean, I know there’s no general solution. What I guess I’m asking is: is there ever a solution in any situation? It seems to me the only answer is rhetorical, not statistical. We can say that diabetics have much higher Covid mortality, and we can postulate underlying reasons why that’s true and why that’s a sensible subgroup to study, but maybe it’s just a subset of diabetics with huge rates, and the rest of the diabetics are fine, so diabetes isn’t the problem at all. And of course with any subgroup with an attenuation problem, the same argument can be used to say: “Sure, but what you’re missing is the subgroup that’s both left-handed and red-haired.” The only constraints on that are based on theory: left-handedness and red-hairedness ought not matter, and, by your (tireless) piranha principle, the addition (multiplication) of trivialities are way more likely to be statistical artifacts than causal.

And even if you argue that the have a good enough understanding of the mechanism of diabetic sensitivity that the rhetorical creation of a diabetic group is sensible, I don’t see how that works in social science where we don’t really have much beyond a rhetorical basis for anything. (Pardon my radical skepticism here: I’m old and jaded.)

What got me thinking about this was this paper. I don’t care whether Senators suck at picking stocks in the aggregate. (That’s actually quite unsurprising.) What I want to know is whether individual Senators with above-average performance are leveraging information they shouldn’t leverage. I see no way to get at that question statistically. The best we can do, it seems to me, is to show the spread of performances across Senators, possibly comparing them to non-Senators. The information to be gleaned from such an exercise is almost guaranteed to be negative, except in the case, like the xkcd this week, that the interocular test removes any need for statistics at all.

OK, so there are a couple points here.

First, regarding “you need much bigger samples to reliably see anything in subgroups,” yes, that’s the whole “you need 16 times as much data to estimate interactions as to estimate main effects” thing. This has nothing to do with cherry picking: it’s just an inherent statistical challenge—even if you knew ahead of time which interactions to look at.

What this implies is that if interactions might be important, we just have to accept uncertainty in our inferences. We should not demand “statistical significance,” we should be able to work with uncertainty intervals that include zero without declaring no effect, we should even accept that we’ll be getting the signs wrong a lot of the time.

The second point is that the average effect can be pretty close to irrelevant. The senatorial stock-picking example is a good one here. Some senators have been credibly accused of unethical or criminal behavior involving insider trading. If we find out that these or other senators are often losing money on stock trades, that doesn’t make the questionable behavior ok. Imagine if the cops arrest someone for drug dealing, or robbing a bank, or holding up 7-11’s, or whatever, and the perp replies with some statistics about how street criminals are generally a pretty sorry lot and can even lose money from their criminal enterprises—it turns out that robbing liquor stores is worse, financially, than working in a liquor store . . . didn’t Steven Levitt publish a paper once, reporting that most drug dealers are barely getting by—then, would that mean that the crimes they’re committing are not actually crimes? No, of course not.

How applicable this is to causal inference more generally, I’m not sure. But, yeah, the fact that senators don’t generally do well at picking stocks (as Falk notes, no surprise, if for no other reason that politicians have a lot of constraints on their investment behavior, for example they might want to avoid investing in all sorts of unpopular companies or companies to which they’re ideologically opposed or are competitors with businesses in their state, etc.) . . . I mean, sure, it’s fine to show this, but this is the baseline; it doesn’t really address the questions that arise involving specific instances of insider trading, and I think it’s misleading for that article to make the connection.


  1. Matt Skaggs says:

    “Is there any possible solution to this? I mean, I know there’s no general solution. What I guess I’m asking is: is there ever a solution in any situation?”

    It’s a good question but I think the answer is “yes.” The solution comes when you can generate useful data on the same topic in an entirely different way, backed up by theory. Suppose you have a lot of survey data on how people respond emotionally to a stimulus, but cannot rule out some of the issues mentioned above about red-haired left handers and meaningless averages. Then you discover that someone has developed a brain scanning technique that seems to show the emotional responses of interest. If you pre-register what you think the brain scans will show for each subgroup, and the data works out the way you expected, you have gone a long way towards a meaningful solution.

    Sort of a “law of two” that crops up in a lot of situations. If you have two entirely independent reasons to believe something is true, it is much more likely than if you only have one.

    • Martha (Smith) says:

      Matt said,
      “The solution comes when you can generate useful data on the same topic in an entirely different way, backed up by theory. ”

      I can’t agree. Theory can be way off the wall, I which case it’s not part of the solution. Saying “sound theory” might be getting somewhat close to reality (although it also is open to different opinions on what is and is not sound.)

      • Michael Lew says:

        If the theory is not well-established (“sound”) then its relationship with the finding might not be supporting. Instead, the data (or analytical results) could be what suggest the theory. Such theories are useful, but mostly as collections of hypotheses that should be tested. It is critically important that the hypotheses be tested with fresh data, ideally data generated in Matt Skagg’s “entirely different way”.

    • Jonathan (another one) says:

      OK… maybe. But (a) what purpose does preregistration serve in this? Why is that a necessary step? (It’s obviously useful as a rhetorical strategy, but is it anything more than that? (b) I can see how *different* data can be superior to *more* data in some circumstances, but we already knew that *more* data would have worked by itself, at least in theory. So there may be an economy in different data, and it may well not be subject to the same biases (confounding, colliding, sample selection, measurement error) present in the original data set, but it might have new ones.

    • Matt Skaggs says:

      In looking at this again, I guess I missed:

      “It seems to me the only answer is rhetorical, not statistical.”

      “Rhetorical” here is meant to encompass “logical” I think. I do not see a way of taking the information from the brain scan – or physics of mask-wearing or whatever – and putting it in to a statistical framework, so I guess my approach is not a solution at all. That is, the approach indisputably increases confidence despite not helping to resolve any of the statistical challenges.

  2. oncodoc says:

    This issue bothered me during my professional life. My patients in clinical trials almost always had disappointing outcomes; the vast majority of clinical trials don’t have great results. Some trials did have great breakthrough results; yeah! However, I had a few cases where an individual patient had a dramatically good result in a trial that as a whole was disappointing. Surely those patients or their tumors had some metabolic quirk that underlay this, but our trials did not retrieve this information. I have thought that a focused look at the outliers, both for good results and bad toxicities, could be productive. There are often more than a hundred people in a clinical trial, and they are not a uniformly syngeneic lot like some lab mouse strain.

    • Anoneuoid says:

      There was a particular lineage of rats in one room at a Harlan (now called Envigo, I think) facility in France. They had an extra branch off the middle cerebral artery and the secret sauce was to do research on interventions for stroke research with those rats.

      I have no doubt animal research is plagued by that type of thing.

      And yes, the outliers are the most interesting outcomes. But those results get “thrown out” in pursuit of some average effect. On the other hand you get stuff like 80 million people in the US on blood pressure meds with an NNT of ~100, meaning ~79 million are not benefiting.

    • Martha (Smith) says:

      oncodoc said,

      “I have thought that a focused look at the outliers, both for good results and bad toxicities, could be productive. There are often more than a hundred people in a clinical trial, and they are not a uniformly syngeneic lot like some lab mouse strain.”

      Very good point.

    • jim says:

      The problem with following up the kind of outliers you describe is that most are probably outliers for all the “right” reasons: some other effect occurred to change their condition that had nothing to do with the treatment effect, their co-occurrence is coincidental.

      If you have a highly controlled experiment, like a chemical reaction in a test tube, you can rule this possibility out. But when you’re just sticking some compound into a human, you’re inserting a compound for which many of the properties are unknown into an astronomically complex chemical machine, about which almost nothing is known relative to what’s “knowable”. Under those circumstances, “no consistent effect” is likely to be the norm; the surprising outcome is when something actually works, even though the compound has been “test-tubed” to have some discernable effect.

  3. Peter Dorman says:

    Isn’t this sort of question (senators and insider trading) what case studies are for? As you say, it isn’t interesting to know whether senators are insider traders on average or what their average profit from trading is (unless most of them do it in a justifiably illegal way). We are really interesting in two questions, (a) are some of them doing it, and (b) from a policy perspective, what institutional or other factors play a meaningful role in facilitating or encouraging insider trading?

    The first question is mostly about individual cases. At most we might be interested in readily observable markers of this behavior so we don’t have to expend as many resources examining future cases as past or current ones, and for this we might want to test candidates that arose through case study in larger sets.

    The second may present opportunities for statistical analysis, since institutional/legal/economic factors pertinent to some possibly egregious senators in case studies may be present for other sets of actors in other contexts. We might end up with bigger samples and broader questions.

    I’m writing this because I think the relationship between deep description and causal explication of individual cases and statistical analysis of sets of them is important for both.

  4. elin says:

    So contrary to popular understanding diabetic is not a dichotomous variable — diagnosis is based on being above a specific cut-off level of glucose, 1 point above you are diagnosed, one point below and you aren’t. Sure that is helpful for doctors and patients in terms of decision making, but in terms of causal analysis it’s not really the way we should think. Sure there maybe a curvilinear relationship but there is a different and more useful way to analyze it. Same thing with BMI (and of course the two are correlated).

    And then, as Andrew said, looking at some kind of average doesn’t really tell us anything about these individuals –not to mention that you can attempt to insider trade and get it wrong since the whole thing is based on thinking you know what is going to happen. The SEC uses really sophisticated analyses of trading to spot potential insider trading. Unlike in these recent Senate cases the people who usually do this work through proxies. Even Chris Collins (the Representative from NY who Trump just pardoned) was about his son, the son’s fiance and her parents. Smarter people would have had an off shore company that was a lot harder to trace. But even if you are an amateur it’s still a crime.

    When I was writing about white collar crime I once had a case in the data where the guy was actually watching trading in a company that he thought looked suspicious. He got swept up when the whole scheme crashed.

    • Jonathan (another one) says:

      This is a hair off-topic, but, technically speaking, Senators aren’t really “insider trading” at all, because they aren’t insiders. The “sophisticated” SEC techniques you cite aren’t really that sophisticated at all. They find extraordinary returns in particular events and then ask the people who profited why they made the trades they did. At that point, they then investigate the guys with the profits, and (as you point out with Collins) their friends and family.

      Technically, once a Senator is told about something, trading on it is trading in public information, which is legal. The exception is when the briefing the Senator attended is itself confidential. Insider trading is a somewhat bizarre crime, because it is not on the books… it is entirely judge-made. But its central theme is violating a duty to the company, not just using information that is out there in the world. Without someone somewhere violating a duty to a company, you can’t have insider trading. So none of the Senators are actually accused of the crime of insider trading, just the unseemly act of making personal profit off of information they were given in the course of their job. This was entirely legal (if unseemly) before the STOCK Act was passed in 2012. And the STOCK Act clarifies that Congressmen have a duty owed to the *public* not to trade on nonpublic information, which is really different that the duty owed to a corporation.

      • elin says:

        Well I think we are not likely prosecuting a criminal case here (mainly these are handled by the SEC not the criminal courts), but the term “insider” trading to mean having access to **material non public information**. If you work for a printing company and use that information (this used to happen regularly). If you work at a newspaper or a cable network and see that someone is going to recommend a buy or sell on a stock and you act on that prior to publication that is also covered. I disagree that most insider cases are about violating duty to a company (unless you mean because insider trading is considered a fiduciary violation); it’s getting an unfair advantage over other investors (who are the one harmed).

        • Jonathan (another one) says:

          With all due respect, you’re wrong. EVERY insider trading case ever has been about violation of a duty to a company. The Dirks case (involving the printer) extended the direct obligation to somewhat more indirect ones, but that’s all. An unfair advantage over other investors does not create insider trading liability. See the great work of Matt Levine, including As another obvious example, Warren Buffett can buy stock and then announce that he has bought stock, which will cause the stock price to rise. He clearly in this example has an unfair advantage over other investors with respect to material nonpublic information (his own purchasing intentions), but he has no duty to the company he bought the stock from or to other investors.

  5. Roger says:

    If I learned that my senator was using his knowledge and skills to make successful stock market trades, then I would be more likely to vote for him!

    I want senators who can mentally process news stories, and take profitable action. I am afraid that most senators are too clueless to act.

  6. Rodney Sparapani says:

    I’m assuming the XKCD reference is to this one

  7. BB says:

    I really miss the concept of “appearance of impropriety”. It almost doesn’t matter if senators are benefiting from inside info – I’m not particularly hurt by them getting 3M more dollars. However, I am hurt if I have coherent reason to believe government officials are selling me out. That causes me (and everyone else) to lose faith in them.

    It is the same reason why some people choose not to hire their own children or sleep with a inferior at work.

Leave a Reply