“When to Control for X”: Gabriel Rossman on Self-Justification Through #science

At Vox, Ezra Klein raises an important meta-theoretical point in social science analysis, which is that it is very hard to distinguish mechanisms from controls. If there is a zero-order correlation between X1 and Y, but it drops out when you introduce X2, this in of itself does not tell you if the X1~Y relationship is spurious or if it is mediated. If being shot at is correlated with death, but the effect falls out of significance once you control for exsanguination, one would not say that the probabilistic effect of being shot at on death is spurious. Rather you would say that losing a lot of blood is the mechanism by which being shot at often kills you.

I have been attentive to this issue at least since I read Lieberson’s Making it Count in grad school. In practice this sometimes comes out as fighting with peer reviewers who demand that I throw controls in a model, when the controls are closely related to the mechanism I had posited and so my model would predict that some or all of the effect would drop out when the controls/mechanisms are introduced. So I was glad to see Klein take up this issue at Vox. Klein’s particular case was that when you introduce a lot of controls, a lot of of ascriptive inequality effects drop out. He argues, very plausibly, that this is missing the point since this does not mean there is no effect, only that the effect is mediated by causal pathways.

There are two issues with this.

more