Informally, a 'spurious correlation' is the dependence of a model on some aspect of the input data that an analyst thinks shouldn't matter. In machine learning, these have a know-it-when-you-see-it character; e.g., changing the gender of a sentence's subject changes a sentiment predictor's output. In this paper, we study counterfactual invariance, a causal formalization of the perturbative stress testing requirement that changing irrelevant parts of the input shouldn’t change model predictions.…