We show that the Invariant Risk Minimization (IRM) formulation of Arjovsky et al. (2019) can fail to capture the "natural" invariance, at least when used in its practical "linear" form, and even in very simple problems which directly follow the motivating example for IRM, leading to worse generalization on new environments, even when compared to unconstrained ERM. The issue stems from a significant gap between the linear variant (as in their concrete method IRMv1) and the full non-linear IRM formulation, as well as from the inherent ill-posedness of generalizing based on a few non-sampled environments. The issues arise even when measuring invariance on the population distributions, but are exacerbated by the fact that "pure" IRM is extremely fragile to sampling.