We propose a framework for learning neural scene representations directly from images, without 3D supervision. Our key insight is that 3D structure can be imposed by ensuring that the learned representation transforms like a real 3D scene. Specifically, we introduce a loss which enforces equivariance of the scene representation with respect to 3D rotations. We show that naive tensor rotations cannot be used to define models that are equivariant with respect to 3D rotations, and instead propose a new operation called invertible shear rotation, which has this equivariance property. In addition, we introduce three challenging new datasets for scene representation and neural rendering, including real images and generated scenes with complex objects and backgrounds. Through experiments, we show that our model achieves compelling results on these datasets as well as on standard ShapeNet benchmarks.