Humans learn to parse scenes more robustly than deep learning vision systems, generalizing across large variations in viewpoint, occlusion, lighting, and clutter. In this work, we present an probabilistic programming architecture for parsing scenes using object models learned from 5 or fewer views. Our architecture called 3DP3 uses voxelized models of object shape, a generative 3D scene graph prior that compositionally represents scenes using shapes and contacts between them, and a likelihood model based on real-time graphics. Our inference algorithm accurately parses scenes, using fast bottom-up pose proposals and novel involutive MCMC updates on scene graph structure. We show this approach allows for rapid object learning, and scene parsing that leverages physical constraints. Our quantitative results demonstrate better generalization to scenes with novel viewpoints, contact, and occlusions than is achieved by deep learning baselines, and accurate parsing of real scenes using neural bottom-up proposals.