More than a dozen excitatory cell types have been identified in the mouse primary visual cortex (V1) based on transcriptomic, morphological and in vitro electrophysiological features. However, little is known about the functional organization of visual cortex neurons and their responses properties beyond orientation selectivity. Here, we combined large-scale two-photon imaging and predictive modeling of neural responses to study the functional organization of mouse V1. We developed a rotation-equivariant model architecture, followed by a rotation-invariant clustering pipeline to map the landscape of neural function in V1. Clustering neurons based on their stimulus response function revealed a continuum with around 30 modes. Each mode represented a group of neurons that exhibited a specific combination of stimulus selectivity and nonlinear response properties such as cross-orientation inhibition, size-contrast tuning and surround suppression. Interestingly, these non-linear properties were expressed independently and all possible combinations were present in the population. Our study shows how building known symmetries into neural response models can reveal interesting insights about the organization of the visual system.