With meaningful and simplified representations of neural activity, we often are afforded insights into how and what information is being processed within a neural circuit. However, without labels, the process of finding representations that reveal the link between the brain and behavior can be challenging. Here, we introduce a novel unsupervised approach for learning representations of neural activity called Swap-VAE. Our approach combines a generative modeling framework with an instance-specifi…