Optimal Transport is gaining increasing importance in machine learning. In this talk, I will highlight directions for generalizing learning ideas with Optimal Transport. In particular, we may want to respect invariances or prior knowledge. For example, GANs have shown remarkable success in learning a distribution that faithfully recovers a reference distribution in its entirety. But, sometimes we may want to only learn some aspects (e.g., cluster or manifold structure), while modifying others (e.g., style, orientation or dimension). We propose a new model for learning across incomparable spaces, and show how to steer it towards target properties. A key component of our model is the Gromov-Wasserstein distance. Second, especially learned representations of distributions may only be alignable after applying a transformation from a known class. I will summarize ideas for incorporating such invariances in Optimal Transport distances, and implications for applications. This talk is based on joint work with David Alvarez Melis, Charlotte Bunne, Tommi Jaakkola and Andreas Krause.