We consider a regression problem, where the correspondence between input data and output data is not accessible. Such shuffled data is commonly observed. For example, in flow cytometry, the measuring instruments are unable to preserve the correspondence between the samples and the measurements. Existing works for this problem generally focus on small number of data and/or linear case. In this work, we propose a novel computational framework (ROBOT) for the shuffled regression problem that can handle large-scale data and complex models more effectively than previous methods. To facilitate this, we consider the interaction between the regression parameter and the data correspondence parameter by building an end-to-end training framework using differentiable programming techniques. As an extension, ROBOT can also be applied to the robust correspondence setting, where the input data and the output data are not exactly aligned. Numerical experiments show that ROBOT achieves better performance than existing methods in both linear and nonlinear regression tasks, including real-world applications such as flow cytometry and multi-object tracking.