To Ensemble or Not Ensemble: When does End-To-End Training Fail?