Efficient Adaptation for End-to-End Vision-Based Robotic Manipulation