Pre-training (PT) followed by fine-tuning (FT) is an effective method for training neural network models, and has led to significant performance improvements in various domains. PT can incorporate task and data reweighting strategies, augmentation policies, and noise models, all of which can significantly impact the quality of representations learned and so the hyperparameters controlling such components must be set appropriately. However, setting the values of these hyperparameters is challenging—existing methods to automatically learn hyperparameters struggle to scale to high-dimensions or to adapt to the two-stage PT FT process. In this work, we propose an efficient, gradient-based algorithm to meta-learn PT hyperparameters. We formalize the PT hyperparameter optimization problem and propose a novel solution combining both direct and implicit differentiation. In experiments, we demonstrate that our method improves predictive performance on two real-world domains. First, we optimize high-dimensional task weighting hyperparameters for multitask pre-training on protein-protein interaction graphs and improve AUROC by up to 3.9