Proposed by Hinton et al. in 2012, dropout has stood the test of time as a regularizer for alleviating neural net overfitting. In this work, we show how dropout can also reduce underfitting, when used at the start of training. At this phase, dropout reduces the gradient variance across mini-batches and helps align the mini-batch gradients with the underlying whole-dataset gradient. Intuitively, dropout counteracts SGD data stochasticity and limits the influence of individual batches on the model…