Recent interest in learning variational Bayesian Neural Networks (BNNs) focused on constructing new variational posteriors and improving the optimization of variational parameters. Variational methods also allow hyperparameters of BNNs to be learned by optimization of the Evidence Lower Bound (ELBO). This approach has received less attention, even though the performance of variational BNNs is known to be very sensitive to the model hyperparameter setting used. Current practice often fixes the hy…