While recent work has shown that the choice of hyperparameters, e.g., the number of training epochs, can affect pruning performance, there is no principled framework to predict when and how tuning a given hyperparameter will affect the performance of a pruning method. In this paper, we perform an empirical analysis of neural network (NN) pruning, and based on this, we propose a simple model to quantify the effect of hyperparameters in NN pruning. Drawing upon ideas from the statistical mechanics of learning, we formulate an empirical analysis of NN pruning in terms of temperature-like parameters, such as the number of training epochs, and load-like parameters, such as the number of pruned parameters. The key ingredient of the empirical results is a transition phenomenon: depending on the value of a load-like parameter, increasing the temperature-like parameter can either improve or damage the pruning performance. Based on this transition, we develop a three-regime model by taxonomizing the global structure of pruned NN loss landscape, and we demonstrate that the dichotomous effect of high temperature corresponds to transitions between different types of global structures. We present three case studies on applying the three-regime model: 1) determining whether to increase or decrease a hyperparameter to improve pruning. 2) selecting the best model to prune from a family of models. 3) tuning the hyperparameter of Sharpness Aware Minimization (SAM) for better pruning performance.