Neural networks have proven effective in constructing surrogate models for parametric partial differential equations (PDEs) and for approximating high-dimensional quantity of interest (QoI) surfaces. A major cost is training such models is collecting training data, which requires solving the target PDE for a variety of different parameter settings. Active learning and experimental design methods have the potential to reduce this cost, but are not yet widely used for training neural networks, nor do there exist methods with strong theoretical foundations. In this work we provide evidence, both empirical and theoretical, that existing active sampling techniques can be used successfully for fitting neural network models for high-dimensional parameteric PDEs. In particular, we show the effectiveness of “coherence motivated” sampling methods (i.e., leverage score sampling), which are widely used to fit PDE surrogate models based on polynomials. We prove that leverage score sampling yields strong theoretical guarantees for fitting single neuron models, even under adversarial label noise. Our theoretical bounds apply to any single neuron model with a Lipschitz non-linearity (ReLU, sigmoid, absolute value, low-degree polynomial, etc.).