We study the problem of training deep networks while enforcing quantization and precision constraints to its parameters, a setting which can reduce energy consumption and inference time of deployed models. Unlike previous works, we propose a method that assigns different precisions (number of bits) to weights in a neural network, yielding an heterogeneous allocation of bits across parameters. Our method is derived from a novel framework, where the intractability of optimizing discrete precisions…