This paper proposes an automatic method to calibrate change point detection algorithms for high-dimensional time series. Our procedure builds on the ability of an expert (e.g. a medical researcher) to produce approximate segmentation estimates, called partial annotations, for a small number of signal examples. This contribution is a supervised approach to learn a diagonal Mahalanobis metric, which, once combined with a detection algorithm, is able to reproduce the expert's segmentation strategy on out-of-sample signals. Unlike previous works for change detection, our method includes a sparsity-inducing regularization which perform supervised dimension selection, and adapts to partial annotations. Experiments on activity signals collected from healthy and neurologically impaired patients support the fact that supervision markedly ameliorate detection accuracy.