An off-the-shelf model as a commercial service could be stolen by model stealing attacks, posing great threats to the rights of the model owner. Model fingerprinting aims to verify whether a suspect model is stolen from the victim model, which gains more and more attention nowadays. Previous methods always leverage the transferable adversarial examples as the model fingerprint, which is sensitive to adversarial defense or transfer learning scenarios. To address this issue, we consider the pairwise relationship between samples instead and propose a novel yet simple model stealing detection method based on SAmple Correlation (SAC). Specifically, we present SAC-w that selects wrongly classified normal samples as model inputs and calculates the mean correlation among their model outputs. To reduce the training time, we further develop SAC-m that selects mixed samples via CutMix as model inputs, without the need for training the surrogate models or generating adversarial examples. Extensive results validate that both SAC-w and SAC-m successfully defend against various model stealing attacks, even including adversarial training or transfer learning, and detect the stolen models with the best performance in terms of AUC across different datasets and model architectures. Code is attached in the supplementary.