Semi-supervised learning (SSL) aims to leverage unlabeled data when training a model with deficient labeled data. A common methodology for SSL is to enforce the consistency of model outputs between similar samples. Empowered by recent data augmentation methods, it helps train neural nets to achieve promising SSL performance. However, the model outputs can vary dramatically on unlabeled data for different training stages especially when using large learning rates. This may introduce unpredictable noise and objective inconsistency over time that can lead to concept drift and training failure. In this paper, we study the dynamics of neural nets outputs in SSL and show that selecting the unlabeled samples with more consistent outputs over the course of training (i.e., time consistency) can improve the final test accuracy and save computations on less informative (or even harmful) unlabeled data. For the selected data, we further design their training objective as two self-taught losses, i.e., a consistency loss between a sample and its augmentation, and a contrastive loss enforcing different sample to have different outputs. In experiments, our approach outperforms recent SSL methods and achieves SOTA on several SSL benchmarks with much less computations.