Policy gradient methods are a class of powerful algorithms in reinforcement learning (RL). More recently, some variance reduced policy gradient methods have been developed to improve sample efficiency and obtain a near-optimal sample complexity O(ϵ^-3) for finding an ϵ-stationary point of non-concave performance function in model-free RL. However, the practical performances of these variance reduced policy gradient methods are not consistent with their near-optimal sample complexity, because these methods require large batches and strict learning rates to achieve this optimal complexity. In the paper, thus, we propose a class of efficient momentum-based policy gradient methods, which use adaptive learning rates and do not require large batches. Specifically, we propose a fast important-sampling momentum-based policy gradient (IS-MBPG) method by using the important sampling technique. Meanwhile, we also propose a fast hessian-aided momentum-based policy gradient (HA-MBPG) method via using the semi-hessian information. In theoretical analysis, we prove that our algorithms also have the sample complexity O(ϵ^-3), as the existing best policy gradient methods. In the experiments, we use some benchmark tasks to demonstrate the effectiveness of algorithms.