Few-shot learning (FSL) aims to achieve good generalization without sufficient annotations in the novel classes. Despite the successes of a number of few-shot learning methods motivated from various perspectives, the sensitivity to the limited amount and the discriminative power of support data is not well understood, which is also called the sampling bias problem. This paper reveals one such phenomenon —- the classification boundary is very sensitive to the position of support samples if they are in the vicinity of the data centroid, which we call the task centroid expressing the data centroids for a given task, degenerated and unstable results are usually observed. To reduce this sampling bias, motivated by the effect of the task centroid, we propose a simple feature transformation, named Task Centroid Projection Removing(TCPR). TCPR aims to remove the component of features along the direction of approximated task centroid which is estimated through similar examples from the base dataset. This effectively prevents features from being too close to the task centroid. Extensive experiments over ten datasets from different domains show that TCPR can reliably improve classification accuracy across various feature extractors, training algorithms, and datasets. The code can be found in the Supplementary.