We study the problem of constructing adversarial examples in the black-box setting, where no model information is revealed except the feedback knowledge of given inputs. To obtain sufficient knowledge for crafting adversarial examples, previous methods query the target model with inputs that are perturbed with different searching directions. However, these methods suffer from poor query efficiency since the employed searching directions are sampled randomly. To mitigate the issue, we capture the goal of mounting efficient attacks as an optimization problem that the adversary tries to fool the target model with limited queries. In such setting, the adversary has to select appropriate searching directions to reduce model queries. By solving the efficient-attack problem, we find that what we need is to distill the knowledge in both the adversarial example path and the searching direction path. Therefore, we propose a novel framework, dual-path distillation, that utilizes the feedback knowledge not only to craft adversarial examples but also alter searching directions for efficient attacks. Experimental results suggest that our framework can significantly increase the query efficiency.