Multi-step greedy policies have been extensively used in model-based Reinforcement Learning (RL), both when a model of the environment is available (e.g., in the game of Go) and when it is learned. In this paper, we explore the benefits of multi-step greedy policies in model-free RL when employed using the multi-step Dynamic Programming algorithms: κ-Policy Iteration (κ-PI) and κ-Value Iteration (κ-VI). These methods iteratively compute the next policy (κ-PI) and value function (κ-VI) by solving a surrogate decision problem with a shaped reward and a smaller discount factor. We derive model-free RL algorithms based on κ-PI and κ-VI in which the surrogate decision problem is solved by DQN and TRPO. We call the resulting algorithms κ-PI-DQN, κ-VI-DQN, κ-PI-TRPO, and κ-VI-TRPO and evaluate them on Atari and MuJoCo benchmarks. Our results indicate that for the right range of κ, our algorithms outperform DQN and TRPO. Moreover, we identify the importance of a hyper-parameter that controls the extent to which the surrogate decision problem is solved, and show how to set this parameter. Finally, we establish that κ-PI-TRPO coincides with the popular GAE algorithm.