In this paper, we introduce a novel form of a value function, Q(s, s'), that expresses the utility of transitioning from a state s to a neighboring state s' and then acting optimally thereafter. In order to derive an optimal policy, we develop a novel forward dynamics model that learns to make next-state predictions that maximize Q(s,s'). This formulation decouples actions from values while still learning off-policy. We highlight the benefits of this approach in terms of value function transfer,…