An important challenge in contextual bandits and reinforcement learning is to develop flexible, general-purpose algorithms that are computationally no harder than classical supervised learning tasks such as classification and regression. Algorithms based on regression have shown promising empirical success, but theoretical guarantees have remained elusive except in special cases. We provide the first universal and optimal reduction from contextual bandits to online regression. Under a standard realizability assumption, we show how to transform any algorithm for online regression with a given class of value functions into an algorithm for contextual bandits with the induced policy class. We characterize the minimax rates for contextual bandits with general function classes, and show that the resulting contextual bandit algorithm is minimax optimal whenever the base algorithm obtains the optimal rate for regression. Compared to previous results, our algorithm requires no distributional assumptions beyond realizability, and works even when contexts are chosen adversarially.