Neural networks have a remarkable capacity for contextual processing–using recent or nearby inputs to modify processing of current input. For example, in natural language, contextual processing is necessary to correctly interpret negations (phrases such as "not bad"). However, our ability to understand how networks process context is missing. Here, we propose general methods for reverse engineering recurrent neural networks (RNNs) to identify and elucidate contextual processing. We apply these methods to understand RNNs trained on sentiment classification. Through this analysis we reveal inputs that induce contextual effects, quantify the strength and timescale of their effects, and identify clusters of these inputs with similar properties. Additionally, we identify and analyze contextual effects related to differential processing of the beginning and end of documents. Using the insights learned from the RNNs we improve baseline Bag-of-Words models with simple extensions that incorporate contextual modification, recovering greater than 85