Efficiently capturing the long-range patterns in sequential data sources salient to a given task—such as classification—poses a fundamental challenge. Popular approaches in the space tradeoff between the memory burden of remembering the entire past, as in transformers, the computational burden of complicated sequential dependencies in the memory computation, as in recurrent neural networks, or the parameter burden of convolutional networks with many or large filters. We instead take inspiration…