S4ND: Modeling Images and Videos as Multidimensional Signals Using State Spaces
Oct 28, 2022
Speakers
Eric Nguyen
Speaker · 0 followers
Karan Goel
Speaker · 0 followers
Gordon W. Downs
Speaker · 0 followers
Preey Shah
Speaker · 0 followers
Stephen A. Baccus
Speaker · 0 followers
Christopher Ré
Speaker · 5 followers
About
Visual data such as images and videos are typically modeled as discretizations of inherently continuous, multidimensional signals. Existing continuous-signal models attempt to exploit this fact by modeling the underlying signals of visual (e.g., image) data directly.However, they have not yet been able to achieve competitive performance on practical vision tasks such as large-scale image and video classification.Building on a recent line of work on deep state space models (SSMs), we propose , a new multidimensional SSM layer that extends SSMs' continuous-signal modeling ability to multidimensional data including images and videos.We show that S4ND can model large-scale visual data in 1D, 2D, and 3D as continuous multidimensional signals and demonstrate strong performance by simply swapping Conv2D and self-attention layers with layers in existing state-of-the-art models.On ImageNet-1k, exceeds the performance of a ViT baseline by 1.5% accuracy when training with a 1D sequence of patches, and matches ConvNeXt when modeling images in 2D. For videos, S4ND improves on an inflated 3D ConvNeXt in activity classification on HMDB-51 by 4% accuracy.S4ND implicitly learns global, continuous convolutional kernels that are resolution invariant by construction, providing an inductive bias that enables generalization across multiple resolutions.By developing a simple bandlimiting modification to S4 to overcome aliasing, S4ND achieves strong zero-shot (unseen at test time) resolution performance, e.g. achieving 88.7% accuracy on CIFAR-10 when trained on 16 × 16 and tested on 32 × 32 images.When trained with progressive resizing, S4ND comes within ∼ 1% of a high-resolution model while training 22% faster.Visual data such as images and videos are typically modeled as discretizations of inherently continuous, multidimensional signals. Existing continuous-signal models attempt to exploit this fact by modeling the underlying signals of visual (e.g., image) data directly.However, they have not yet been able to achieve competitive performance on practical vision tasks such as large-scale image and video classification.Building on a recent line of work on deep state space models (SSMs), we propose , a…
Organizer
NeurIPS 2022
Account · 952 followers
Like the format? Trust SlidesLive to capture your next event!
Professional recording and live streaming, delivered globally.
Sharing
Recommended Videos
Presentations on similar topic, category or speaker
Few-shot Relational Reasoning via Connection Subgraph Pretraining
Watch later
Qian Huang, …
Total of 0 viewers voted for saving the presentation to eternal vault which is 0.0%
Prospects for understanding thy physics of the Universe
Watch later
Total of 0 viewers voted for saving the presentation to eternal vault which is 0.0%
Domain Adaptation: Theory, Algorithms, and Open Library
Watch later
Total of 1 viewers voted for saving the presentation to eternal vault which is 0.1%
This is the way: designing and compiling LEPISZCZE, a comprehensive NLP benchmark for Polish
Watch later
Total of 0 viewers voted for saving the presentation to eternal vault which is 0.0%
Using Shannon Information to Probe the Precision of Synaptic Strengths
Watch later
Total of 0 viewers voted for saving the presentation to eternal vault which is 0.0%
Bridging Central and Local Differential Privacy in Data Acquisition Mechanisms
Watch later
Total of 0 viewers voted for saving the presentation to eternal vault which is 0.0%