Spatiotemporal Joint Filter Decomposition in 3D Convolutional Neural Networks
Dec 6, 2021
Speakers
Zichen Miao
Speaker · 0 followers
Xiuyuan Cheng
Speaker · 0 followers
About
In this paper, we introduce spatiotemporal joint filter decomposition to decouple spatial and temporal learning, while preserving spatiotemporal dependency in a video. A 3D convolutional filter is now jointly decomposed over a set of spatial and temporal filter atoms respectively. In this way, a 3D convolution layer becomes three: a temporal atom layer, a spatial atom layer, and a joint coefficient layer, all three remain convolutional. Different from methods that decorrelate the spatial and temporal modeling, the proposed decomposition can still capture spatiotemporal correlations in the joint coefficients. One obvious arithmetic manipulation allowed in our joint decomposition is to swap spatial or temporal atoms with a set of atoms with the same number but different sizes, while keeping the remaining unchanged. For example, we can now achieve tempo-invariance by simply dilating temporal atoms only. To illustrate this useful atom-swapping property, we further demonstrate how such a decomposition permits the direct learning of 3D CNNs with full-size videos through iterations of two consecutive sub-stages of learning: In the temporal stage, full-temporal downsampled-spatial data are used to learn temporal atoms and joint coefficients while fixing spatial atoms. In the spatial stage, full-spatial downsampled-temporal data are used for spatial atoms and joint coefficients while fixing temporal atoms. We show empirically on multiple action recognition datasets that, the decoupled spatial-temporal learning significantly reduces the model memory footprints, and allows deep CNNs to model high-spatial long-temporal dependency with limited computational resources, while delivering comparable performance.In this paper, we introduce spatiotemporal joint filter decomposition to decouple spatial and temporal learning, while preserving spatiotemporal dependency in a video. A 3D convolutional filter is now jointly decomposed over a set of spatial and temporal filter atoms respectively. In this way, a 3D convolution layer becomes three: a temporal atom layer, a spatial atom layer, and a joint coefficient layer, all three remain convolutional. Different from methods that decorrelate the spatial and tem…
Organizer
NeurIPS 2021
Account · 1.9k followers
About NeurIPS 2021
Neural Information Processing Systems (NeurIPS) is a multi-track machine learning and computational neuroscience conference that includes invited talks, demonstrations, symposia and oral and poster presentations of refereed papers. Following the conference, there are workshops which provide a less formal setting.
Like the format? Trust SlidesLive to capture your next event!
Professional recording and live streaming, delivered globally.
Sharing
Recommended Videos
Presentations on similar topic, category or speaker
Private AI
Watch later
Total of 0 viewers voted for saving the presentation to eternal vault which is 0.0%
Self-supervised Semi-supervised Learning for Data Labeling and Quality Evaluation
Watch later
Haoping Bai, …
Total of 0 viewers voted for saving the presentation to eternal vault which is 0.0%
Benign Overfitting
Watch later
Total of 0 viewers voted for saving the presentation to eternal vault which is 0.0%
My road of becoming an YouTuber: Tips for Academic Talks
Watch later
Total of 0 viewers voted for saving the presentation to eternal vault which is 0.0%
Opening Remarks
Watch later
Qi Dou, …
Total of 0 viewers voted for saving the presentation to eternal vault which is 0.0%
The Value of Information When Deciding What to Learn
Watch later
Total of 0 viewers voted for saving the presentation to eternal vault which is 0.0%