Recurrent Video Restoration Transformer with Guided Deformable Attention
Nov 28, 2022
Speakers
Jingyun Liang
Speaker · 0 followers
Yuchen Fan
Speaker · 0 followers
Xiaoyu Xiang
Speaker · 0 followers
Rakesh Ranjan
Speaker · 0 followers
Simon Green
Speaker · 0 followers
Jiezhang Cao
Speaker · 0 followers
Radu Timofte
Speaker · 0 followers
Luc Van Gool
Speaker · 0 followers
About
Video restoration aims at restoring multiple high-quality frames from multiple low-quality frames. Existing video restoration methods generally fall into two extreme cases, i.e., they either restore all frames in parallel or restore the video frame by frame in a recurrent way, which would result in different merits and drawbacks. Typically, the former has the advantage of temporal information fusion. However, it suffers from large model size and intensive memory consumption; the latter has a relatively small model size as it shares parameters across frames; however, it lacks long-range dependency modeling ability and parallelizability. In this paper, we attempt to integrate the advantages of the two cases by proposing a recurrent video restoration transformer, namely RVRT. RVRT processes local neighboring frames in parallel within a globally recurrent framework which can achieve a good trade-off between model size, effectiveness, and efficiency. Specifically, RVRT divides the video into multiple clips and uses the previously inferred clip feature to estimate the subsequent clip feature. Within each clip, different frame features are jointly updated with implicit feature aggregation. Across different clips, the guided deformable attention is designed for clip-to-clip alignment, which predicts multiple relevant locations from the whole inferred clip and aggregates their features by the attention mechanism. Extensive experiments on video super-resolution, deblurring, and denoising show that the proposed RVRT achieves state-of-the-art performance on benchmark datasets with balanced model size, testing memory, and runtime. The codes will be made publicly available.Video restoration aims at restoring multiple high-quality frames from multiple low-quality frames. Existing video restoration methods generally fall into two extreme cases, i.e., they either restore all frames in parallel or restore the video frame by frame in a recurrent way, which would result in different merits and drawbacks. Typically, the former has the advantage of temporal information fusion. However, it suffers from large model size and intensive memory consumption; the latter has a rel…
Organizer
NeurIPS 2022
Account · 962 followers
Like the format? Trust SlidesLive to capture your next event!
Professional recording and live streaming, delivered globally.
Sharing
Recommended Videos
Presentations on similar topic, category or speaker
A High-Throughput Platform for Efficient Exploration of Polypeptides Chemical Space via Automation and Machine Learning
Watch later
Total of 0 viewers voted for saving the presentation to eternal vault which is 0.0%
Causality-driven Hierarchical Structure Discovery for Reinforcement Learning
Watch later
Shaohui Peng, …
Total of 0 viewers voted for saving the presentation to eternal vault which is 0.0%
Emotional Glossary of Creative Al
Watch later
Total of 0 viewers voted for saving the presentation to eternal vault which is 0.0%
SAViT: Structure-Aware Vision Transformer Pruning via Collaborative Optimization
Watch later
Total of 0 viewers voted for saving the presentation to eternal vault which is 0.0%
Supporting Food Security in Africa using Machine Learning and Earth Observations
Watch later
Total of 0 viewers voted for saving the presentation to eternal vault which is 0.0%
An Empirical Study on Disentanglement of Negative-free Contrastive Learning
Watch later
Jinkun Cao, …
Total of 0 viewers voted for saving the presentation to eternal vault which is 0.0%