Bridging the Gap from Asymmetry Tricks to Decorrelation Principles in Non-contrastive Self-supervised Learning
Nov 28, 2022
Speakers
Kang-Jun Liu
Speaker · 0 followers
Masanori Suganuma
Speaker · 0 followers
Takayuki Okatani
Speaker · 0 followers
About
Recent non-contrastive methods for self-supervised representation learning show promising performance. While they are attractive since they do not need negative samples, it necessitates some mechanism to avoid collapsing into a trivial solution. Currently, there are two approaches to collapse prevention. One uses an asymmetric architecture on a joint embedding of input, e.g., BYOL and SimSiam, and the other imposes decorrelation criteria on the same joint embedding, e.g., Barlow-Twins and VICReg. The latter methods have theoretical support from information theory as to why they can learn good representation. However, it is not fully understood why the former performs equally well. In this paper, focusing on BYOL/SimSiam, which uses the stop-gradient and a predictor as asymmetric tricks, we present a novel interpretation of these tricks; they implicitly impose a constraint that encourages feature decorrelation similar to Barlow-Twins/VICReg. We then present a novel non-contrastive method, which replaces the stop-gradient in BYOL/SimSiam with the derived constraint; the method empirically shows comparable performance to the above SOTA methods in the standard benchmark test using ImageNet. This result builds a bridge from BYOL/SimSiam to the decorrelation-based methods, contributing to demystifying their secrets.Recent non-contrastive methods for self-supervised representation learning show promising performance. While they are attractive since they do not need negative samples, it necessitates some mechanism to avoid collapsing into a trivial solution. Currently, there are two approaches to collapse prevention. One uses an asymmetric architecture on a joint embedding of input, e.g., BYOL and SimSiam, and the other imposes decorrelation criteria on the same joint embedding, e.g., Barlow-Twins and VICReg…
Organizer
NeurIPS 2022
Account · 961 followers
Like the format? Trust SlidesLive to capture your next event!
Professional recording and live streaming, delivered globally.
Sharing
Recommended Videos
Presentations on similar topic, category or speaker
On Analyzing Generative and Denoising Capabilities of Diffusion-based Deep Generative Models
Watch later
Kamil Deja, …
Total of 0 viewers voted for saving the presentation to eternal vault which is 0.0%
Video compression dataset and benchmark of learning-based video-quality metrics
Watch later
Total of 0 viewers voted for saving the presentation to eternal vault which is 0.0%
Spectrum Random Masking for Generalization in Image-based Reinforcement Learning
Watch later
Yangru Huang, …
Total of 0 viewers voted for saving the presentation to eternal vault which is 0.0%
Kernel Memory Networks: A Unifying Framework for Memory Modeling
Watch later
Total of 0 viewers voted for saving the presentation to eternal vault which is 0.0%
Squeezeformer: An Efficient Transformer for Automatic Speech Recognition
Watch later
Sehoon Kim, …
Total of 0 viewers voted for saving the presentation to eternal vault which is 0.0%
fficient Finetuning of Transformers for Source Code
Watch later
Total of 0 viewers voted for saving the presentation to eternal vault which is 0.0%