RényiCL: Contrastive Representation Learning with skew Rényi Divergence
Nov 28, 2022
Speakers
Kyungmin Lee
Speaker · 0 followers
Jinwoo Shin
Speaker · 2 followers
About
Contrastive representation learning seeks to acquire useful representations by estimating the shared information between multiple views of data. Here, the choice of data augmentation is sensitive to the quality of learned representations: as harder the data augmentations are applied, the views share more task-relevant information, but also task-irrelevant one that can hinder the generalization capability of representation. Motivated by this, we present a new robust contrastive learning scheme, coined RényiCL, which can effectively manage harder augmentations by utilizing Rényi divergence. Our method is built upon the variational lower bound of a Renyi divergence, but a naive usage of a variational method exhibits unstable training due to the large variance. To tackle this challenge, we propose a novel contrastive objective that conducts variational estimation of a skew Renyi divergence and provides a theoretical guarantee on how variational estimation of skew divergence leads to stable training. We show that Rényi contrastive learning objectives perform innate hard negative sampling and easy positive sampling simultaneously so that it can selectively learn useful features and ignore nuisance features. Through experiments on ImageNet, we show that Rényi contrastive learning with stronger augmentations outperforms other self-supervised methods without extra regularization or computational overhead. Also, we validate our method on various domains such as graph and tabular datasets, showing empirical gain over original contrastive methods.Contrastive representation learning seeks to acquire useful representations by estimating the shared information between multiple views of data. Here, the choice of data augmentation is sensitive to the quality of learned representations: as harder the data augmentations are applied, the views share more task-relevant information, but also task-irrelevant one that can hinder the generalization capability of representation. Motivated by this, we present a new robust contrastive learning scheme, c…
Organizer
NeurIPS 2022
Account · 962 followers
Like the format? Trust SlidesLive to capture your next event!
Professional recording and live streaming, delivered globally.
Sharing
Recommended Videos
Presentations on similar topic, category or speaker
Provable Benefit of Multitask Representation Learning in Reinforcement Learning
Watch later
Yuan Cheng, …
Total of 0 viewers voted for saving the presentation to eternal vault which is 0.0%
Transcormer: Transformer for Sentence Scoring with Sliding Language Modeling
Watch later
Kaitao Song, …
Total of 0 viewers voted for saving the presentation to eternal vault which is 0.0%
Evaluating neural network models of vision and audition with model metamers
Watch later
Total of 0 viewers voted for saving the presentation to eternal vault which is 0.0%
Proactive Detractor Detection Framework Based on Message-Wise Sentiment Analysis Over Customer Support Interactions
Watch later
Total of 0 viewers voted for saving the presentation to eternal vault which is 0.0%
Revisiting Injective Attacks on Recommender Systems
Watch later
Haoyang Li, …
Total of 0 viewers voted for saving the presentation to eternal vault which is 0.0%
Measuring Data Reconstruction Defences in Collaborative Inference Systems
Watch later
Mengda Yang, …
Total of 0 viewers voted for saving the presentation to eternal vault which is 0.0%