Self-supervised Representation Learning with Relative Predictive Coding
May 3, 2021
Speakers
Yao-Hung Hubert Tsai
Řečník · 1 sledující
Martin Q. Ma
Řečník · 0 sledujících
Muqiao Yang
Řečník · 0 sledujících
Louis-Philippe Morency
Řečník · 2 sledující
Russ Salakhutdinov
Řečník · 13 sledujících
About
This paper introduces Relative Predictive Coding (RPC), a new contrastive representation learning objective that maintains a good balance among training stability, minibatch size sensitivity, and downstream task performance. The key to the success of RPC is two-fold. First, RPC introduces the relative parameters to regularize the objective for boundedness and low variance. Second, RPC contains no logarithm and exponential score functions, which are the main cause of training instability in prior contrastive objectives. We empirically verify the effectiveness of RPC on benchmark vision and speech self-supervised learning tasks. Lastly, we relate RPC with mutual information (MI) estimation, showing RPC can be used to estimate MI with low variance.This paper introduces Relative Predictive Coding (RPC), a new contrastive representation learning objective that maintains a good balance among training stability, minibatch size sensitivity, and downstream task performance. The key to the success of RPC is two-fold. First, RPC introduces the relative parameters to regularize the objective for boundedness and low variance. Second, RPC contains no logarithm and exponential score functions, which are the main cause of training instability in prior…
Categories
Umělá inteligence a data science
Kategorie · 10,8k prezentací
About ICLR 2021
The International Conference on Learning Representations (ICLR) is the premier gathering of professionals dedicated to the advancement of the branch of artificial intelligence called representation learning, but generally referred to as deep learning. ICLR is globally renowned for presenting and publishing cutting-edge research on all aspects of deep learning used in the fields of artificial intelligence, statistics and data science, as well as important application areas such as machine vision, computational biology, speech recognition, text understanding, gaming, and robotics.
Like the format? Trust SlidesLive to capture your next event!
Professional recording and live streaming, delivered globally.
Sharing
Recommended Videos
Presentations on similar topic, category or speaker
Interactive Weak Supervision: Learning Useful Heuristics for Data Labeling
Zhlédnout později
Topological Data Analysis
Zhlédnout později
Invited Talk - Michael Bronstein
Zhlédnout později
Go with the flow: Adaptive control for Neural ODEs
Zhlédnout později
CausalWorld: A Robotic Manipulation Benchmark for Causal Structure and Transfer Learning
Zhlédnout později
Ossama Ahmed, …
Efficient Abstract Reasoning with Dual-Contrast Network
Zhlédnout později
Tao Zhuo, …