Self-supervised Representation Learning with Relative Predictive Coding
3. května 2021
Řečníci
Yao-Hung Hubert Tsai
Řečník · 1 sledující
Martin Q. Ma
Řečník · 0 sledujících
Muqiao Yang
Řečník · 0 sledujících
Louis-Philippe Morency
Řečník · 2 sledující
Russ Salakhutdinov
Řečník · 13 sledujících
O prezentaci
This paper introduces Relative Predictive Coding (RPC), a new contrastive representation learning objective that maintains a good balance among training stability, minibatch size sensitivity, and downstream task performance. The key to the success of RPC is two-fold. First, RPC introduces the relative parameters to regularize the objective for boundedness and low variance. Second, RPC contains no logarithm and exponential score functions, which are the main cause of training instability in prior contrastive objectives. We empirically verify the effectiveness of RPC on benchmark vision and speech self-supervised learning tasks. Lastly, we relate RPC with mutual information (MI) estimation, showing RPC can be used to estimate MI with low variance.This paper introduces Relative Predictive Coding (RPC), a new contrastive representation learning objective that maintains a good balance among training stability, minibatch size sensitivity, and downstream task performance. The key to the success of RPC is two-fold. First, RPC introduces the relative parameters to regularize the objective for boundedness and low variance. Second, RPC contains no logarithm and exponential score functions, which are the main cause of training instability in prior…
Kategorie
Umělá inteligence a data science
Kategorie · 10,8k prezentací
O organizátorovi (ICLR 2021)
The International Conference on Learning Representations (ICLR) is the premier gathering of professionals dedicated to the advancement of the branch of artificial intelligence called representation learning, but generally referred to as deep learning. ICLR is globally renowned for presenting and publishing cutting-edge research on all aspects of deep learning used in the fields of artificial intelligence, statistics and data science, as well as important application areas such as machine vision, computational biology, speech recognition, text understanding, gaming, and robotics.
Baví vás formát? Nechte SlidesLive zachytit svou akci!
Profesionální natáčení a streamování po celém světě.
Sdílení
Doporučená videa
Prezentace na podobné téma, kategorii nebo přednášejícího
Return-Based Contrastive Representation Learning for Reinforcement Learning
Zhlédnout později
Guoqing Liu, …
DELTA-CLUE: Diverse Sets of Explanations for Uncertainty Estimates
Zhlédnout později
Dan Ley, …
Improving Lossless Compression Rates via Monte Carlo Bits-Back Coding
Zhlédnout později
Yangjun Ruan, …
Inference in Network-based Epidemiological Simulations with Probabilistic Programming
Zhlédnout později
Energy-Based Models: Current Perspectives, Challenges, and Opportunities
Zhlédnout později
