Generalization on the Unseen, Logic Reasoning and Degree Curriculum
Jul 28, 2023
Speakers
Emmanuel Abbé
Speaker · 0 followers
Samy Bengio
Speaker · 1 follower
Aryo Lotfi
Speaker · 0 followers
Kevin Rizk
Speaker · 0 followers
About
This presentation considers the learning of logical (Boolean) functions with focus on the generalization on the unseen (GOTU) setting, a strong case of out-of-distribution generalization. This is motivated by the fact that the rich combinatorial nature of data in certain reasoning tasks (e.g., arithmetic/logic) makes representative data sampling challenging, and learning successfully under GOTU gives a first vignette of an 'extrapolating' or 'reasoning' learner. We then study how different network architectures trained by (S)GD perform under GOTU and provide both theoretical and experimental evidence that for a class of network models including instances of Transformers, random features models, and diagonal linear networks, a min-degree-interpolator (MDI) is learned on the unseen. We also provide evidence that other instances with larger learning rates or mean-field networks reach leaky MDIs. These findings lead to two implications: (1) we provide an explanation to the length generalization problem (e.g., Anil et al. 2022); (2) we introduce a curriculum learning algorithm called Degree-Curriculum that learns monomials more efficiently by incrementing supports.This presentation considers the learning of logical (Boolean) functions with focus on the generalization on the unseen (GOTU) setting, a strong case of out-of-distribution generalization. This is motivated by the fact that the rich combinatorial nature of data in certain reasoning tasks (e.g., arithmetic/logic) makes representative data sampling challenging, and learning successfully under GOTU gives a first vignette of an 'extrapolating' or 'reasoning' learner. We then study how different netwo…
Like the format? Trust SlidesLive to capture your next event!
Professional recording and live streaming, delivered globally.
Sharing
Recommended Videos
Presentations on similar topic, category or speaker
Disentangled Multiplex Graph Representation Learning
Watch later
Yujie Mo, …
Approximate Causal Effect Identification under Weak Confounding
Watch later
Ziwei Jiang, …
Abstracting Imperfect Information Away from Two-Player Zero-Sum Games
Watch later
Ewald-Based Long-Range Message Passing for Molecular Graphs
Watch later
MAHALO: Unifying Offline Reinforcement Learning and Imitation Learning from Observations
Watch later
Anqi Li, …