When MAML Can Adapt Fast and How to Assist When It Cannot
Apr 14, 2021
Speakers
Séb Arnold
Speaker · 0 followers
Shariq Iqbal
Speaker · 0 followers
About
Model-Agnostic Meta-Learning (MAML) and its variants have achieved success in meta-learning tasks on many datasets and settings. Nonetheless, we have just started to understand and analyze how they are able to adapt fast to new tasks. In this work, we contribute by conducting a series of empirical and theoretical studies, and discover several interesting, previously unknown properties of the algorithm. First, we find MAML adapts better with a deep architecture even if the tasks need only a shallow one. Secondly, linear layers can be added to the output layers of a shallower model to increase the depth without altering the modelling capacity, leading to improved performance in adaptation. Alternatively, an external and separate neural network meta-optimizer can also be used to transform the gradient updates of a smaller model so as to obtain improved performances in adaptation. Drawing from these evidences, we theorize that for a deep neural network to meta-learn well, the upper layers must transform the gradients of the bottom layers as if the upper layers were an external meta-optimizer, operating on a smaller network that is composed of the bottom layers.Model-Agnostic Meta-Learning (MAML) and its variants have achieved success in meta-learning tasks on many datasets and settings. Nonetheless, we have just started to understand and analyze how they are able to adapt fast to new tasks. In this work, we contribute by conducting a series of empirical and theoretical studies, and discover several interesting, previously unknown properties of the algorithm. First, we find MAML adapts better with a deep architecture even if the tasks need only a shall…
Organizer
AISTATS 2021
Account · 63 followers
Categories
AI & Data Science
Category · 10.8k presentations
About AISTATS 2021
The 24th International Conference on Artificial Intelligence and Statistics was held virtually from Tuesday, 13 April 2021 to Thursday, 15 April 2021.
Like the format? Trust SlidesLive to capture your next event!
Professional recording and live streaming, delivered globally.
Sharing
Recommended Videos
Presentations on similar topic, category or speaker
A Variational Inference Approach to Learning Multivariate Wold Processes
Watch later
Total of 0 viewers voted for saving the presentation to eternal vault which is 0.0%
Causal Modeling with Stochastic Confounders
Watch later
Total of 0 viewers voted for saving the presentation to eternal vault which is 0.0%
Non-asymptotic Performance Guarantees for Neural Estimation of f-Divergences
Watch later
Total of 0 viewers voted for saving the presentation to eternal vault which is 0.0%
On the Generalization Properties of Adversarial Training
Watch later
Yue Xing, …
Total of 0 viewers voted for saving the presentation to eternal vault which is 0.0%
Sequential Random Sampling Revisited: Hidden Shuffle Method
Watch later
Total of 0 viewers voted for saving the presentation to eternal vault which is 0.0%
Stable ResNet
Watch later
Total of 0 viewers voted for saving the presentation to eternal vault which is 0.0%