Towards Sample-Efficient Overparameterized Meta-learning
Dez 6, 2021
Sprecher:innen
Adhyyan Narang
Speaker · 0 followers
Halil Ibrahim Gulluk
Speaker · 0 followers
Samet Oymak
Speaker · 0 followers
Maryam Fazel
Speaker · 0 followers
Über
An overarching goal in machine learning is to build a generalizable model with few samples. To this end, overparameterization has been the subject of immense interest to explain the generalization ability of deep nets even when the size of the dataset is smaller than that of the model. While the prior literature focuses on the classical supervised setting, this paper aims to demystify overparameterization for meta-learning. Here we have a sequence of linear-regression tasks and we ask: (1) Given earlier tasks, what is the optimal linear representation of features for a new downstream task? and (2) How many samples do we need to build this representation? This work shows that surprisingly, overparameterization arises as a natural answer to these fundamental meta-learning questions. Specifically, for (1), we first show that learning the optimal representation coincides with the problem of designing a task-aware regularization to promote inductive bias. We leverage this inductive bias to explain how the downstream task actually benefits from overparameterization, in contrast to prior works on few-shot learning. For (2), we develop a theory to explain how feature covariance can implicitly help reduce the sample complexity well below the degrees of freedom and lead to small estimation error. We then integrate these findings to obtain an overall performance guarantee for our meta-learning algorithm. Numerical experiments on real and synthetic data verify our insights on overparameterized meta-learning.An overarching goal in machine learning is to build a generalizable model with few samples. To this end, overparameterization has been the subject of immense interest to explain the generalization ability of deep nets even when the size of the dataset is smaller than that of the model. While the prior literature focuses on the classical supervised setting, this paper aims to demystify overparameterization for meta-learning. Here we have a sequence of linear-regression tasks and we ask: (1) Given…
Organisator
NeurIPS 2021
Account · 1.9k followers
Über NeurIPS 2021
Neural Information Processing Systems (NeurIPS) is a multi-track machine learning and computational neuroscience conference that includes invited talks, demonstrations, symposia and oral and poster presentations of refereed papers. Following the conference, there are workshops which provide a less formal setting.
Gefällt euch das Format? Vertraut auf SlidesLive, um euer nächstes Event festzuhalten!
Professionelle Aufzeichnung und Livestreaming – weltweit.
Freigeben
Empfohlene Videos
Präsentationen, deren Thema, Kategorie oder Sprecher:in ähnlich sind
Alchemy: A benchmark and analysis toolkit for meta-reinforcement learning agents
Watch later
Jane X. Wang, …
Total of 0 viewers voted for saving the presentation to eternal vault which is 0.0%
Neural Trees for Learning on Graphs
Watch later
Rajat Talak, …
Total of 0 viewers voted for saving the presentation to eternal vault which is 0.0%
Why do we need structure and where does it come from?
Watch later
Total of 0 viewers voted for saving the presentation to eternal vault which is 0.0%
Continual Learning via Local Module Composition
Watch later
Total of 0 viewers voted for saving the presentation to eternal vault which is 0.0%
Garment4D: Garment Reconstruction from Point Cloud Sequences
Watch later
Total of 0 viewers voted for saving the presentation to eternal vault which is 0.0%
Sparse Quadratic Optimisation over the Stiefel Manifold with Application to Permutation Synchronisation
Watch later
Total of 0 viewers voted for saving the presentation to eternal vault which is 0.0%