The Curse of Low Task Diversity: On the Failure of Transfer Learning to Outperform MAML and Their Empirical Equivalence
Dec 2, 2022
Speakers
Brando Miranda
Speaker · 0 followers
Patrick Yu
Speaker · 0 followers
Yu-Xiong Wang
Speaker · 0 followers
Sanmi Koyejo
Speaker · 2 followers
About
Recently, it has been observed that a transfer learning solution might be all we need to solve many few-shot learning benchmarks – thus raising important questions about when and how meta-learning algorithms should be deployed. In this paper, we seek to clarify these questions by 1. proposing a novel metric – the diversity coefficient – to measure the diversity of tasks in a few-shot learning benchmark and 2. by comparing Model-Agnostic Meta-Learning (MAML) and transfer learning under fair conditions (same architecture, same optimizer, and all models trained to convergence).Using the diversity coefficient, we show that the popular MiniImageNet and CIFAR-FS few-shot learning benchmarks have low diversity. This novel insight contextualizes claims that transfer learning solutions are better than meta-learned solutions in the regime of low diversity under a fair comparison. Specifically, we empirically find that a low diversity coefficient correlates with a high similarity between transfer learning and MAML learned solutions in terms of accuracy at meta-test time and classification layer similarity (using feature based distance metrics like SVCCA, PWCCA, CKA, and OPD). To further support our claim, we find this meta-test accuracy holds even as the model size changes. Therefore, we conclude that in the low diversity regime, MAML and transfer learning have equivalent meta-test performance when both are compared fairly.We also hope our work inspires more thoughtful constructions and quantitative evaluations of meta-learning benchmarks in the future.Recently, it has been observed that a transfer learning solution might be all we need to solve many few-shot learning benchmarks – thus raising important questions about when and how meta-learning algorithms should be deployed. In this paper, we seek to clarify these questions by 1. proposing a novel metric – the diversity coefficient – to measure the diversity of tasks in a few-shot learning benchmark and 2. by comparing Model-Agnostic Meta-Learning (MAML) and transfer learning under fair condi…
Organizer
NeurIPS 2022
Account · 954 followers
Like the format? Trust SlidesLive to capture your next event!
Professional recording and live streaming, delivered globally.
Sharing
Recommended Videos
Presentations on similar topic, category or speaker
Lifelong Neural Predictive Coding: Learning Cumulatively Online without Forgetting
Watch later
Alex Ororbia, …
Total of 0 viewers voted for saving the presentation to eternal vault which is 0.0%
HyperDomainNet: Universal Domain Adaptation for Generative Adversarial Networks
Watch later
Aibek Alanov, …
Total of 0 viewers voted for saving the presentation to eternal vault which is 0.0%
Bayesian Optimistic Optimization: Optimistic Exploration for Model-based Reinforcement Learning
Watch later
Chenyang Wu, …
Total of 0 viewers voted for saving the presentation to eternal vault which is 0.0%
Design Process is a Reinforcement Learning Problem
Watch later
Reza Kakooee, …
Total of 0 viewers voted for saving the presentation to eternal vault which is 0.0%
Is Conditional Generative Modeling all you need for Decision-Making?
Watch later
Anurag Ajay, …
Total of 3 viewers voted for saving the presentation to eternal vault which is 0.3%
On the SDEs and Scaling Rules for Adaptive Gradient Algorithms
Watch later
Total of 0 viewers voted for saving the presentation to eternal vault which is 0.0%