Towards Gradient-based Bilevel Optimization with Non-convex Followers and Beyond
Dec 6, 2021
Speakers
Risheng Liu
Speaker · 0 followers
Yaohua Liu
Speaker · 0 followers
Shangzhi Zeng
Speaker · 0 followers
About
In recent years, Bi-Level Optimization (BLO) techniques have received extensive attentions from both learning and vision communities. A variety of BLO models in complex and practical tasks are of non-convex follower structure in nature (a.k.a., without Lower-Level Convexity, LLC for short). However, this challenging class of BLOs is lack of developments on both efficient solution strategies and solid theoretical guarantees. In this work, we propose a new algorithmic framework, named Initialization Auxiliary and Pessimistic Trajectory Truncated Gradient Method (IAPTT-GM), to partially address the above issues. In particular, by introducing an auxiliary as initialization to guide the optimization dynamics and designing a pessimistic trajectory truncation operation, we construct a reliable approximate version of the original BLO in the absence of LLC hypothesis. Our theoretical investigations establish the convergence of solutions returned by IAPTT-GM towards those of the original BLO without LLC. As an additional bonus, we also theoretically justify the quality of our IAPTT-GM embedded with Nesterov's accelerated dynamics under LLC. The experimental results confirm both the convergence of our algorithm without LLC, and the theoretical findings under LLC.In recent years, Bi-Level Optimization (BLO) techniques have received extensive attentions from both learning and vision communities. A variety of BLO models in complex and practical tasks are of non-convex follower structure in nature (a.k.a., without Lower-Level Convexity, LLC for short). However, this challenging class of BLOs is lack of developments on both efficient solution strategies and solid theoretical guarantees. In this work, we propose a new algorithmic framework, named Initializati…
Organizer
NeurIPS 2021
Account · 1.9k followers
About NeurIPS 2021
Neural Information Processing Systems (NeurIPS) is a multi-track machine learning and computational neuroscience conference that includes invited talks, demonstrations, symposia and oral and poster presentations of refereed papers. Following the conference, there are workshops which provide a less formal setting.
Like the format? Trust SlidesLive to capture your next event!
Professional recording and live streaming, delivered globally.
Sharing
Recommended Videos
Presentations on similar topic, category or speaker
Practical Near Neighbor Search via Group Testing
Watch later
Total of 0 viewers voted for saving the presentation to eternal vault which is 0.0%
Disentangling Identifiable Features from Noisy Data with Structured Nonlinear ICA
Watch later
Total of 0 viewers voted for saving the presentation to eternal vault which is 0.0%
Scalable Quasi-Bayesian Instrumental Variable Regression
Watch later
Ziyu Wang, …
Total of 0 viewers voted for saving the presentation to eternal vault which is 0.0%
No Fear of Heterogeneity: Classifier Calibration for Federated Learning with Non-IID Data
Watch later
Mi Luo, …
Total of 0 viewers voted for saving the presentation to eternal vault which is 0.0%
As easy as APC: Overcoming missing data and class imbalance in time series with self-supervised learning
Watch later
Total of 0 viewers voted for saving the presentation to eternal vault which is 0.0%
Multi-Agent Reinforcement Learning in Stochastic Networked Systems
Watch later
Yiheng Lin, …
Total of 0 viewers voted for saving the presentation to eternal vault which is 0.0%