Differentiable Optimization of Generalized Nondecomposable Functions using Linear Programs
Dec 6, 2021
Speakers
Zihang Meng
Speaker · 0 followers
Lopamudra Mukherjee
Speaker · 0 followers
Vikas Singh
Speaker · 1 follower
Sathya Ravi
Speaker · 0 followers
About
We propose a framework which makes it feasible to directly train deep neural networks with respect to popular families of task-specific non-decomposable performance measures such as AUC, multi-class AUC, F-measure and others. A common feature of the optimization model that emerges from these tasks is that it involves solving a Linear Programs (LP) during training where representations learned by upstream layers characterize the constraints or the feasible set. The constraint matrix is not only large but the constraints are also modified at each iteration. We show how adopting a set of ingenious ideas proposed by Mangasarian for 1-norm SVMs – which advocates for solving LPs with a generalized Newton method – provides a simple and effective solution that can be run on the GPU. In particular, this strategy needs little unrolling, which makes it more efficient during backward pass. Further, even when the constraint matrix is too large to fit on the GPU memory (say large minibatch settings), we show that running the Newton method in a lower dimensional space yields accurate gradients for training, by utilizing a statistical concept called sufficient dimension reduction. While a number of specialized algorithms have been proposed for the models that we describe here, our module turns out to be applicable without any specific adjustments or relaxations. We describe each use case, study its properties and demonstrate the efficacy of the approach over alternatives which use surrogate lower bounds and often, specialized optimization schemes. Frequently, we achieve superior computational behavior and performance improvements on common datasets used in the literature.We propose a framework which makes it feasible to directly train deep neural networks with respect to popular families of task-specific non-decomposable performance measures such as AUC, multi-class AUC, F-measure and others. A common feature of the optimization model that emerges from these tasks is that it involves solving a Linear Programs (LP) during training where representations learned by upstream layers characterize the constraints or the feasible set. The constraint matrix is not only l…
Organizer
NeurIPS 2021
Account · 1.9k followers
About NeurIPS 2021
Neural Information Processing Systems (NeurIPS) is a multi-track machine learning and computational neuroscience conference that includes invited talks, demonstrations, symposia and oral and poster presentations of refereed papers. Following the conference, there are workshops which provide a less formal setting.
Like the format? Trust SlidesLive to capture your next event!
Professional recording and live streaming, delivered globally.
Sharing
Recommended Videos
Presentations on similar topic, category or speaker
Discussion panel: Curiosity
Watch later
Chelsea Finn, …
Total of 0 viewers voted for saving the presentation to eternal vault which is 0.0%
Test-Agnostic Long-Tailed Recognition by Test-Time Aggregating Diverse Experts
Watch later
Yifan Zhang, …
Total of 1 viewers voted for saving the presentation to eternal vault which is 0.1%
Reinforcement Learning in Real-World Control Systems
Watch later
Total of 0 viewers voted for saving the presentation to eternal vault which is 0.0%
Dynamic Causal Bayesian Optimization
Watch later
Total of 0 viewers voted for saving the presentation to eternal vault which is 0.0%
FakeAVCeleb: A Novel Audio-Video Multimodal Deepfake Dataset
Watch later
Hasam Khalid, …
Total of 1 viewers voted for saving the presentation to eternal vault which is 0.1%
Offline Reinforcement Learning
Watch later
Total of 0 viewers voted for saving the presentation to eternal vault which is 0.0%