TorchOpt: An Efficient Library for Differentiable Optimization
Dec 2, 2022
Speakers
Xidong Feng
Speaker · 0 followers
Xuehai Pan
Speaker · 0 followers
Yaodong Yang
Speaker · 1 follower
About
Recent years have witnessed the booming of various differentiable optimization algorithms. These algorithms exhibit different execution patterns, and their execution needs massive computational resources that go beyond a single CPU and GPU. Existing differentiable optimization libraries, however, cannot support efficient algorithm development and multi-CPU/GPU execution, making the development of differentiable optimization algorithms often cumbersome and expensive.This paper introduces TorchOpt, a PyTorch-based efficient library for differentiable optimization. TorchOpt provides a unified and expressive bi-level optimization programming abstraction. This abstraction allows users to efficiently declare and analyze various differentiable optimization programs with explicit gradients, implicit gradients, and zero-order gradients. TorchOpt further provides a high-performance distributed execution runtime. This runtime can fully parallelize computation-intensive differentiation operations (e.g. tensor tree flatten) on CPUs/GPUs and automatically distribute computation to distributed devices. Experimental results show that TorchOpt outperforms state-of-the-art libraries by 7× on an 8-GPU server. TorchOpt will be open source with a permissive Apache-2.0 License.Recent years have witnessed the booming of various differentiable optimization algorithms. These algorithms exhibit different execution patterns, and their execution needs massive computational resources that go beyond a single CPU and GPU. Existing differentiable optimization libraries, however, cannot support efficient algorithm development and multi-CPU/GPU execution, making the development of differentiable optimization algorithms often cumbersome and expensive.This paper introduces TorchOpt…
Organizer
NeurIPS 2022
Account · 960 followers
Like the format? Trust SlidesLive to capture your next event!
Professional recording and live streaming, delivered globally.
Sharing
Recommended Videos
Presentations on similar topic, category or speaker
Approaches to Optimizing Medical Treatment Policy using Temporal Causal Model-Based Simulation
Watch later
Total of 0 viewers voted for saving the presentation to eternal vault which is 0.0%
Stable Diffusion and Friends
Watch later
Total of 0 viewers voted for saving the presentation to eternal vault which is 0.0%
DTG-SSOD: Dense Teacher Guidance for Semi-Supervised Object Detection
Watch later
Gang Li, …
Total of 0 viewers voted for saving the presentation to eternal vault which is 0.0%
Communication Efficient Distributed Learning for Kernelized Contextual Bandits
Watch later
Chuanhao Li, …
Total of 0 viewers voted for saving the presentation to eternal vault which is 0.0%
On Analyzing Generative and Denoising Capabilities of Diffusion-based Deep Generative Models
Watch later
Kamil Deja, …
Total of 0 viewers voted for saving the presentation to eternal vault which is 0.0%
Zeroth-Order Negative Curvature Finding: Escaping Saddle Points without Gradients
Watch later
Hualin Zhang, …
Total of 0 viewers voted for saving the presentation to eternal vault which is 0.0%