Dynamic Tensor Rematerialization
May 3, 2021
Speakers
Marisa Kirisame
Sprecher:in · 0 Follower:innen
Steven Lyubomirsky
Sprecher:in · 0 Follower:innen
Altan Haan
Sprecher:in · 0 Follower:innen
Jennifer R Brennan
Sprecher:in · 0 Follower:innen
Jared Roesch
Sprecher:in · 0 Follower:innen
Tianqi Chen
Sprecher:in · 2 Follower:innen
Zachary Tatlock
Sprecher:in · 0 Follower:innen
About
Checkpointing enables training deep learning models under restricted memory budgets by freeing intermediate activations from memory and recomputing them on demand. Previous checkpointing techniques statically plan these recomputations offline and assume static computation graphs. We demonstrate that a simple online algorithm can achieve comparable performance by introducing Dynamic Tensor Rematerialization (DTR), a greedy online algorithm for checkpointing that is extensible and general, is parameterized by eviction policy, and supports dynamic models. We prove that DTR can train an $N$-layer linear feedforward network on an $\Omega(\smash{\sqrt{N}})$ memory budget with only $\mathcal{O}(N)$ tensor operations. DTR closely matches the performance of optimal static checkpointing in simulated experiments. We incorporate a DTR prototype into PyTorch just by interposing on tensor allocations and operator calls and collecting lightweight metadata on tensors.Checkpointing enables training deep learning models under restricted memory budgets by freeing intermediate activations from memory and recomputing them on demand. Previous checkpointing techniques statically plan these recomputations offline and assume static computation graphs. We demonstrate that a simple online algorithm can achieve comparable performance by introducing Dynamic Tensor Rematerialization (DTR), a greedy online algorithm for checkpointing that is extensible and general, is para…
About ICLR 2021
The International Conference on Learning Representations (ICLR) is the premier gathering of professionals dedicated to the advancement of the branch of artificial intelligence called representation learning, but generally referred to as deep learning. ICLR is globally renowned for presenting and publishing cutting-edge research on all aspects of deep learning used in the fields of artificial intelligence, statistics and data science, as well as important application areas such as machine vision, computational biology, speech recognition, text understanding, gaming, and robotics.
Like the format? Trust SlidesLive to capture your next event!
Professional recording and live streaming, delivered globally.
Sharing
Recommended Videos
Presentations on similar topic, category or speaker
Learning 3D Granular Flow Simulations
Später ansehen
Andreas Mayr, …
EigenGame: PCA as a Nash Equilibrium
Später ansehen
Ian Gemp, …
ICLR 2021 Workshop on Embodied Multimodal Learning (EML)
Später ansehen
On the mapping between Hopfield networks and Restricted Boltzmann Machines
Später ansehen
Saliency is a Possible Red Herring When Diagnosing Poor Generalization
Später ansehen
