Multi-Prize Lottery Ticket Hypothesis: Finding Accurate Binary Neural Networks by Pruning A Randomly Weighted Network
May 3, 2021
Speakers
James Diffenderfer
Speaker · 0 followers
Bhavya Kailkhura
Speaker · 0 followers
About
Recently, \cite{frankle2018lottery} demonstrated that randomly-initialized dense networks contain subnetworks that once found can be trained to reach test accuracy comparable to the trained dense network. However, finding these high performing trainable subnetworks is expensive, requiring iterative process of training and pruning weights. In this paper, we propose (and prove) a stronger \emph{Multi-Prize Lottery Ticket Hypothesis}: \emph{A sufficiently over-parameterized neural network with random weights contains several subnetworks (winning tickets) that (a) have comparable accuracy to a dense target network with learned weights (prize 1), (b) do not require any further training to achieve prize 1 (prize 2), and (c) is robust to extreme forms of quantization (i.e., binary weights and/or activation) (prize 3).} \noindent This provides a new paradigm for learning compact yet highly accurate binary neural networks by pruning and quantizing randomly weighted full precision neural networks. These multi-prize tickets enjoy a number of desirable properties including drastically reduced memory size, faster test-time inference, and lower power consumption compared to their dense and full-precision counterparts. Furthermore, we propose an algorithm for finding multi-prize tickets and test it by performing a series of experiments on CIFAR-10 and ImageNet datasets. Empirical results indicate that as models grow deeper and wider, untrained multi-prize tickets start to reach similar (and sometimes even higher) test accuracy compared to their significantly larger and full-precision counterparts that have been weight-trained. With minimal hyperparameter tuning, our binary weight multi-prize tickets outperform current state-of-the-art in binary neural networks.Recently, \cite{frankle2018lottery} demonstrated that randomly-initialized dense networks contain subnetworks that once found can be trained to reach test accuracy comparable to the trained dense network. However, finding these high performing trainable subnetworks is expensive, requiring iterative process of training and pruning weights. In this paper, we propose (and prove) a stronger \emph{Multi-Prize Lottery Ticket Hypothesis}: \emph{A sufficiently over-parameterized neural network with ran…
About ICLR 2021
The International Conference on Learning Representations (ICLR) is the premier gathering of professionals dedicated to the advancement of the branch of artificial intelligence called representation learning, but generally referred to as deep learning. ICLR is globally renowned for presenting and publishing cutting-edge research on all aspects of deep learning used in the fields of artificial intelligence, statistics and data science, as well as important application areas such as machine vision, computational biology, speech recognition, text understanding, gaming, and robotics.
Like the format? Trust SlidesLive to capture your next event!
Professional recording and live streaming, delivered globally.
Sharing
Recommended Videos
Presentations on similar topic, category or speaker
The Recurrent Neural Tangent Kernel
Watch later
Plan-Based Relaxed Reward Shaping for Goal-Directed Tasks
Watch later
Motivational System
Watch later
Workshop on Distributed and Private Machine Learning
Watch later
Bayesian Context Aggregation for Neural Processes
Watch later
Synthetic Data for Model Selection
Watch later
Matan Fitz, …