Divide and Conquer for Quantization: Leveraging Intermediate Feature Representations for Quantized Training of Neural Networks
Jul 12, 2020
Speakers
Ahmed T. Elthakeb
Speaker · 0 followers
Prannoy Pilligundla
Speaker · 0 followers
FatemehSadat Mireshghallah
Speaker · 0 followers
Alexander Cloninger
Speaker · 0 followers
Hadi Esmaeilzadeh
Speaker · 0 followers
About
The deep layers of modern neural networks extract a rather rich set of features as an input propagates through the network. This paper sets out to harvest these rich intermediate representations for quantization with minimal accuracy loss while significantly reducing the memory footprint and compute intensity of the DNN. This paper utilizes knowledge distillation through teacher-student paradigm (Hinton et al., 2015) in a novel setting that exploits the feature extraction capability of DNNs for higher-accuracy quantization. As such, our algorithm logically divides a pretrained full-precision DNN to multiple sections, each of which exposes intermediate features to train a team of students independently in the quantized domain. This divide and conquer strategy, in fact, makes the training of each student section possible in isolation while all these independently trained sections are later stitched together to form the equivalent fully quantized network. Our algorithm is a sectional approach towards knowledge distillation and is not treating the intermediate representation as a hint for pretraining before one knowledge distillation pass over the entire network (Romero et al., 2015). Experiments on various DNNs (AlexNet, LeNet, ResNet-18, ResNet-20, SVHN and VGG-11) show that, on average, this approach—called DCQ (Divide and Conquer Quantization)—on average closes the accuracy gap between a state-of-the-art quantized training technique, DoReFa-Net (Zhou et al., 2016) and the full-precision runs by 85The deep layers of modern neural networks extract a rather rich set of features as an input propagates through the network. This paper sets out to harvest these rich intermediate representations for quantization with minimal accuracy loss while significantly reducing the memory footprint and compute intensity of the DNN. This paper utilizes knowledge distillation through teacher-student paradigm (Hinton et al., 2015) in a novel setting that exploits the feature extraction capability of DNNs for…
Categories
AI & Data Science
Category · 10.8k presentations
About ICML 2020
The International Conference on Machine Learning (ICML) is the premier gathering of professionals dedicated to the advancement of the branch of artificial intelligence known as machine learning. ICML is globally renowned for presenting and publishing cutting-edge research on all aspects of machine learning used in closely related areas like artificial intelligence, statistics and data science, as well as important application areas such as machine vision, computational biology, speech recognition, and robotics. ICML is one of the fastest growing artificial intelligence conferences in the world. Participants at ICML span a wide range of backgrounds, from academic and industrial researchers, to entrepreneurs and engineers, to graduate students and postdocs.
Like the format? Trust SlidesLive to capture your next event!
Professional recording and live streaming, delivered globally.
Sharing
Recommended Videos
Presentations on similar topic, category or speaker
Implicit Class-Conditioned Domain Alignment for Unsupervised Domain Adaptation
Watch later
Xiang Jiang, …
Implicit Generative Modeling for Efficient Exploration
Watch later
Buying Data over Time
Watch later
Endless Frontiers?
Watch later
Ordering Dimensions with Nested Dropout Normalising Flows
Watch later
Ian Murray, …