Improve Object Detection with Feature-based Knowledge Distillation: Towards Accurate and Efficient Detectors
May 3, 2021
Speakers
Linfeng Zhang
Speaker · 0 followers
Kaisheng Ma
Speaker · 0 followers
About
Knowledge distillation, in which a student model is trained to mimic a teacher model, has been proved as an effective technique for model compression and model accuracy boosting. However, most knowledge distillation methods, designed for image classification, have failed on more challenging tasks, such as object detection. In this paper, we suggest that the failure of knowledge distillation on object detection is mainly caused by two reasons: (1) the imbalance between pixels of foreground and background and (2) lack of distillation on the relation between different pixels. Observing the above reasons, we propose attention-guided distillation and non-local distillation to address the two problems, respectively. Attention-guided distillation is proposed to find the crucial pixels of foreground objects with attention mechanism and then make the students take more effort to learn their features. Non-local distillation is proposed to enable students to learn not only the feature of an individual pixel but also the relation between different pixels captured by non-local modules. Experiments show that our methods achieve excellent AP improvements on both one-stage and two-stage, both anchor-based and anchor-free detectors. For example, Faster RCNN101 with our distillation achieves 43.9 AP on COCO2017, which is 4.1 higher than the baseline. Codes will be released on Github.Knowledge distillation, in which a student model is trained to mimic a teacher model, has been proved as an effective technique for model compression and model accuracy boosting. However, most knowledge distillation methods, designed for image classification, have failed on more challenging tasks, such as object detection. In this paper, we suggest that the failure of knowledge distillation on object detection is mainly caused by two reasons: (1) the imbalance between pixels of foreground and ba…
Categories
AI & Data Science
Category · 10.8k presentations
About ICLR 2021
The International Conference on Learning Representations (ICLR) is the premier gathering of professionals dedicated to the advancement of the branch of artificial intelligence called representation learning, but generally referred to as deep learning. ICLR is globally renowned for presenting and publishing cutting-edge research on all aspects of deep learning used in the fields of artificial intelligence, statistics and data science, as well as important application areas such as machine vision, computational biology, speech recognition, text understanding, gaming, and robotics.
Like the format? Trust SlidesLive to capture your next event!
Professional recording and live streaming, delivered globally.
Sharing
Recommended Videos
Presentations on similar topic, category or speaker
CaPC Learning: Confidential & Private Collaborative Learning
Watch later
Shift Invariance Can Reduce Adversarial Robustness
Watch later
Songwei Ge, …
Topology-Aware Segmentation Using Discrete Morse Theory
Watch later
Xiaoling Hu, …
Fast and Slow Learning of Recurrent Independent Mechanisms
Watch later
Kanika Madan, …
Sparse Quantized Spectral Clustering
Watch later
Zhenyu Liao, …
