Improve Object Detection with Feature-based Knowledge Distillation: Towards Accurate and Efficient Detectors
May 3, 2021
Speakers
Linfeng Zhang
Speaker · 0 followers
Kaisheng Ma
Speaker · 0 followers
About
Knowledge distillation, in which a student model is trained to mimic a teacher model, has been proved as an effective technique for model compression and model accuracy boosting. However, most knowledge distillation methods, designed for image classification, have failed on more challenging tasks, such as object detection. In this paper, we suggest that the failure of knowledge distillation on object detection is mainly caused by two reasons: (1) the imbalance between pixels of foreground and background and (2) lack of distillation on the relation between different pixels. Observing the above reasons, we propose attention-guided distillation and non-local distillation to address the two problems, respectively. Attention-guided distillation is proposed to find the crucial pixels of foreground objects with attention mechanism and then make the students take more effort to learn their features. Non-local distillation is proposed to enable students to learn not only the feature of an individual pixel but also the relation between different pixels captured by non-local modules. Experiments show that our methods achieve excellent AP improvements on both one-stage and two-stage, both anchor-based and anchor-free detectors. For example, Faster RCNN101 with our distillation achieves 43.9 AP on COCO2017, which is 4.1 higher than the baseline. Codes will be released on Github.Knowledge distillation, in which a student model is trained to mimic a teacher model, has been proved as an effective technique for model compression and model accuracy boosting. However, most knowledge distillation methods, designed for image classification, have failed on more challenging tasks, such as object detection. In this paper, we suggest that the failure of knowledge distillation on object detection is mainly caused by two reasons: (1) the imbalance between pixels of foreground and ba…
Categories
AI & Data Science
Category · 10.8k presentations
About ICLR 2021
The International Conference on Learning Representations (ICLR) is the premier gathering of professionals dedicated to the advancement of the branch of artificial intelligence called representation learning, but generally referred to as deep learning. ICLR is globally renowned for presenting and publishing cutting-edge research on all aspects of deep learning used in the fields of artificial intelligence, statistics and data science, as well as important application areas such as machine vision, computational biology, speech recognition, text understanding, gaming, and robotics.
Like the format? Trust SlidesLive to capture your next event!
Professional recording and live streaming, delivered globally.
Sharing
Recommended Videos
Presentations on similar topic, category or speaker
Few-Shot Bayesian Optimization with Deep Kernel Surrogates
Watch later
Does Enhanced Shape Bias Improve Neural Network Robustness to Common Corruptions?
Watch later
Oral Session 1 - QA 3
Watch later
Continuous Coordination As a Realistic Scenario For Lifelong Learning
Watch later
Hadi Nekoei, …
Learning Advanced Mathematical Computations from Examples
Watch later
