Generalization bounds via distillation
May 3, 2021
Speakers
Daniel Hsu
Speaker · 1 follower
Matus Telgarsky
Speaker · 0 followers
About
This paper provides a suite of mathematical tools to bound the generalization error of networks that possess low-complexity distillations --- that is, when there exist simple networks whose softmax outputs approximately match those of the original network. The primary contribution is the aforementioned bound, which upper bounds the test error of a network by the sum of its training error, the distillation error, and the complexity of the distilled network. Supporting this, secondary contributions include: a generalization bound which can handle convolutions and skip connections, a generalization analysis of the compression step leading to a bound with small width- and depth-dependence via weight matrix stable ranks, and a sampling theorem to sparsify dense networks. The bounds and their behavior are illustrated empirically on the standard mnist and cifar datasets.This paper provides a suite of mathematical tools to bound the generalization error of networks that possess low-complexity distillations --- that is, when there exist simple networks whose softmax outputs approximately match those of the original network. The primary contribution is the aforementioned bound, which upper bounds the test error of a network by the sum of its training error, the distillation error, and the complexity of the distilled network. Supporting this, secondary contribution…
Categories
AI & Data Science
Category · 10.8k presentations
About ICLR 2021
The International Conference on Learning Representations (ICLR) is the premier gathering of professionals dedicated to the advancement of the branch of artificial intelligence called representation learning, but generally referred to as deep learning. ICLR is globally renowned for presenting and publishing cutting-edge research on all aspects of deep learning used in the fields of artificial intelligence, statistics and data science, as well as important application areas such as machine vision, computational biology, speech recognition, text understanding, gaming, and robotics.
Like the format? Trust SlidesLive to capture your next event!
Professional recording and live streaming, delivered globally.
Sharing
Recommended Videos
Presentations on similar topic, category or speaker
Probing BERT in Hyperbolic Spaces
Watch later
Boli Chen, …
Learning Reasoning Paths over Semantic Graphs for Video-grounded Dialogues
Watch later
Hung Le, …
Towards Faster and Stabilized GAN Training for High-Fidelity Few-Shot Image Synthesis
Watch later
Bingchen Liu, …
NONSEPARABLE SYMPLECTIC NEURAL NETWORKS
Watch later
