Fast Certified Robust Training with Short Warmup
Dec 6, 2021
Speakers
Zhouxing Shi
Speaker · 0 followers
Yihan Wang
Speaker · 0 followers
Huan Zhang
Speaker · 0 followers
Jinfeng Yi
Speaker · 0 followers
Cho-Jui Hsieh
Speaker · 1 follower
About
Recently, bound propagation based certified robust training methods have been proposed for training neural networks with certifiable robustness guarantees. Despite that state-of-the-art (SOTA) methods including interval bound propagation (IBP) and CROWN-IBP have per-batch training complexity similar to standard neural network training, they usually use a long warmup schedule with hundreds or thousands epochs to reach SOTA performance and are thus still costly. In this paper, we identify two important issues in existing methods, namely exploded bounds at initialization, and the imbalance in ReLU activation states. These two issues make certified training difficult and unstable, and thereby long warmup schedules were needed in prior works. To mitigate these issues and conduct certified training with shorter warmup, we propose three improvements: 1) We derive a new weight initialization method for IBP training; 2) We propose to fully add Batch Normalization (BN) to each layer in the model, since we find BN can reduce the imbalance in ReLU activation states; 3) We also design regularization to explicitly tighten certified bounds and balance ReLU activation states. In our experiments, we are able to obtain 65.03Recently, bound propagation based certified robust training methods have been proposed for training neural networks with certifiable robustness guarantees. Despite that state-of-the-art (SOTA) methods including interval bound propagation (IBP) and CROWN-IBP have per-batch training complexity similar to standard neural network training, they usually use a long warmup schedule with hundreds or thousands epochs to reach SOTA performance and are thus still costly. In this paper, we identify two impo…
Organizer
NeurIPS 2021
Account · 1.9k followers
About NeurIPS 2021
Neural Information Processing Systems (NeurIPS) is a multi-track machine learning and computational neuroscience conference that includes invited talks, demonstrations, symposia and oral and poster presentations of refereed papers. Following the conference, there are workshops which provide a less formal setting.
Like the format? Trust SlidesLive to capture your next event!
Professional recording and live streaming, delivered globally.
Sharing
Recommended Videos
Presentations on similar topic, category or speaker
Deep Generative Analysis for Task-Based fMRI Experiments
Watch later
Total of 0 viewers voted for saving the presentation to eternal vault which is 0.0%
NTopo: Mesh-free Topology Optimization using Implicit Neural Representations
Watch later
Total of 0 viewers voted for saving the presentation to eternal vault which is 0.0%
Developing Trustworthy AI for Weather and Climate
Watch later
Total of 0 viewers voted for saving the presentation to eternal vault which is 0.0%
Latent Equilibrium: Arbitrarily fast computation with arbitrarily slow neurons
Watch later
Paul Haider, …
Total of 0 viewers voted for saving the presentation to eternal vault which is 0.0%
ABC: Auxiliary Balanced Classifier for Class-imbalanced Semi-supervised Learning
Watch later
Hyuck Lee, …
Total of 0 viewers voted for saving the presentation to eternal vault which is 0.0%
Machine Learning For Numerical PDE : Scaling Law and Minimax Optimality
Watch later
Total of 0 viewers voted for saving the presentation to eternal vault which is 0.0%