ResNEsts and DenseNEsts: Block-based DNN Models with Improved Representation Guarantees
Dec 6, 2021
Speakers
Kuan-Lin Chen
Speaker · 0 followers
Ching-Hua Lee
Speaker · 0 followers
Harinath Garudadri
Speaker · 0 followers
Bhaskar D. Rao
Speaker · 0 followers
About
Models recently used in the literature proving residual networks (ResNets) are better than linear predictors are actually different from standard ResNets that have been widely used in computer vision. In addition to the assumptions such as scalar-valued output or single residual block, the models fundamentally considered in the literature have no nonlinearities at the final residual representation that feeds into the final affine layer. To codify such a difference in nonlinearities and reveal a linear estimation property, we define ResNEsts, i.e., Residual Nonlinear Estimators, by simply dropping nonlinearities at the last residual representation from standard ResNets. We show that wide ResNEsts with bottleneck blocks can always guarantee a very desirable training property that standard ResNets aim to achieve, i.e., adding more blocks does not decrease performance given the same set of basis elements. To prove that, we first recognize ResNEsts are basis function models that are limited by a coupling problem in basis learning and linear prediction. Then, to decouple prediction weights from basis learning, we construct a special architecture termed augmented ResNEst (A-ResNEst) that always guarantees no worse performance with the addition of a block. As a result, such an A-ResNEst establishes empirical risk lower bounds for a ResNEst using corresponding bases. Our results demonstrate ResNEsts indeed have a problem of diminishing feature reuse; however, it can be avoided by sufficiently expanding or widening the input space, leading to the above-mentioned desirable property. Inspired by the densely connected networks (DenseNets) that have been shown to outperform ResNets, we also propose a corresponding new model called Densely connected Nonlinear Estimator (DenseNEst). We show that any DenseNEst can be represented as a wide ResNEst with bottleneck blocks. Unlike ResNEsts, DenseNEsts exhibit the desirable property without any special architectural re-design.Models recently used in the literature proving residual networks (ResNets) are better than linear predictors are actually different from standard ResNets that have been widely used in computer vision. In addition to the assumptions such as scalar-valued output or single residual block, the models fundamentally considered in the literature have no nonlinearities at the final residual representation that feeds into the final affine layer. To codify such a difference in nonlinearities and reveal a…
Organizer
NeurIPS 2021
Account · 1.9k followers
About NeurIPS 2021
Neural Information Processing Systems (NeurIPS) is a multi-track machine learning and computational neuroscience conference that includes invited talks, demonstrations, symposia and oral and poster presentations of refereed papers. Following the conference, there are workshops which provide a less formal setting.
Like the format? Trust SlidesLive to capture your next event!
Professional recording and live streaming, delivered globally.
Sharing
Recommended Videos
Presentations on similar topic, category or speaker
Framing RNN as a kernel method: A neural ODE approach
Watch later
Total of 0 viewers voted for saving the presentation to eternal vault which is 0.0%
Select, Label, and Mix: Learning Discriminative Invariant Feature Representations for Partial Domain Adaptation
Watch later
Total of 0 viewers voted for saving the presentation to eternal vault which is 0.0%
Dynamic Inference with Neural Interpreters
Watch later
Total of 0 viewers voted for saving the presentation to eternal vault which is 0.0%
On Stochastic Stability of Deep Markov Models
Watch later
Jan Drgona, …
Total of 0 viewers voted for saving the presentation to eternal vault which is 0.0%
Counterexample Guided RL Policy Refinement Using Bayesian Optimization
Watch later
Total of 1 viewers voted for saving the presentation to eternal vault which is 0.1%
Vector-valued Gaussian Processes on Riemannian Manifolds via Gauge-Equivariant Projected Kernels
Watch later
Total of 0 viewers voted for saving the presentation to eternal vault which is 0.0%