Representation Costs of Linear Neural Networks: Analysis and Design
Dec 6, 2021
Speakers
Mina Karzand
Speaker · 0 followers
Nathan Srebro
Speaker · 0 followers
About
For different parameterizations (mappings from parameters to predictors), we study the regularization cost in predictor space induced by l_2 regularization on the parameters (weights). We focus on linear neural networks as parameterizations of linear predictors. We identify the representation cost of certain sparse linear ConvNets and residual networks. In order to get a better understanding of how the architecture and parameterization affect the representation cost, we also study the reverse problem, identifying which regularizers on linear predictors (e.g., l_p norms, group norms, the k-support-norm, elastic net) can be the representation cost induced by simple l_2 regularization, and designing the parameterizations that do so.For different parameterizations (mappings from parameters to predictors), we study the regularization cost in predictor space induced by l_2 regularization on the parameters (weights). We focus on linear neural networks as parameterizations of linear predictors. We identify the representation cost of certain sparse linear ConvNets and residual networks. In order to get a better understanding of how the architecture and parameterization affect the representation cost, we also study the reverse pr…
Organizer
NeurIPS 2021
Account · 1.9k followers
About NeurIPS 2021
Neural Information Processing Systems (NeurIPS) is a multi-track machine learning and computational neuroscience conference that includes invited talks, demonstrations, symposia and oral and poster presentations of refereed papers. Following the conference, there are workshops which provide a less formal setting.
Like the format? Trust SlidesLive to capture your next event!
Professional recording and live streaming, delivered globally.
Sharing
Recommended Videos
Presentations on similar topic, category or speaker
Obnline Lazy Subgradient is Universal on Strongly Convex Domains
Watch later
Total of 0 viewers voted for saving the presentation to eternal vault which is 0.0%
Beta-CROWN: Efficient Bound Propagation with Per-neuron Split Constraints for Neural Network Robustness Verification
Watch later
Shiqi Wang, …
Total of 0 viewers voted for saving the presentation to eternal vault which is 0.0%
Undivided Attention: Are Intermediate Layers Necessary for BERT?
Watch later
Total of 0 viewers voted for saving the presentation to eternal vault which is 0.0%
Appendix: Proofs and Derivations
Watch later
Total of 0 viewers voted for saving the presentation to eternal vault which is 0.0%
Mixture of Basis for Interpretable Continual Learning with Distribution Shifts
Watch later
Mengda Xu, …
Total of 1 viewers voted for saving the presentation to eternal vault which is 0.1%
A Regression Approach to Learning-Augmented Online Algorithms
Watch later
Keerti Anand, …
Total of 0 viewers voted for saving the presentation to eternal vault which is 0.0%