The Limitations of Large Width in Neural Networks: A Deep Gaussian Process Perspective
6. prosince 2021
Řečníci
Geoff Pleiss
Řečník · 0 sledujících
John P. Cunningham
Řečník · 0 sledujících
O prezentaci
Large width limits have been a recent focus of deep learning research: modulo computational practicalities, do wider networks outperform narrower ones? Answering this question has been challenging, as conventional networks gain representational power with width, potentially masking any negative effects. Our analysis in this paper decouples capacity and width via the generalization of neural networks to Deep Gaussian Processes (Deep GP), a class of hierarchical models that subsume neural nets. In doing so, we aim to understand how width affects standard neural networks once they have sufficient capacity for a given modeling task. Our theoretical and empirical results on Deep GP suggest that large width is generally detrimental to hierarchical models. Surprisingly, we prove that even nonparametric Deep GP converge to Gaussian processes, effectively becoming shallower without any increase in representational power. The posterior, which corresponds to a mixture of data-adaptable basis functions, becomes less data-dependent with width. Our tail analysis demonstrates that width and depth have opposite effects: depth accentuates a model's non-Gaussianity, while width makes models increasingly Gaussian. We find there is a "sweet spot" that maximizes test set performance before the limiting GP behavior prevents adaptability, occurring at width = 1 or width = 2 for nonparametric Deep GP. These results make strong predictions about the same phenomenon in conventional neural networks: we show empirically that many neural network architectures need 10 - 500 hidden units for sufficient capacity - depending on the dataset - but further width degrades test performance.Large width limits have been a recent focus of deep learning research: modulo computational practicalities, do wider networks outperform narrower ones? Answering this question has been challenging, as conventional networks gain representational power with width, potentially masking any negative effects. Our analysis in this paper decouples capacity and width via the generalization of neural networks to Deep Gaussian Processes (Deep GP), a class of hierarchical models that subsume neural nets. In…
Organizátor
NeurIPS 2021
Účet · 1,9k sledujících
O organizátorovi (NeurIPS 2021)
Neural Information Processing Systems (NeurIPS) is a multi-track machine learning and computational neuroscience conference that includes invited talks, demonstrations, symposia and oral and poster presentations of refereed papers. Following the conference, there are workshops which provide a less formal setting.
Baví vás formát? Nechte SlidesLive zachytit svou akci!
Profesionální natáčení a streamování po celém světě.
Sdílení
Doporučená videa
Prezentace na podobné téma, kategorii nebo přednášejícího
Learning the optimal Tikhonov regularizer for inverse problems
Zhlédnout později
Luca Ratti, …
Pro uložení prezentace do věčného trezoru hlasovalo 0 diváků, což je 0.0 %
Implicit Semantic Response Alignment for Partial Domain Adaptation
Zhlédnout později
Wenxiao Xiao, …
Pro uložení prezentace do věčného trezoru hlasovalo 0 diváků, což je 0.0 %
A 3D Generative Model for Structure-Based Drug Design
Zhlédnout později
Shitong Luo, …
Pro uložení prezentace do věčného trezoru hlasovalo 0 diváků, což je 0.0 %
Data-Driven Offline Optimization for Architecting Hardware Accelerators
Zhlédnout později
Aviral Kumar, …
Pro uložení prezentace do věčného trezoru hlasovalo 0 diváků, což je 0.0 %
A Closer Look at the Worst-case Behavior of Multi-armed Bandit Algorithms
Zhlédnout později
Anand Kalvit, …
Pro uložení prezentace do věčného trezoru hlasovalo 0 diváků, což je 0.0 %
Basil: A Fast and Byzantine-Resilient Approach for Decentralized Training
Zhlédnout později
Pro uložení prezentace do věčného trezoru hlasovalo 1 diváků, což je 0.1 %