One Layer is All You Need
Nov 28, 2022
Speakers
About
A deeper network structure generally handles more complicated non-linearity and performs more competitively. Nowadays, advanced network designs often contain a large number of repetitive structures (e.g., Transformer). They empower the network capacity to a new level but also increase the model size inevitably, which is unfriendly to either model restoring or transferring. In this study, we are the first to investigate the representative potential of fixed random weights with limited unique values by iteratively learning different masks, leading to a new paradigm for model compression to diminish the model size. Concretely, we utilize one random initialized layer, accompanied with different masks, to convey different feature mappings and represent repetitive modules in a deep network. As a result, the model can be expressed as one-layer with a bunch of masks, which significantly reduce the model storage cost. Furthermore, we enhance our strategy by learning masks for a model filled by padding a given random weights sequence. In this way, our method can further lower the space complexity, especially for models without many repetitive architectures. We validate the potential of leveraging random weights and test our compression paradigm based on different network architectures.A deeper network structure generally handles more complicated non-linearity and performs more competitively. Nowadays, advanced network designs often contain a large number of repetitive structures (e.g., Transformer). They empower the network capacity to a new level but also increase the model size inevitably, which is unfriendly to either model restoring or transferring. In this study, we are the first to investigate the representative potential of fixed random weights with limited unique valu…
Organizer
NeurIPS 2022
Account · 952 followers
Like the format? Trust SlidesLive to capture your next event!
Professional recording and live streaming, delivered globally.
Sharing
Recommended Videos
Presentations on similar topic, category or speaker
Addressing Resource Scarcity across Sign Languages with Multilingual Pretraining and Unified-Vocabulary Datasets
Watch later
Gokul NC, …
Total of 0 viewers voted for saving the presentation to eternal vault which is 0.0%
Cross-Domain MetaDL: Any-Way Any-Shot Learning Competition with Novel Datasets from Practical Domains
Watch later
Total of 0 viewers voted for saving the presentation to eternal vault which is 0.0%
Synthetic Data Research at J.P. Morgan AI Research
Watch later
Total of 0 viewers voted for saving the presentation to eternal vault which is 0.0%
GLIPv2: Unifying Localization and Vision-Language Understanding
Watch later
Total of 0 viewers voted for saving the presentation to eternal vault which is 0.0%
Posterior samples of source galaxies in strong gravitational lenses with score-based priors
Watch later
Total of 0 viewers voted for saving the presentation to eternal vault which is 0.0%
RISE: Robust Individualized Decision Learning with Sensitive Variables
Watch later
Total of 0 viewers voted for saving the presentation to eternal vault which is 0.0%