Equivariant Architectures for Learning in Deep Weight Spaces
Jul 24, 2023
Speakers
Aviv Navon
Speaker · 1 follower
Aviv Shamsian
Speaker · 1 follower
Idan Achituve
Speaker · 0 followers
Ethan Fetaya
Speaker · 1 follower
Gal Chechik
Speaker · 0 followers
Haggai Maron
Speaker · 0 followers
About
Designing machine learning architectures for processing neural networks in their raw weight matrix form is a newly introduced research direction. Unfortunately, the unique symmetry structure of deep weight spaces makes this design very challenging. If successful, such architectures would be capable of performing a wide range of intriguing tasks, from adapting a pre-trained network to a new domainto editing objects represented as functions (INRs or NeRFs). As a first step towards this goal, we present here a novel network architecture for learning in deep weight spaces. It takes as input a concatenation of weights and biases of a pre-trained MLP and processes it using a composition of layers that are equivariant to the natural permutation symmetry of the MLP's weights: Changing the order of neurons in intermediate layers of the MLP does not affect the function it represents. We provide a full characterization of all affine equivariant and invariant layers for these symmetries and show how these layers can be implemented using three basic operations: pooling, broadcasting, and fully connected layers applied to the input in an appropriate manner. We demonstrate the effectiveness of our architecture and its advantages over natural baselines in a variety of learning tasks.Designing machine learning architectures for processing neural networks in their raw weight matrix form is a newly introduced research direction. Unfortunately, the unique symmetry structure of deep weight spaces makes this design very challenging. If successful, such architectures would be capable of performing a wide range of intriguing tasks, from adapting a pre-trained network to a new domainto editing objects represented as functions (INRs or NeRFs). As a first step towards this goal, we pr…
Like the format? Trust SlidesLive to capture your next event!
Professional recording and live streaming, delivered globally.
Sharing
Recommended Videos
Presentations on similar topic, category or speaker
When do minimax-fair learning and ERM coincide?
Watch later
The Power of Uniform Sampling for k-Median
Watch later
The Benefits of Model-Based Generalization in Reinforcement Learning
Watch later
Kenny Young, …
Stratified Adversarial Robustness with Rejection
Watch later
Jiefeng Chen, …
Model Ratatouille: Recycling Diverse Models for Out-of-Distribution Generalization
Watch later
