Going Beyond Linear Transformers with Recurrent Fast Weight Programmers
Dec 6, 2021
Speakers
Kazuki Irie
Speaker · 0 followers
Imanol Schlag
Speaker · 0 followers
Róbert Csordás
Speaker · 0 followers
Jürgen Schmidhuber
Speaker · 2 followers
About
Transformers with linearised attention ("linear Transformers") have demonstrated the practical scalability and effectiveness of outer product-based Fast Weight Programmers (FWPs) from the '90s. However, the original FWP formulation is more general than the one of linear Transformers: a slow neural network (NN) continually reprograms the weights of a fast NN with arbitrary NN architecture. In existing linear Transformers, both NNs are feedforward and consist of a single layer. Here we explore new variations by adding recurrence to the slow and fast nets. We evaluate our novel recurrent FWPs (RFWPs) on two synthetic algorithmic tasks (code execution and sequential ListOps), Wikitext-103 language models, and on the Atari 2600 2D game environment.Our models exhibit properties of Transformers and RNNs. In the reinforcement learning setting, we report large improvements over the LSTM in several Atari games. Our code is publicly available.Transformers with linearised attention ("linear Transformers") have demonstrated the practical scalability and effectiveness of outer product-based Fast Weight Programmers (FWPs) from the '90s. However, the original FWP formulation is more general than the one of linear Transformers: a slow neural network (NN) continually reprograms the weights of a fast NN with arbitrary NN architecture. In existing linear Transformers, both NNs are feedforward and consist of a single layer. Here we explore new…
Organizer
NeurIPS 2021
Account · 1.9k followers
About NeurIPS 2021
Neural Information Processing Systems (NeurIPS) is a multi-track machine learning and computational neuroscience conference that includes invited talks, demonstrations, symposia and oral and poster presentations of refereed papers. Following the conference, there are workshops which provide a less formal setting.
Like the format? Trust SlidesLive to capture your next event!
Professional recording and live streaming, delivered globally.
Sharing
Recommended Videos
Presentations on similar topic, category or speaker
PYLON: A PyTorch Framework for Learning with Constraints
Watch later
Kareem Ahmed, …
Total of 0 viewers voted for saving the presentation to eternal vault which is 0.0%
The Caltech Mouse Social Interactions 2021 Dataset
Watch later
Jennifer Sun, …
Total of 0 viewers voted for saving the presentation to eternal vault which is 0.0%
Variable Importance on Medical Images and Socio-Demographic Data
Watch later
Ahmad Chamma, …
Total of 0 viewers voted for saving the presentation to eternal vault which is 0.0%
Numerical influence of ReLU’(0) on backpropagation
Watch later
Total of 0 viewers voted for saving the presentation to eternal vault which is 0.0%
Invited talk 5: Jeremy Epstein
Watch later
Total of 0 viewers voted for saving the presentation to eternal vault which is 0.0%
On the Value of Interaction and Function Approximation in Imitation Learning
Watch later
Total of 0 viewers voted for saving the presentation to eternal vault which is 0.0%