Activation Sharing with Asymmetric Paths Solves Weight Transport Problem without Bidirectional Connection
Dec 6, 2021
Speakers
Sunghyeon Woo
Speaker · 0 followers
Jeongwoo Park
Speaker · 0 followers
Jiwoo Hong
Speaker · 0 followers
Dongsuk Jeon
Speaker · 0 followers
About
One of the reasons why it is difficult for the brain to perform backpropagation (BP) is the weight transport problem, which argues forward and feedback neurons cannot share the same synaptic weights during learning in biological neural networks. Recently proposed algorithms address the weight transport problem while providing good performance similar to BP in large-scale networks. However, they require bidirectional connections between the forward and feedback neurons to train their weights, which is observed to be rare in general neural network structures in the biological brain. In this work, we propose an Activation Sharing algorithm that removes the need for bidirectional connections between the two types of neurons. In this algorithm, hidden layer outputs (activations) are shared across multiple layers during weight update. By applying the proposed learning rule to both forward and feedback networks, we solve the weight transport problem without the constraint of bidirectional connections, also achieving good performance even on deep convolutional neural networks, unlike other biologically plausible algorithms. In addition, our algorithm could significantly reduce memory access overhead when implemented in hardware.One of the reasons why it is difficult for the brain to perform backpropagation (BP) is the weight transport problem, which argues forward and feedback neurons cannot share the same synaptic weights during learning in biological neural networks. Recently proposed algorithms address the weight transport problem while providing good performance similar to BP in large-scale networks. However, they require bidirectional connections between the forward and feedback neurons to train their weights, whi…
Organizer
NeurIPS 2021
Account · 1.9k followers
About NeurIPS 2021
Neural Information Processing Systems (NeurIPS) is a multi-track machine learning and computational neuroscience conference that includes invited talks, demonstrations, symposia and oral and poster presentations of refereed papers. Following the conference, there are workshops which provide a less formal setting.
Like the format? Trust SlidesLive to capture your next event!
Professional recording and live streaming, delivered globally.
Sharing
Recommended Videos
Presentations on similar topic, category or speaker
Leveraging the Inductive Bias of Large Language Models for Abstract Textual Reasoning
Watch later
Christopher Michael Rytting, …
Total of 0 viewers voted for saving the presentation to eternal vault which is 0.0%
Permuton-induced Chinese Restaurant Process
Watch later
Total of 0 viewers voted for saving the presentation to eternal vault which is 0.0%
Incentrivizing Exploration in Reinforcement Learning
Watch later
Total of 0 viewers voted for saving the presentation to eternal vault which is 0.0%
The Complexity of Sparse Tensor PCA
Watch later
Davin Choo, …
Total of 0 viewers voted for saving the presentation to eternal vault which is 0.0%
GeoRE: A Relation Extraction Dataset for Chinese Geometry Problems
Watch later
Wei Yu, …
Total of 0 viewers voted for saving the presentation to eternal vault which is 0.0%
Real Robot Challenge: Learning Dexterous Manipulation on Real Robots in the Cloud
Watch later
Total of 0 viewers voted for saving the presentation to eternal vault which is 0.0%