Activation Sharing with Asymmetric Paths Solves Weight Transport Problem without Bidirectional Connection
Dec 6, 2021
Speakers
Sunghyeon Woo
Speaker · 0 followers
Jeongwoo Park
Speaker · 0 followers
Jiwoo Hong
Speaker · 0 followers
Dongsuk Jeon
Speaker · 0 followers
About
One of the reasons why it is difficult for the brain to perform backpropagation (BP) is the weight transport problem, which argues forward and feedback neurons cannot share the same synaptic weights during learning in biological neural networks. Recently proposed algorithms address the weight transport problem while providing good performance similar to BP in large-scale networks. However, they require bidirectional connections between the forward and feedback neurons to train their weights, which is observed to be rare in general neural network structures in the biological brain. In this work, we propose an Activation Sharing algorithm that removes the need for bidirectional connections between the two types of neurons. In this algorithm, hidden layer outputs (activations) are shared across multiple layers during weight update. By applying the proposed learning rule to both forward and feedback networks, we solve the weight transport problem without the constraint of bidirectional connections, also achieving good performance even on deep convolutional neural networks, unlike other biologically plausible algorithms. In addition, our algorithm could significantly reduce memory access overhead when implemented in hardware.One of the reasons why it is difficult for the brain to perform backpropagation (BP) is the weight transport problem, which argues forward and feedback neurons cannot share the same synaptic weights during learning in biological neural networks. Recently proposed algorithms address the weight transport problem while providing good performance similar to BP in large-scale networks. However, they require bidirectional connections between the forward and feedback neurons to train their weights, whi…
Organizer
NeurIPS 2021
Account · 1.9k followers
About NeurIPS 2021
Neural Information Processing Systems (NeurIPS) is a multi-track machine learning and computational neuroscience conference that includes invited talks, demonstrations, symposia and oral and poster presentations of refereed papers. Following the conference, there are workshops which provide a less formal setting.
Like the format? Trust SlidesLive to capture your next event!
Professional recording and live streaming, delivered globally.
Sharing
Recommended Videos
Presentations on similar topic, category or speaker
Change Point Detection via Multivariate Singular Spectrum Analysis
Watch later
Total of 0 viewers voted for saving the presentation to eternal vault which is 0.0%
Do Different Tracking Tasks Require Different Appearance Models?
Watch later
Total of 0 viewers voted for saving the presentation to eternal vault which is 0.0%
DR3: Value-Based Deep Reinforcement Learning Requires Explicit Regularization
Watch later
Aviral Kumar, …
Total of 0 viewers voted for saving the presentation to eternal vault which is 0.0%
Can we have it all? On the Trade-off between Spatial and Adversarial Robustness of Neural Networks
Watch later
Total of 0 viewers voted for saving the presentation to eternal vault which is 0.0%
Offline Off-Policy RL for Hospital Patients when Every Patient is Different
Watch later
Total of 0 viewers voted for saving the presentation to eternal vault which is 0.0%
Optimal Uniform OPE and Model-based Offline Reinforcement Learning in Time-Homogeneous, Reward-Free and Task-Agnostic Settings
Watch later
Ming Yin, …
Total of 0 viewers voted for saving the presentation to eternal vault which is 0.0%