Predictive Coding beyond Gaussian Distributions
Nov 28, 2022
Speakers
Luca Pinchetti
Speaker · 0 followers
Tommaso Salvatori
Speaker · 0 followers
Yordan Yordanov
Speaker · 0 followers
Beren Millidge
Speaker · 0 followers
Yuhang Song
Speaker · 0 followers
Thomas Lukasiewicz
Speaker · 2 followers
About
A large amount of recent research has the far-reaching goal of finding training methods for deep neural networks that can serve as alternatives to backpropagation (BP). A prominent example is predictive coding (PC), which is a neuroscience-inspired method that performs inference on hierarchical Gaussian generative models. These methods, however, fail to keep up with modern neural networks, as they are unable to replicate the dynamics of complex layers and activation functions. In this work, we solve this problem by generalizing PC to arbitrary probability distributions, enabling the training of architectures, such as transformers, that are hard to approximate with only Gaussian assumptions. We perform three experimental analyses. First, we study the gap between our method and the standard formulation of PC on multiple toy examples. Second, we test the reconstruction quality on variational autoencoders, where our method reaches the same reconstruction quality as BP. Third, we show that our method allows us to train transformer networks and achieve performance comparable with BP on conditional language models. More broadly, this method allows neuroscience-inspired learning to be applied to multiple domains, since the internal distributions can be flexibly adapted to the data, tasks, and architectures used.A large amount of recent research has the far-reaching goal of finding training methods for deep neural networks that can serve as alternatives to backpropagation (BP). A prominent example is predictive coding (PC), which is a neuroscience-inspired method that performs inference on hierarchical Gaussian generative models. These methods, however, fail to keep up with modern neural networks, as they are unable to replicate the dynamics of complex layers and activation functions. In this work…
Organizer
NeurIPS 2022
Account · 952 followers
Like the format? Trust SlidesLive to capture your next event!
Professional recording and live streaming, delivered globally.
Sharing
Recommended Videos
Presentations on similar topic, category or speaker
The geometry of hidden representations of protein language models
Watch later
Total of 0 viewers voted for saving the presentation to eternal vault which is 0.0%
Exploiting Variable Correlation with Masked Modeling for Anomaly Detection in Time Series
Watch later
Total of 0 viewers voted for saving the presentation to eternal vault which is 0.0%
A Coupled Design of Exploiting Record Similarity for Vertical Federated Learning
Watch later
Zhaomin Wu, …
Total of 0 viewers voted for saving the presentation to eternal vault which is 0.0%
Attracting and Dispersing: A Simple Approach for Source-free Domain Adaptation
Watch later
Shiqi Yang, …
Total of 0 viewers voted for saving the presentation to eternal vault which is 0.0%
Fast-BEV: Towards Real-time On-vehicle Bird’s-Eye View Perception
Watch later
Bin Huang, …
Total of 0 viewers voted for saving the presentation to eternal vault which is 0.0%
Adaptive Methods for Nonconvex Continual Learning
Watch later
Seungyub Han, …
Total of 0 viewers voted for saving the presentation to eternal vault which is 0.0%