Neural Differential Equations for Learning to Program Neural Nets Through Continuous Learning Rules
Nov 28, 2022
Speakers
Kazuki Irie
Speaker · 0 followers
Francesco Faccio
Speaker · 0 followers
Jürgen Schmidhuber
Speaker · 2 followers
About
Neural ordinary differential equations (ODEs) have attracted much attention as continuous-time counterparts of deep residual neural networks (NNs), and numerous extensions for recurrent NNs have been proposed. Since the 1980s, ODEs have also been used to derive theoretical results for NN learning rules, e.g., the famous connection between Oja's rule and principal component analysis. Such rules are typically expressed as additive iterative update processes which have straightforward ODE counterparts. Here we introduce a novel combination of learning rules and Neural ODEs to build continuous-time sequence processing nets that learn to manipulate short-term memory in rapidly changing synaptic connections of other nets. This yields continuous-time counterparts of Fast Weight Programmers and linear Transformers. Our novel models outperform the best existing Neural Controlled Differential Equation based models on various time series classification tasks, while also addressing their scalability limitations.Neural ordinary differential equations (ODEs) have attracted much attention as continuous-time counterparts of deep residual neural networks (NNs), and numerous extensions for recurrent NNs have been proposed. Since the 1980s, ODEs have also been used to derive theoretical results for NN learning rules, e.g., the famous connection between Oja's rule and principal component analysis. Such rules are typically expressed as additive iterative update processes which have straightforward ODE counterpa…
Organizer
NeurIPS 2022
Account · 962 followers
Like the format? Trust SlidesLive to capture your next event!
Professional recording and live streaming, delivered globally.
Sharing
Recommended Videos
Presentations on similar topic, category or speaker
Target-independent XLA optimization using Reinforcement Learning
Watch later
Milan Ganai, …
Total of 0 viewers voted for saving the presentation to eternal vault which is 0.0%
Feature-Proxy Transformer for Few-Shot Segmentation
Watch later
Total of 0 viewers voted for saving the presentation to eternal vault which is 0.0%
Personalizing Sustainable Agriculture with Causal Machine Learning
Watch later
Total of 0 viewers voted for saving the presentation to eternal vault which is 0.0%
Directed Information for Point Process Systems
Watch later
Total of 0 viewers voted for saving the presentation to eternal vault which is 0.0%
Transformers are Sample-Efficient World Models
Watch later
Total of 0 viewers voted for saving the presentation to eternal vault which is 0.0%
Federated Submodel Optimization for Hot and Cold Data Features
Watch later
Yucheng Ding, …
Total of 0 viewers voted for saving the presentation to eternal vault which is 0.0%