Online Training Through Time for Spiking Neural Networks
Nov 28, 2022
Speakers
Mingqing Xiao
Speaker · 0 followers
Qingyan Meng
Speaker · 0 followers
Zongpeng Zhang
Speaker · 0 followers
Zhouchen Lin
Speaker · 0 followers
About
Spiking neural networks (SNNs) are promising brain-inspired energy-efficient models. Recent progress in training methods has enabled successful deep SNNs on large-scale tasks with low latency. Particularly, backpropagation through time (BPTT) with surrogate gradients (SG) is popularly used to enable models to achieve high performance in a very small number of time steps. However, it is at the cost of large memory consumption for training, lack of theoretical clarity for optimization, and inconsistency with the online property of biological learning rules and rules on neuromorphic hardware. Other works connect the spike representations of SNNs with equivalent artificial neural network formulation and train SNNs by gradients from equivalent mappings to ensure descent directions. But they fail to achieve low latency and are also not online. In this work, we propose online training through time (OTTT) for SNNs, which is derived from BPTT to enable forward-in-time learning by tracking presynaptic activities and leveraging instantaneous loss and gradients. Meanwhile, we theoretically analyze and prove that the gradients of OTTT can provide a similar descent direction for optimization as gradients from equivalent mapping between spike representations under both feedforward and recurrent conditions. OTTT only requires constant training memory costs agnostic to time steps, avoiding the significant memory costs of BPTT for GPU training. Furthermore, the update rule of OTTT is in the form of three-factor Hebbian learning, which could pave a path for online on-chip learning. With OTTT, it is the first time that the two mainstream supervised SNN training methods, BPTT with SG and spike representation-based training, are connected, and meanwhile it is in a biologically plausible form. Experiments on CIFAR-10, CIFAR-100, ImageNet, and CIFAR10-DVS demonstrate the superior performance of our method on large-scale static and neuromorphic datasets in a small number of time steps.Spiking neural networks (SNNs) are promising brain-inspired energy-efficient models. Recent progress in training methods has enabled successful deep SNNs on large-scale tasks with low latency. Particularly, backpropagation through time (BPTT) with surrogate gradients (SG) is popularly used to enable models to achieve high performance in a very small number of time steps. However, it is at the cost of large memory consumption for training, lack of theoretical clarity for optimization, and inconsi…
Organizer
NeurIPS 2022
Account · 952 followers
Like the format? Trust SlidesLive to capture your next event!
Professional recording and live streaming, delivered globally.
Sharing
Recommended Videos
Presentations on similar topic, category or speaker
Wasserstein K-means for clustering probability distributions
Watch later
Yubo Zhuang, …
Total of 0 viewers voted for saving the presentation to eternal vault which is 0.0%
Strategies for Applying Low Rank Decomposition to Transformer-Based Models
Watch later
Total of 0 viewers voted for saving the presentation to eternal vault which is 0.0%
Return Augmentation gives Supervised RL Temporal Compositionality
Watch later
Total of 0 viewers voted for saving the presentation to eternal vault which is 0.0%
![[Re] Exacerbating Algorithmic Bias through Fairness Attacks](https://ma.slideslive.com/library/presentations/38992698/thumbnail/re-exacerbating-algorithmic-bias-through-fairness-attacks_FYFNSd_small.jpg)
[Re] Exacerbating Algorithmic Bias through Fairness Attacks
Watch later
Total of 0 viewers voted for saving the presentation to eternal vault which is 0.0%
Homomorphic Matrix Completion
Watch later
Total of 0 viewers voted for saving the presentation to eternal vault which is 0.0%
GNM: A General Navigation Model to Driv Any Robot
Watch later
Dhruv Shah, …
Total of 0 viewers voted for saving the presentation to eternal vault which is 0.0%