Improving Anytime Prediction with Parallel Cascaded Networks and a Temporal-Difference Loss
Dec 6, 2021
Speakers
Michael L. Luzzolino
Speaker · 0 followers
Michael C. Mozer
Speaker · 0 followers
Samy Bengio
Speaker · 1 follower
About
Although deep feedforward neural networks share some characteristics with the primate visual system, a key distinction is their dynamics. Deep nets typically operate in serial stages wherein each layer completes its computation before processing begins in subsequent layers. In contrast, biological systems have cascaded dynamics: information propagates from neurons at all layers in parallel but transmission occurs gradually over time, leading to speed-accuracy trade offs even in feedforward architectures. We explore the consequences of biologically inspired parallel hardware by constructing cascaded ResNets in which each residual block has propagation delays but all blocks update in parallel in a stateful manner. Because information transmitted through skip connections avoids delays, the functional depth of the architecture increases over time, yielding anytime predictions that improve with internal-processing time. We introduce a temporal-difference training loss that achieves a strictly superior speed-accuracy profile over standard losses and enables the cascaded architecture to outperform state-of-the-art anytime-prediction methods. The cascaded architecture has intriguing properties, including: it classifies typical instances more rapidly than atypical instances; it is more robust to both persistent and transient noise than is a conventional ResNet; and its time-varying output trace provides a signal that can be exploited to improve information processing and inference.Although deep feedforward neural networks share some characteristics with the primate visual system, a key distinction is their dynamics. Deep nets typically operate in serial stages wherein each layer completes its computation before processing begins in subsequent layers. In contrast, biological systems have cascaded dynamics: information propagates from neurons at all layers in parallel but transmission occurs gradually over time, leading to speed-accuracy trade offs even in feedforward archi…
Organizer
NeurIPS 2021
Account · 1.9k followers
About NeurIPS 2021
Neural Information Processing Systems (NeurIPS) is a multi-track machine learning and computational neuroscience conference that includes invited talks, demonstrations, symposia and oral and poster presentations of refereed papers. Following the conference, there are workshops which provide a less formal setting.
Like the format? Trust SlidesLive to capture your next event!
Professional recording and live streaming, delivered globally.
Sharing
Recommended Videos
Presentations on similar topic, category or speaker
Kernel Functional Optimisation
Watch later
Total of 0 viewers voted for saving the presentation to eternal vault which is 0.0%
Open Catalyst Project
Watch later
Total of 0 viewers voted for saving the presentation to eternal vault which is 0.0%
NeRV: Neural Representations for Videos
Watch later
Hao Chen, …
Total of 0 viewers voted for saving the presentation to eternal vault which is 0.0%
Locality defeats the curse of dimensionality in convolutional teacher-student scenarios
Watch later
Total of 0 viewers voted for saving the presentation to eternal vault which is 0.0%
Discussion: Aleksander Mądry, Ernest Mwebaze, Suchi Saria
Watch later
Total of 0 viewers voted for saving the presentation to eternal vault which is 0.0%
Robot Learning Debate
Watch later
Total of 0 viewers voted for saving the presentation to eternal vault which is 0.0%