Charting and Navigating the Space of Solutions for Recurrent Neural Networks
Dez 6, 2021
Sprecher:innen
Elia Turner
Sprecher:in · 0 Follower:innen
Kabir Dabholkar
Sprecher:in · 0 Follower:innen
Omri Barak
Sprecher:in · 0 Follower:innen
Über
In recent years Recurrent Neural Networks (RNNs) were successfully used to model the way neural activity drives task-related behavior in animals, operating under the implicit assumption that the obtained solutions are universal. Observations in both neuroscience and in machine learning challenge this assumption. Animals can approach a given task with a variety of strategies, and training machine learning algorithms introduces the phenomenon of underspecification. These observations imply that every task is associated with a space of solutions. To date, the structure of this space is not understood, limiting the approach of comparing RNNs with neural data.Here, we characterize the space of solutions associated with a given task, and how hyperparameters bias networks within this space. We first study a simple two-neuron network on a task that leads to multiple solutions. We trace the nature of the final solution back to the network’s initial connectivity, and identify discrete dynamical regimes that underly this diversity. We then examine the neuroscience-inspired Ready-Set-Go timing task. We find a rich set of solutions, even under identical hyperparameters. We uncover this variety by testing the trained networks' ability to extrapolate, as a perturbation to a system often reveals hidden structure. Furthermore, we relate extrapolation patterns to specific dynamical objects and to effective algorithms found by the networks. Using a set of features, we define a space of solutions. We find that most networks stay within the vicinity of the solution defined by their initial connectivity. Taken together, our results shed light on the concept of the space of solutions and its uses both in Machine learning and in Neuroscience.In recent years Recurrent Neural Networks (RNNs) were successfully used to model the way neural activity drives task-related behavior in animals, operating under the implicit assumption that the obtained solutions are universal. Observations in both neuroscience and in machine learning challenge this assumption. Animals can approach a given task with a variety of strategies, and training machine learning algorithms introduces the phenomenon of underspecification. These observations imply that ev…
Organisator
NeurIPS 2021
Konto · 1,9k Follower:innen
Über NeurIPS 2021
Neural Information Processing Systems (NeurIPS) is a multi-track machine learning and computational neuroscience conference that includes invited talks, demonstrations, symposia and oral and poster presentations of refereed papers. Following the conference, there are workshops which provide a less formal setting.
Gefällt euch das Format? Vertraut auf SlidesLive, um euer nächstes Event festzuhalten!
Professionelle Aufzeichnung und Livestreaming – weltweit.
Freigeben
Empfohlene Videos
Präsentationen, deren Thema, Kategorie oder Sprecher:in ähnlich sind
Active 3D Shape Reconstruction from Vision and Touch
Später ansehen
Ewigspeicher-Fortschrittswert: 0 = 0.0%
Compressing Pre-trained Language Models using Progressive Low Rank Decomposition
Später ansehen
Ewigspeicher-Fortschrittswert: 0 = 0.0%
Challenges of Adversarial Image Augmentations
Später ansehen
Arno Blaas, …
Ewigspeicher-Fortschrittswert: 0 = 0.0%
USCO-Solver: Solving Undetermined Stochastic Combinatorial Optimization Problems
Später ansehen
Ewigspeicher-Fortschrittswert: 0 = 0.0%
UFC-BERT: Unifying Multi-Modal Controls for Conditional Image Synthesis
Später ansehen
Zhu Zhang, …
Ewigspeicher-Fortschrittswert: 0 = 0.0%
OVD-Explorer: A General Information-theoretic Exploration Approach for Reinforcement Learning
Später ansehen
Jinyi Liu, …
Ewigspeicher-Fortschrittswert: 0 = 0.0%