Interpreting Operation Selection in Differentiable Architecture Search: A Perspective from Influence-Directed Explanations
Nov 28, 2022
Speakers
About
The Differentiable ARchiTecture Search (DARTS) has dominated the neural architecture search community due to its search efficiency and simplicity. DARTS leverages continuous relaxation to convert the intractable operation selection problem into a continuous magnitude optimization problem which can be easily handled with gradient-descent, while it poses an additional challenge in measuring the operation importance or selecting an architecture from the optimized magnitudes. The vanilla DARTS assumes the optimized magnitudes reflect the importance of operations, while more recent works find this naive assumption leads to poor generalization and is without any theoretical guarantees. In this work, we leverage influence functions, the functional derivatives of the loss function, to theoretically reveal the operation selection part in DARTS and estimate the candidate operation importance by approximating its influence on the supernet with Taylor expansions. We show the operation strength is not only related to the magnitude but also second-order information, leading to a fundamentally new criterion for operation selection in DARTS, named Influential Magnitude. Empirical studies across different tasks on several spaces show that vanilla DARTS and its variants can avoid most failures by leveraging the proposed theory-driven operation selection criterion.The Differentiable ARchiTecture Search (DARTS) has dominated the neural architecture search community due to its search efficiency and simplicity. DARTS leverages continuous relaxation to convert the intractable operation selection problem into a continuous magnitude optimization problem which can be easily handled with gradient-descent, while it poses an additional challenge in measuring the operation importance or selecting an architecture from the optimized magnitudes. The vanilla DARTS assum…
Organizer
NeurIPS 2022
Account · 953 followers
Like the format? Trust SlidesLive to capture your next event!
Professional recording and live streaming, delivered globally.
Sharing
Recommended Videos
Presentations on similar topic, category or speaker
Benchopt: Reproducible, efficient and collaborative optimization benchmarks
Watch later
Total of 0 viewers voted for saving the presentation to eternal vault which is 0.0%
Single Loop Gaussian Homotopy for Non-convex Functions
Watch later
Total of 0 viewers voted for saving the presentation to eternal vault which is 0.0%
Integral Probability Metrics PAC-Bayes Bounds
Watch later
Ron Amit, …
Total of 0 viewers voted for saving the presentation to eternal vault which is 0.0%
tntorch: Tensor Network Learning with PyTorch
Watch later
Total of 0 viewers voted for saving the presentation to eternal vault which is 0.0%
Cross-Domain MetaDL: Any-Way Any-Shot Learning Competition with Novel Datasets from Practical Domains
Watch later
Total of 0 viewers voted for saving the presentation to eternal vault which is 0.0%
Functional Ensemble Distillation
Watch later
Coby Penso, …
Total of 0 viewers voted for saving the presentation to eternal vault which is 0.0%