NAS-Bench-360: Benchmarking Neural Architecture Search on Diverse Tasks
Nov 28, 2022
Speakers
Nicholas Roberts
Speaker · 1 follower
Misha Khodak
Speaker · 0 followers
Junhong Shen
Speaker · 0 followers
Frederic Sala
Speaker · 0 followers
Ameet Talwalkar
Speaker · 2 followers
About
Most existing neural architecture search (NAS) benchmarks and algorithms prioritize well-studied tasks, e.g. image classification on CIFAR or ImageNet. This makes the performance of NAS approaches in more diverse areas poorly understood. In this paper, we present NAS-Bench-360, a benchmark suite to evaluate methods on domains beyond those traditionally studied in architecture search, and use it to address the following question: do state-of-the-art NAS methods perform well on diverse tasks? To construct the benchmark, we curate ten tasks spanning a diverse array of application domains, dataset sizes, problem dimensionalities, and learning objectives. Each task is carefully chosen to interoperate with modern CNN-based search methods while possibly being far-afield from its original development domain. To speed up and reduce the cost of NAS research, for two of the tasks we release the precomputed performance of 15,625 architectures comprising a standard CNN search space. Experimentally, we show the need for more robust NAS evaluation of the kind NAS-Bench-360 enables by showing that several modern NAS procedures perform inconsistently across the ten tasks, with many catastrophically poor results. We also demonstrate how NAS-Bench-360 and its associated precomputed results will enable future scientific discoveries by testing whether several recent hypotheses promoted in the NAS literature hold on diverse tasks. NAS-Bench-360 is hosted at https://nb360.ml.cmu.edu.Most existing neural architecture search (NAS) benchmarks and algorithms prioritize well-studied tasks, e.g. image classification on CIFAR or ImageNet. This makes the performance of NAS approaches in more diverse areas poorly understood. In this paper, we present NAS-Bench-360, a benchmark suite to evaluate methods on domains beyond those traditionally studied in architecture search, and use it to address the following question: do state-of-the-art NAS methods perform well on diverse tasks? To …
Organizer
NeurIPS 2022
Account · 962 followers
Like the format? Trust SlidesLive to capture your next event!
Professional recording and live streaming, delivered globally.
Sharing
Recommended Videos
Presentations on similar topic, category or speaker
Wukong: A 100 Million Large-scale Chinese Cross-modal Pre-training Benchmark
Watch later
Jiaxi Gu, …
Total of 0 viewers voted for saving the presentation to eternal vault which is 0.0%
Algorithms with Prediction Portfolios
Watch later
Total of 0 viewers voted for saving the presentation to eternal vault which is 0.0%
Robust On-Policy Sampling for Data-Efficient Policy Evaluation
Watch later
Rujie Zhong, …
Total of 0 viewers voted for saving the presentation to eternal vault which is 0.0%
Adv-Attribute: Inconspicuous and Transferable Adversarial Attack on Face Recognition
Watch later
Shuai Jia, …
Total of 0 viewers voted for saving the presentation to eternal vault which is 0.0%
Efficient learning of nonlinear prediction models using privileged information
Watch later
Bastian Jung, …
Total of 0 viewers voted for saving the presentation to eternal vault which is 0.0%
Learning to Understand Plane Geometry Diagram
Watch later
Total of 0 viewers voted for saving the presentation to eternal vault which is 0.0%