Maximum-and-Concatenation Networks
Jul 12, 2020
Speakers
Xingyu Xie
Speaker · 0 followers
Jianlong Wu
Speaker · 0 followers
Wayne Zhang
Speaker · 0 followers
Guangcan Liu
Speaker · 0 followers
Zhouchen Lin
Speaker · 0 followers
About
While successful in many fields, deep neural networks (DNNs) still suffer from some open problems such as bad local minima and unsatisfactory generalization performance. Despite the progresses achieved during the past several years, those difficulties have not been overcome completely and are still preventing DNNs from being more successful. In this work, we propose a novel architecture called Maximum-and-Concatenation Networks (MCN) to try eliminating bad local minima and improving generalization ability as well. MCN is a multi-layer network concatenated by a linear part and the maximum of two piecewise smooth functions, and it can approximate a wide range of functions used in practice. Remarkably, we prove that MCN has a very nice property; that is, every local minimum of an (l+1)-layer MCN can be better than, at least as good as, the global minima of the network consisting of its first l layers. In other words, via increasing the network depth, MCN can autonomously improve the goodness of its local minima. What is more, it is easy to plug MCN into an existing deep model to make it also have this property. Finally, under mild conditions, we show that MCN can approximate certain continuous functions arbitrarily well with high efficiency; that is, the covering number of MCN is much smaller than most existing DNNs such as deep ReLU. Based on this, we further provide a tight generalization bound to guarantee the inference ability of MCN when dealing with testing samples. Experiments on the CIFAR datasets confirm the effectiveness of MCN.While successful in many fields, deep neural networks (DNNs) still suffer from some open problems such as bad local minima and unsatisfactory generalization performance. Despite the progresses achieved during the past several years, those difficulties have not been overcome completely and are still preventing DNNs from being more successful. In this work, we propose a novel architecture called Maximum-and-Concatenation Networks (MCN) to try eliminating bad local minima and improving generalizati…
Categories
AI & Data Science
Category · 10.8k presentations
About ICML 2020
The International Conference on Machine Learning (ICML) is the premier gathering of professionals dedicated to the advancement of the branch of artificial intelligence known as machine learning. ICML is globally renowned for presenting and publishing cutting-edge research on all aspects of machine learning used in closely related areas like artificial intelligence, statistics and data science, as well as important application areas such as machine vision, computational biology, speech recognition, and robotics. ICML is one of the fastest growing artificial intelligence conferences in the world. Participants at ICML span a wide range of backgrounds, from academic and industrial researchers, to entrepreneurs and engineers, to graduate students and postdocs.
Like the format? Trust SlidesLive to capture your next event!
Professional recording and live streaming, delivered globally.
Sharing
Recommended Videos
Presentations on similar topic, category or speaker
Context-Aware Local Differential Privacy
Watch later
When Demands Evolve Larger and Noisier: Learning and Earning in a Growing Environment
Watch later
Feng Zhu, …
On Gradient Descent Ascent for Nonconvex-Concave Minimax Problems
Watch later
Tianyi Lin, …
Optimizing Data Usage via Differentiable Rewards
Watch later
Xinyi Wang, …
Oral presentation: Q&A - 1
Watch later
