Robust Statistics and Machine Learning
12. června 2019
Řečníci
Ameer Mohammed
Řečník · 0 sledujících
Ben Ben Liao
Řečník · 0 sledujících
Bowen Zhou
Řečník · 0 sledujících
Christoph H. Lampert
Řečník · 0 sledujících
Daniel Průša
Řečník · 0 sledujících
Dimitris Papailiopoulos
Řečník · 0 sledujících
Erik Learned-Miller
Řečník · 0 sledujících
Guangyong Chen
Řečník · 0 sledujících
Hwanjun Song
Řečník · 0 sledujících
Indranil Gupta
Řečník · 0 sledujících
Jae-Gil Lee
Řečník · 0 sledujících
James Bailey
Řečník · 0 sledujících
Jinfeng Yi
Řečník · 0 sledujících
Minseok Kim
Řečník · 0 sledujících
Mohammad Mahmoody
Řečník · 0 sledujících
Nikola Konstantinov
Řečník · 0 sledujících
Pengfei Chen
Řečník · 0 sledujících
Philip S. Thomas
Řečník · 1 sledující
Po-Ling Loh
Řečník · 0 sledujících
Quanquan Gu
Řečník · 5 sledujících
Saeed Mahloujifar
Řečník · 0 sledujících
Sanmi Koyejo
Řečník · 2 sledující
Shashank Rajput
Řečník · 0 sledujících
Shengyu Zhang
Řečník · 0 sledujících
Sujay Sanghavi
Řečník · 0 sledujících
Vojtěch Franc
Řečník · 0 sledujících
Xingjun Ma
Řečník · 0 sledujících
Yanyao Shen
Řečník · 0 sledujících
Yifei Wang
Řečník · 0 sledujících
Zachary Charles
Řečník · 0 sledujících
Zhili Feng
Řečník · 0 sledujících
O prezentaci
On the Convergence and Robustness of Adversarial Training Improving the robustness of deep neural networks (DNNs) to adversarial examples is an important yet challenging problem for secure deep learning. Across existing defense techniques, adversarial training with Projected Gradient Decent (PGD) is amongst the most effective. Adversarial training solves a min-max optimization problem, with the \textit{inner maximization} generating adversarial examples by maximizing the classification loss, and the \textit{outer minimization} finding model parameters by minimizing the loss on adversarial examples generated from the inner maximization. A criterion that measures how well the inner maximization is solved is therefore crucial for adversarial training. In this paper, we propose such a criterion, namely First-Order Stationary Condition for constrained optimization (FOSC), to quantitatively evaluate the convergence quality of adversarial examples found in the inner maximization. With FOSC, we find that to ensure better robustness, it is essential to use adversarial examples with better convergence quality at the \textit{later stages} of training. Yet at the early stages, high convergence quality adversarial examples are not necessary and may even lead to poor robustness. Based on these observations, we propose a \textit{dynamic} training strategy to gradually increase the convergence quality of the generated adversarial examples, which significantly improves the robustness of adversarial training. Our theoretical and empirical results show the effectiveness of the proposed method. Learning with Bad Training Data via Iterative Trimmed Loss Minimization In this paper, we study a simple and generic framework to tackle the problem of learning model parameters when a fraction of the training samples are corrupted. Our approach is motivated by a simple observation: in a variety of such settings, the evolution of training accuracy (as a function of training epochs) is different for clean samples and bad samples. We propose to iteratively minimize the trimmed loss, by alternating between (a) selecting samples with lowest current loss, and (b) retraining a model on only these samples. Analytically, we characterize the statistical performance and convergence rate of the algorithm for simple and natural linear and non-linear models. Experimentally, we demonstrate its effectiveness in three settings: (a) deep image classifiers with errors only in labels, (b) generative adversarial networks with bad training images, and (c) deep image classifiers with adversarial (image, label) pairs (i.e., backdoor attacks). For the well-studied setting of random label noise, our algorithm achieves state-of-the-art performance without having access to any a-priori guaranteed clean samples. On discriminative learning of prediction uncertainty In classification with reject option the classifier is allowed in uncertain cases to abstain from prediction. The classical cost based model of an optimal classifier with reject option requires the cost of rejection to be defined explicitly. An alternative bounded-improvement model, avoiding the notion of the reject cost, seeks for a classifier with a guaranteed selective risk and maximal cover. We prove that both models share the same class of optimal strategies, and we provide an explicit relation between the reject cost and the target risk being the parameters of the two models. An optimal rejection strategy for both models is based on thresholding the conditional risk defined by posterior probabilities which are usually unavailable. We propose a discriminative algorithm learning an uncertainty function which preserves ordering of the input space induced by the conditional risk, and hence can be used to construct optimal rejection strategies. We show experimentally that the proposed algorithm learns better rejection strategies than common plug-in rules and rules based on the distance to the decision boundary. Understanding and Utilizing Deep Neural Networks Trained with Noisy Labels Noisy labels are ubiquitous in real-world datasets, which poses a challenge for robustly training deep neural networks (DNNs) as DNNs usually have the high capacity to memorize the noisy labels. In this paper, we find that the test accuracy can be quantitatively characterized in terms of the noise ratio in datasets. In particular, the test accuracy is a quadratic function of the noise ratio in the case of symmetric noise, which explains the experimental findings previously published. Based on our analysis, we apply cross-validation to randomly split noisy datasets, which identifies most samples that have correct labels. Then we adopt the Co-teaching strategy which takes full advantage of the identified samples to train DNNs robustly against noisy labels. Compared with extensive state-of-the-art methods, our strategy consistently improves the generalization performance of DNNs under both synthetic and real-world training noise. Does Data Augmentation Lead to Positive Margin? Data augmentation (DA) is commonly used during model training, as it significantly improves test error and model robustness. DA artificially expands the training set by applying random noise, rotations, crops, or even adversarial perturbations to the input data. Although DA is widely used, its capacity to provably improve robustness is not fully understood. In this work, we analyze the robustness that DA begets by quantifying the margin that DA enforces on empirical risk minimizers. We first focus on linear separators, and then a class of nonlinear models whose labeling is constant within small convex hulls of data points. We present lower bounds on the number of augmented data points required for non-zero margin, and show that commonly used DA techniques may only introduce significant margin after adding exponentially many points to the data set. Robust Learning from Untrusted Sources Modern machine learning methods often require more data for training than a single expert can provide. Therefore, it has become a standard procedure to collect data from external sources, e.g. via crowdsourcing. Unfortunately, the quality of these sources is not always guaranteed. As additional complications, the data might be stored in a distributed way, or might even have to remain private. In this work, we address the question of how to learn robustly in such scenarios. Studying the problem through the lens of statistical learning theory, we derive a procedure that allows for learning from all available sources, yet automatically suppresses irrelevant or corrupted data. We show by extensive experiments that our method provides significant improvements over alternative approaches from robust statistics and distributed optimization. SELFIE: Refurbishing Unclean Samples for Robust Deep Learning Owing to the extremely high expressive power of deep neural networks, their side effect is to totally memorize training data even when the labels are extremely noisy. To overcome overfitting on the noisy labels, we propose a novel robust training method called SELFIE. Our key idea is to selectively refurbish and exploit unclean samples that can be corrected with high precision, thereby gradually increasing the number of available training samples. Taking advantage of this design, SELFIE effectively prevents the risk of noise accumulation from the false correction and fully exploits the training data. To validate the superiority of SELFIE, we conducted extensive experimentation using three data sets simulated with varying noise rates. The result showed that SELFIE remarkably improved absolute test error by up to 10.5 percentage points compared with two state-of-the-art robust training methods. Zeno: Distributed Stochastic Gradient Descent with Suspicion-based Fault-tolerance We present Zeno, a technique to make distributed machine learning, particularly Stochastic Gradient Descent (SGD), tolerant to an arbitrary number of faulty workers. This generalizes previous results that assumed a majority of non-faulty nodes; we need assume only one non-faulty worker. Our key idea is to suspect workers that are potentially defective. Since this is likely to lead to false positives, we use a ranking-based preference mechanism. We prove the convergence of SGD for non-convex problems under these scenarios. Experimental results show that Zeno outperforms existing approaches. Concentration Inequalities for Conditional Value at Risk In this paper we derive new concentration inequalities for the conditional value at risk (CVaR) of a random variable, and compare them to the previous state of the art (Brown, 2007). We show analytically that our lower bound is strictly tighter than Brown's, and empirically that this difference is significant. While our upper bound may be looser than Brown's in some cases, we show empirically that in most cases our bound is significantly tighter. After discussing when each upper bound is superior, we conclude with empirical results which suggest that both of our bounds will often be significantly tighter than Brown's. Data Poisoning Attacks in Multi-Party Learning In this work, we demonstrate universal multi-party poisoning attacks that adapt and apply to any multi-party learning process with arbitrary interaction pattern between the parties.On the Convergence and Robustness of Adversarial Training Improving the robustness of deep neural networks (DNNs) to adversarial examples is an important yet challenging problem for secure deep learning. Across existing defense techniques, adversarial training with Projected Gradient Decent (PGD) is amongst the most effective. Adversarial training solves a min-max optimization problem, with the \textit{inner maximization} generating adversarial examples by maximizing the classification loss, an…
Kategorie
Umělá inteligence a data science
Kategorie · 10,8k prezentací
O organizátorovi (ICML 2019)
The International Conference on Machine Learning (ICML) is the premier gathering of professionals dedicated to the advancement of the branch of artificial intelligence known as machine learning. ICML is globally renowned for presenting and publishing cutting-edge research on all aspects of machine learning used in closely related areas like artificial intelligence, statistics and data science, as well as important application areas such as machine vision, computational biology, speech recognition, and robotics. ICML is one of the fastest growing artificial intelligence conferences in the world. Participants at ICML span a wide range of backgrounds, from academic and industrial researchers, to entrepreneurs and engineers, to graduate students and postdocs.
Baví vás formát? Nechte SlidesLive zachytit svou akci!
Profesionální natáčení a streamování po celém světě.
Sdílení
Doporučená videa
Prezentace na podobné téma, kategorii nebo přednášejícího
Planetary Scale Monitoring of Urban Growth in High Flood Risk Areas
Zhlédnout později
Human in the Loop Learning Panel Discussion
Zhlédnout později
Beyond demonstrations: Learning behavior from higher-level supervision
Zhlédnout později
Optimization: Convex and Non-convex
Zhlédnout později
Real/World Bayesian Optimization
Zhlédnout později
