Risk Monotonicity in Statistical Learning
Dec 6, 2021
Speakers
Zakaria Mhammedi
Speaker · 0 followers
About
Acquisition of data is a difficult task in many applications of machine learning, and it is only natural that one hopes and expects the populating risk to decrease (better performance) monotonically with increasing data points. It turns out, somewhat surprisingly, that this is not the case even for the most standard algorithms such as empirical risk minimization. Non-monotonic behavior of the risk and instability in training have manifested and appeared in the popular deep learning paradigm under the description of double descent. These problems highlight the current lack of understanding of learning algorithms and generalization. It is, therefore, crucial to pursue this concern and provide a characterization of such behavior. In this paper, we derive the first consistent and risk-monotonic algorithms for a general statistical learning setting under weak assumptions, consequently resolving an open problem Viering et al. 2019 on how to avoid non-monotonic behavior of risk curves. We further show that risk monotonicity need not necessarily come at the price of worse excess risk rates. To achieve this, we derive new empirical Bernstein-like concentration inequalities of independent interest that hold for certain non-i.i.d. processes such as Martingale Difference Sequences.Acquisition of data is a difficult task in many applications of machine learning, and it is only natural that one hopes and expects the populating risk to decrease (better performance) monotonically with increasing data points. It turns out, somewhat surprisingly, that this is not the case even for the most standard algorithms such as empirical risk minimization. Non-monotonic behavior of the risk and instability in training have manifested and appeared in the popular deep learning paradigm unde…
Organizer
NeurIPS 2021
Account · 1.9k followers
Categories
AI & Data Science
Category · 10.8k presentations
Mathematics
Category · 2.4k presentations
About NeurIPS 2021
Neural Information Processing Systems (NeurIPS) is a multi-track machine learning and computational neuroscience conference that includes invited talks, demonstrations, symposia and oral and poster presentations of refereed papers. Following the conference, there are workshops which provide a less formal setting.
Like the format? Trust SlidesLive to capture your next event!
Professional recording and live streaming, delivered globally.
Sharing
Recommended Videos
Presentations on similar topic, category or speaker
Solving Radiology
Watch later
Total of 0 viewers voted for saving the presentation to eternal vault which is 0.0%
On Data-centric Myths
Watch later
Total of 0 viewers voted for saving the presentation to eternal vault which is 0.0%
Width-based Lookaheads with Learnt Base Policies and Heuristics Over the Atari-2600 Benchmark
Watch later
Total of 0 viewers voted for saving the presentation to eternal vault which is 0.0%
Supercharging Imbalanced Data Learning With Energy-based Contrastive Representation Transfer
Watch later
Junya Chen, …
Total of 0 viewers voted for saving the presentation to eternal vault which is 0.0%
Solving Graph-based Public Good Games with Tree Search and Imitation Learning
Watch later
Total of 0 viewers voted for saving the presentation to eternal vault which is 0.0%
Proportional Participatory Budgeting with Cardinal Utilities
Watch later
Total of 0 viewers voted for saving the presentation to eternal vault which is 0.0%