Asymptotics of Ridge(less) Regression under General Source Condition
Apr 14, 2021
Speakers
Dominic Richards
Speaker · 0 followers
Jaouad Mourtada
Speaker · 0 followers
Lorenzo Rosasco
Speaker · 0 followers
About
We analyze the prediction error of ridge regression in an asymptotic regime where the sample size and dimension go to infinity at a proportional rate. In particular, we consider the role played by the structure of the true regression parameter. We observe that the case of a general deterministic parameter can be reduced to the case of a random parameter from a structured prior. The latter assumption is a natural adaptation of classic smoothness assumptions in nonparametric regression, which are known as source conditions in the the context of regularization theory for inverse problems. Roughly speaking, we assume the large coefficients of the parameter are in correspondence to the principal components. In this setting a precise characterisation of the test error is obtained, depending on the inputs covariance and regression parameter structure. We illustrate this characterisation in a simplified setting to investigate the influence of the true parameter on optimal regularisation for overparameterized models. We show that interpolation (no regularisation) can be optimal even with bounded signal-to-noise ratio (SNR), provided that the parameter coefficients are larger on high-variance directions of the data, corresponding to a more regular function than posited by the regularization term. This contrasts with previous work considering ridge regression with isotropic prior, in which case interpolation is only optimal in the limit of infinite SNR.We analyze the prediction error of ridge regression in an asymptotic regime where the sample size and dimension go to infinity at a proportional rate. In particular, we consider the role played by the structure of the true regression parameter. We observe that the case of a general deterministic parameter can be reduced to the case of a random parameter from a structured prior. The latter assumption is a natural adaptation of classic smoothness assumptions in nonparametric regression, which are…
Organizer
AISTATS 2021
Account · 63 followers
Categories
AI & Data Science
Category · 10.8k presentations
Mathematics
Category · 2.4k presentations
About AISTATS 2021
The 24th International Conference on Artificial Intelligence and Statistics was held virtually from Tuesday, 13 April 2021 to Thursday, 15 April 2021.
Like the format? Trust SlidesLive to capture your next event!
Professional recording and live streaming, delivered globally.
Sharing
Recommended Videos
Presentations on similar topic, category or speaker
Quantum Tensor Networks, Stochastic Processes, and Weighted Automata
Watch later
Total of 0 viewers voted for saving the presentation to eternal vault which is 0.0%
Prediction with Finitely many Errors Almost Surely
Watch later
Changlong Wu, …
Total of 0 viewers voted for saving the presentation to eternal vault which is 0.0%
Predictive Complexity Priors
Watch later
Total of 0 viewers voted for saving the presentation to eternal vault which is 0.0%
An Efficient Algorithm For Generalized Linear Bandit: Online Stochastic Gradient Descent and Thompson Sampling
Watch later
Qin Ding, …
Total of 0 viewers voted for saving the presentation to eternal vault which is 0.0%
Learning Complexity of Simulated Annealing
Watch later
Avrim Blum, …
Total of 0 viewers voted for saving the presentation to eternal vault which is 0.0%
Oral: Federated Multi-armed Bandits with Personalization
Watch later
Total of 0 viewers voted for saving the presentation to eternal vault which is 0.0%