spred: Solving L1 Penalty with SGD
Jul 24, 2023
Speakers
Zihao Wang
Speaker · 0 followers
About
We propose to minimize a generic differentiable loss function with L_1 penalty with a redundant reparametrization and straightforward stochastic gradient descent. Our proposal is the direct generalization of a series of previous ideas that the L_1 penalty may be equivalent to a differentiable reparametrization with weight decay. We prove that the proposed method, spred, is an exact solver of L_1 and that the reparametrization trick is completely “benign" for a generic nonconvex function. Practically, we demonstrate the usefulness of the method in (1) training sparse neural networks to perform gene selection tasks, which involves finding relevant features in a very high dimensional space, and (2) neural network compression task, to which previous attempts at applying the L_1-penalty have been unsuccessful. Conceptually, our result bridges the gap between the sparsity in deep learning and conventional statistical learning.We propose to minimize a generic differentiable loss function with L_1 penalty with a redundant reparametrization and straightforward stochastic gradient descent. Our proposal is the direct generalization of a series of previous ideas that the L_1 penalty may be equivalent to a differentiable reparametrization with weight decay. We prove that the proposed method, spred, is an exact solver of L_1 and that the reparametrization trick is completely “benign" for a generic nonconvex function. Practic…
Like the format? Trust SlidesLive to capture your next event!
Professional recording and live streaming, delivered globally.
Sharing
Recommended Videos
Presentations on similar topic, category or speaker
The Value of Out-of-Distribution Data
Watch later
Revisiting Simple Regret: Fast Rates for Returning a Good Arm
Watch later
Yao Zhao, …
Decoding Layer Saliency in Transformers
Watch later
Doubly Optimal No-Regret Learning in Monotone Games
Watch later
Yang Cai, …
PLay: Parametrically Conditioned Layout Generation Using Latent Diffusion
Watch later
