Doping: A technique for Extreme Compression of LSTM Models using Sparse Structured Additive Matrices
4. dubna 2021
Řečníci
Urmish Thakker
Sprecher:in · 1 Follower:in
Paul Whatmough
Sprecher:in · 0 Follower:innen
Zhi-gang Liu
Sprecher:in · 0 Follower:innen
Matthew Mattina
Sprecher:in · 0 Follower:innen
O prezentaci
Structured matrices, such as those derived from Kronecker products (KP), are effective at compressing neural networks, but can lead to unacceptable accuracy loss when applied to large models. In this paper, we propose the notion of doping - addition of an extremely sparse matrix to a structured matrix. Doping facilitates additional degrees of freedom for a small number of parameters, allowing them to independently diverge from the fixed structure. To train LSTMs with doped structured matrices, we introduce the additional parameter matrix while slowly annealing its sparsity level. However, we find that performance degrades as we slowly sparsify the doping matrix, due to co-matrix adaptation (CMA) between the structured and the sparse matrices. We address this over dependence on the sparse matrix using a co-matrix dropout regularization (CMR) scheme. We provide empirical evidence to show that doping, CMA and CMR are concepts generally applicable to multiple structured matrices (Kronecker Product, LMF, Hybrid Matrix Decomposition). Additionally, results with doped kronecker product matrices demonstrate state-of-the-art accuracy at large compression factors (10 − 25x) across 4 natural language processing applications with minor loss in accuracy. Doped KP compression technique outperforms previous state-of-the art compression results by achieving 1.3−2.4x higher compression factor at a similar accuracy, while also beating strong alternatives like pruning and low-rank methods by a large margin (8% or more). Additionally, we show that doped KP can be deployed on commodity hardware using the current software stack and achieve 2.5 − 5.5x inference run-time speed-up over baseline.Structured matrices, such as those derived from Kronecker products (KP), are effective at compressing neural networks, but can lead to unacceptable accuracy loss when applied to large models. In this paper, we propose the notion of doping - addition of an extremely sparse matrix to a structured matrix. Doping facilitates additional degrees of freedom for a small number of parameters, allowing them to independently diverge from the fixed structure. To train LSTMs with doped structured matrices, w…
Organizátor
MLSys 2021
Konto · 159 Follower:innen
Kategorie
Informatik und IT
Kategorie · 14,8k Präsentationen
O organizátorovi (MLSys 2021)
The Conference on Machine Learning and Systems targets research at the intersection of machine learning and systems. The conference aims to elicit new connections amongst these fields, including identifying best practices and design principles for learning systems, as well as developing novel learning methods and theory tailored to practical machine learning workflows.
Baví vás formát? Nechte SlidesLive zachytit svou akci!
Profesionální natáčení a streamování po celém světě.
Sdílení
Doporučená videa
Prezentace na podobné téma, kategorii nebo přednášejícího
Elliot: A Comprehensive and Rigorous Framework For Reproducible Recommender Systems Evaluation
Später ansehen
Ewigspeicher-Fortschrittswert: 0 = 0.0%
Opening Remarks
Später ansehen
Ewigspeicher-Fortschrittswert: 0 = 0.0%
QAPPA: Quantization-Aware Power, Performance, and Area Modeling of DNN Accelerators
Später ansehen
Ahmet Inci, …
Ewigspeicher-Fortschrittswert: 0 = 0.0%
Bit Error Robustness for Energy-Efficient DNN Accelerators
Später ansehen
David Stutz, …
Ewigspeicher-Fortschrittswert: 0 = 0.0%
Towards Scalable Distributed Training of Deep Learning on Public Cloud Clusters
Später ansehen
Shaohuai Shi, …
Ewigspeicher-Fortschrittswert: 0 = 0.0%
Science to Fuel Neural Nets and TPU Design
Später ansehen
Ewigspeicher-Fortschrittswert: 0 = 0.0%