CODA: Contrast-Enhanced and Diversity-promoting Data Augmentation for Natural Language Understanding
May 3, 2021
Speakers
Dinghan Shen
Speaker · 0 followers
Yelong Shen
Speaker · 0 followers
Sandra Sajeev
Speaker · 0 followers
Jiawei Han
Speaker · 1 follower
Weizhu Chen
Speaker · 0 followers
About
Data augmentation has been demonstrated as an effective strategy for improving model generalization and data efficiency. However, due to the discrete nature of natural language, designing label-preserving transformations for text data tends to be more challenging. In this paper, we propose a novel data augmentation frame-work dubbed CoDA, which synthesizes diverse and informative augmented examples by integrating multiple transformations organically. Moreover, a contrastive regularization is introduced to capture the global relationship among all the data samples. A momentum encoder along with a memory bank is further leveraged to better estimate the contrastive loss. To verify the effectiveness of the proposed framework, we apply CoDA to Transformer-based models on a wide range of natural language understanding tasks. On the GLUE benchmark, CoDA gives rise to an average improvement of 2.2%while applied to the Roberta-large model. More importantly, it consistently exhibits stronger results relative to several competitive data augmentation and adversarial training baselines (including the low-resource settings). Extensive experiments show that the proposed contrastive objective can be flexibly combined with various data augmentation approaches to further boost their performance, highlighting the wide applicability of the CoDA framework.Data augmentation has been demonstrated as an effective strategy for improving model generalization and data efficiency. However, due to the discrete nature of natural language, designing label-preserving transformations for text data tends to be more challenging. In this paper, we propose a novel data augmentation frame-work dubbed CoDA, which synthesizes diverse and informative augmented examples by integrating multiple transformations organically. Moreover, a contrastive regularization is int…
About ICLR 2021
The International Conference on Learning Representations (ICLR) is the premier gathering of professionals dedicated to the advancement of the branch of artificial intelligence called representation learning, but generally referred to as deep learning. ICLR is globally renowned for presenting and publishing cutting-edge research on all aspects of deep learning used in the fields of artificial intelligence, statistics and data science, as well as important application areas such as machine vision, computational biology, speech recognition, text understanding, gaming, and robotics.
Like the format? Trust SlidesLive to capture your next event!
Professional recording and live streaming, delivered globally.
Sharing
Recommended Videos
Presentations on similar topic, category or speaker
Undistillable: Making A Nasty Teacher That CANNOT Teach Students
Watch later
Haoyu Ma, …
Categorical Normalizing Flows via Continuous Transformations
Watch later
Measuring Adversarial Robustness using a Voronoi-Epsilon Adversary
Watch later
Hyeongji Kim, …
Plan-Based Relaxed Reward Shaping for Goal-Directed Tasks
Watch later
ARMOURED: Adversarially Robust MOdels using Unlabeled data by REgularizing Diversity
Watch later
Kangkang Lu, …
