TGEA 2.0: A Large-Scale Diagnostically Annotated Dataset with Benchmark Tasks for Text Generation of Pretrained Language Models
Nov 28, 2022
Speakers
Xiaohu Zhao
Speaker · 0 followers
Chuang Liu
Speaker · 0 followers
Yulong Zeng
Speaker · 0 followers
Deyi Xiong
Speaker · 0 followers
About
In order to diagnostically analyze and improve the capability of pretrained language models (PLMs) in text generation, we propose TGEA 2.0, to date the largest dataset built on machine-authored texts by PLMs with fine-grained semantic annotations on a wide variety of pathological generation errors. We collect 170K nominal, phrasal and sentential prompts from 6M natural sentences in 3 domains. These prompts are fed into 4 generative PLMs with their best decoding strategy to generate paragraphs. 195,629 sentences are extracted from these generated paragraphs for manual annotation, where 36K erroneous sentences are detected, 42K erroneous spans are located and categorized into an error type defined in a two-level error taxonomy. We define a Minimal Set of Error-related Words (MiSEW) for each erroneous span, which not only provides error-associated words but also rationalizes the reasoning behind the error. Quality control with a pre-annotation and feedback loop is performed before and during the entire annotation process. With the diagnostically annotated dataset, we propose 5 diagnosis benchmark tasks (i.e., erroneous text detection, MiSEW extraction, erroneous span location and correction together with error type classification) and 2 pathology mitigation benchmark tasks (pairwise comparison and word prediction). Experiment results on these benchmark tasks demonstrate that TGEA 2.0 is a challenging dataset that could facilitate further research on automatic diagnosis and pathology mitigation over machine texts. The dataset will be publicly available at https://github.com/tjunlp-lab/TGEA/.In order to diagnostically analyze and improve the capability of pretrained language models (PLMs) in text generation, we propose TGEA 2.0, to date the largest dataset built on machine-authored texts by PLMs with fine-grained semantic annotations on a wide variety of pathological generation errors. We collect 170K nominal, phrasal and sentential prompts from 6M natural sentences in 3 domains. These prompts are fed into 4 generative PLMs with their best decoding strategy to generate paragraphs. …
Organizer
NeurIPS 2022
Account · 953 followers
Like the format? Trust SlidesLive to capture your next event!
Professional recording and live streaming, delivered globally.
Sharing
Recommended Videos
Presentations on similar topic, category or speaker
Researchers Comparing DNNs to Brains Need to Adopt Standard Methods of Science.
Watch later
Total of 0 viewers voted for saving the presentation to eternal vault which is 0.0%
Engineering Uncertainty Representations to Monitor Distribution Shifts
Watch later
Total of 0 viewers voted for saving the presentation to eternal vault which is 0.0%
Cost-Sensitive Self-Training for Optimizing Non-Decomposable Metrics
Watch later
Total of 0 viewers voted for saving the presentation to eternal vault which is 0.0%
AutoST: Towards the Universal Modeling of Spatio-temporal Sequences
Watch later
Jianxin Li, …
Total of 0 viewers voted for saving the presentation to eternal vault which is 0.0%
Contrastive Graph Structure Learning via Information Bottleneck for Recommendation
Watch later
Chunyu Wei, …
Total of 0 viewers voted for saving the presentation to eternal vault which is 0.0%
Old can be Gold: Better Gradient Flow can make Vanilla-GCNs Great Again
Watch later
Ajay Jaiswal, …
Total of 0 viewers voted for saving the presentation to eternal vault which is 0.0%