Learning Reasoning Paths over Semantic Graphs for Video-grounded Dialogues
May 3, 2021
Speakers
Nancy Chen
Speaker · 0 followers
Steven Hoi
Speaker · 0 followers
About
Compared to traditional visual question answering, video-grounded dialogues require additional reasoning over dialogue context to answer questions in a multi-turn setting. Previous approaches to video-grounded dialogues mostly use dialogue context as a simple text input without modelling the inherent information flows at the turn level. We propose to discover information flows among dialogue turns through a semantic graph constructed based on lexical components in each question and answer. We then introduce a new approach that learns to predict reasoning paths over this semantic graph. Our path prediction model predicts a path from the current turn through past dialogue turns that contain additional visual cues to answer the current question. Our reasoning model sequentially processes both visual and textual information through this reasoning path and the propagated features are used to generate the answer. Our experimental results demonstrate the effectiveness of our method and provide additional insights on how models use semantic dependencies in a dialogue context to retrieve visual cues.Compared to traditional visual question answering, video-grounded dialogues require additional reasoning over dialogue context to answer questions in a multi-turn setting. Previous approaches to video-grounded dialogues mostly use dialogue context as a simple text input without modelling the inherent information flows at the turn level. We propose to discover information flows among dialogue turns through a semantic graph constructed based on lexical components in each question and answer. We th…
About ICLR 2021
The International Conference on Learning Representations (ICLR) is the premier gathering of professionals dedicated to the advancement of the branch of artificial intelligence called representation learning, but generally referred to as deep learning. ICLR is globally renowned for presenting and publishing cutting-edge research on all aspects of deep learning used in the fields of artificial intelligence, statistics and data science, as well as important application areas such as machine vision, computational biology, speech recognition, text understanding, gaming, and robotics.
Like the format? Trust SlidesLive to capture your next event!
Professional recording and live streaming, delivered globally.
Sharing
Recommended Videos
Presentations on similar topic, category or speaker
On the Origin of Implicit Regularization in Stochastic Gradient Descent
Watch later
Samuel Smith, …
Boosting Classification Accuracy of Fertile Sperm Cell Images leveraging cDCGAN
Watch later
Dipam Paul, …
Oral Session 2
Watch later
Multi-Level Local SGD: Distributed SGD for Heterogeneous Hierarchical Networks
Watch later
Continual Meta Policy Search
Watch later
Glen Berseth, …
