Grounded Video Situation Recognition
Nov 28, 2022
Speakers
Zeeshan Khan
Speaker · 0 followers
C.V. Jawahar
Speaker · 0 followers
Makarand Tapaswi
Speaker · 0 followers
About
Dense video understanding requires answering several questions such as who is doing what to whom, with what, how, why, and where. Recently, Video Situation recognition (VidSitu) is framed as a task for structured prediction of multiple events, their relationships, their actions and various verb-role pairs attached to descriptive entities. This task poses several challenges in identifying, disambiguating, and co-referencing entities across multiple verb-role pairs, but also faces some challenges of evaluation. In this work, we propose to add spatio-temporal grounding as an essential component of the structured prediction task and present a novel three stage Transformer model, VideoWhisperer, that is empowered to make joint predictions. In stage one, we learn contextualised embeddings for video features in parallel with key objects that appear in the video clips to enable fine-grained spatio-temporal reasoning. The second stage sees verb-role queries attend and pool information from object embeddings, localising answers to questions posed about the action. The final stage generates these answers as captions to describe each verb-role pair present in the video. Our model operates on a group of events (clips) simultaneously and predicts verbs, verb-role pairs, their nouns, and their grounding on-the-fly. When evaluated on a grounding-augmented version of the VidSitu dataset, we observe a large improvement in entity captioning accuracy, as well as the ability to localize verb-roles without grounding annotations at training time.Dense video understanding requires answering several questions such as who is doing what to whom, with what, how, why, and where. Recently, Video Situation recognition (VidSitu) is framed as a task for structured prediction of multiple events, their relationships, their actions and various verb-role pairs attached to descriptive entities. This task poses several challenges in identifying, disambiguating, and co-referencing entities across multiple verb-role pairs, but also faces some challenges…
Organizer
NeurIPS 2022
Account · 952 followers
Like the format? Trust SlidesLive to capture your next event!
Professional recording and live streaming, delivered globally.
Sharing
Recommended Videos
Presentations on similar topic, category or speaker
On Learning Fairness and Accuracy on Multiple Subgroups
Watch later
Total of 0 viewers voted for saving the presentation to eternal vault which is 0.0%
Identify event causality with knowledge and analogy
Watch later
Total of 0 viewers voted for saving the presentation to eternal vault which is 0.0%
A Stochastic Prox-Linear Method for CVaR Minimization
Watch later
Si Yi Meng, …
Total of 0 viewers voted for saving the presentation to eternal vault which is 0.0%
Towards Optimal Communication Complexity in Distributed Non-Convex Optimization
Watch later
Total of 0 viewers voted for saving the presentation to eternal vault which is 0.0%
HyperTime: Implicit Neural Representations for Time Series
Watch later
Total of 0 viewers voted for saving the presentation to eternal vault which is 0.0%
Generative Visual Prompt: Unified Distributional Control of Pre-Trained Generative Vision Models
Watch later
Total of 0 viewers voted for saving the presentation to eternal vault which is 0.0%