Grounded Video Situation Recognition
Nov 28, 2022
Speakers
Zeeshan Khan
Speaker · 0 followers
C.V. Jawahar
Speaker · 0 followers
Makarand Tapaswi
Speaker · 0 followers
About
Dense video understanding requires answering several questions such as who is doing what to whom, with what, how, why, and where. Recently, Video Situation recognition (VidSitu) is framed as a task for structured prediction of multiple events, their relationships, their actions and various verb-role pairs attached to descriptive entities. This task poses several challenges in identifying, disambiguating, and co-referencing entities across multiple verb-role pairs, but also faces some challenges of evaluation. In this work, we propose to add spatio-temporal grounding as an essential component of the structured prediction task and present a novel three stage Transformer model, VideoWhisperer, that is empowered to make joint predictions. In stage one, we learn contextualised embeddings for video features in parallel with key objects that appear in the video clips to enable fine-grained spatio-temporal reasoning. The second stage sees verb-role queries attend and pool information from object embeddings, localising answers to questions posed about the action. The final stage generates these answers as captions to describe each verb-role pair present in the video. Our model operates on a group of events (clips) simultaneously and predicts verbs, verb-role pairs, their nouns, and their grounding on-the-fly. When evaluated on a grounding-augmented version of the VidSitu dataset, we observe a large improvement in entity captioning accuracy, as well as the ability to localize verb-roles without grounding annotations at training time.Dense video understanding requires answering several questions such as who is doing what to whom, with what, how, why, and where. Recently, Video Situation recognition (VidSitu) is framed as a task for structured prediction of multiple events, their relationships, their actions and various verb-role pairs attached to descriptive entities. This task poses several challenges in identifying, disambiguating, and co-referencing entities across multiple verb-role pairs, but also faces some challenges…
Organizer
NeurIPS 2022
Account · 952 followers
Like the format? Trust SlidesLive to capture your next event!
Professional recording and live streaming, delivered globally.
Sharing
Recommended Videos
Presentations on similar topic, category or speaker
Learning Predictions for Algorithms with Predictions
Watch later
Misha Khodak, …
Total of 0 viewers voted for saving the presentation to eternal vault which is 0.0%
DMAP: a Distributed Morphological Attention Policy for Learning to Locomote with a Changing Body
Watch later
Total of 0 viewers voted for saving the presentation to eternal vault which is 0.0%
Error Analysis of Tensor-Train Cross Approximation
Watch later
Total of 0 viewers voted for saving the presentation to eternal vault which is 0.0%
SeqPATE: Differentially Private Text Generation via Knowledge Distillation
Watch later
Total of 0 viewers voted for saving the presentation to eternal vault which is 0.0%
A study of natural robustness of deep reinforcement learning algorithms towards adversarial perturbations
Watch later
Qisai Liu, …
Total of 0 viewers voted for saving the presentation to eternal vault which is 0.0%
Control Graph as Unified IO for Morphology-Task Generalization
Watch later
Total of 0 viewers voted for saving the presentation to eternal vault which is 0.0%