Grounded Video Situation Recognition
Dec 6, 2022
Speakers
Zeeshan Khan
Speaker · 0 followers
C.V. Jawahar
Speaker · 0 followers
Makarand Tapaswi
Speaker · 0 followers
About
Dense video understanding requires answering several questions such as who is doing what to whom, with what, how, why, and where. Recently, Video Situation recognition (VidSitu) is framed as a task for structured prediction of multiple events, their relationships, their actions and various verb-role pairs attached to descriptive entities. This task poses several challenges in identifying, disambiguating, and co-referencing entities across multiple verb-role pairs, but also faces some challenges of evaluation. In this work, we propose to add spatio-temporal grounding as an essential component of the structured prediction task and present a novel three stage Transformer model, VideoWhisperer, that is empowered to make joint predictions. In stage one, we learn contextualised embeddings for video features in parallel with key objects that appear in the video clips to enable fine-grained spatio-temporal reasoning. The second stage sees verb-role queries attend and pool information from object embeddings, localising answers to questions posed about the action. The final stage generates these answers as captions to describe each verb-role pair present in the video. Our model operates on a group of events (clips) simultaneously and predicts verbs, verb-role pairs, their nouns, and their grounding on-the-fly. When evaluated on a grounding-augmented version of the VidSitu dataset, we observe a large improvement in entity captioning accuracy, as well as the ability to localize verb-roles without grounding annotations at training time.Dense video understanding requires answering several questions such as who is doing what to whom, with what, how, why, and where. Recently, Video Situation recognition (VidSitu) is framed as a task for structured prediction of multiple events, their relationships, their actions and various verb-role pairs attached to descriptive entities. This task poses several challenges in identifying, disambiguating, and co-referencing entities across multiple verb-role pairs, but also faces some challenges…
Organizer
NeurIPS 2022
Account · 952 followers
Like the format? Trust SlidesLive to capture your next event!
Professional recording and live streaming, delivered globally.
Sharing
Recommended Videos
Presentations on similar topic, category or speaker
PatchComplete: Learning Multi-Resolution Patch Priors for 3D Shape Completion on Unseen Categories
Watch later
Yuchen Rao, …
Total of 0 viewers voted for saving the presentation to eternal vault which is 0.0%
Federated Learning on Patient Data for Privacy-Protecting Polycystic Ovary Syndrome Treatment
Watch later
Lucia Morris, …
Total of 0 viewers voted for saving the presentation to eternal vault which is 0.0%
Compositional generalization through abstract representations in human and artificial neural networks
Watch later
Takuya Ito, …
Total of 0 viewers voted for saving the presentation to eternal vault which is 0.0%
NeurIPS 2022 Workshop: Self-Supervised Learning - Theory and Practice
Watch later
Total of 0 viewers voted for saving the presentation to eternal vault which is 0.0%
Hierarchical Abstraction for Combinatorial Generalization in Object Rearrangement
Watch later
Total of 0 viewers voted for saving the presentation to eternal vault which is 0.0%
Transformer-based normative modelling for anomaly detection of early schizophrenia
Watch later
Total of 0 viewers voted for saving the presentation to eternal vault which is 0.0%