Grounded Video Situation Recognition
Dec 6, 2022
Speakers
Zeeshan Khan
Speaker · 0 followers
C.V. Jawahar
Speaker · 0 followers
Makarand Tapaswi
Speaker · 0 followers
About
Dense video understanding requires answering several questions such as who is doing what to whom, with what, how, why, and where. Recently, Video Situation recognition (VidSitu) is framed as a task for structured prediction of multiple events, their relationships, their actions and various verb-role pairs attached to descriptive entities. This task poses several challenges in identifying, disambiguating, and co-referencing entities across multiple verb-role pairs, but also faces some challenges of evaluation. In this work, we propose to add spatio-temporal grounding as an essential component of the structured prediction task and present a novel three stage Transformer model, VideoWhisperer, that is empowered to make joint predictions. In stage one, we learn contextualised embeddings for video features in parallel with key objects that appear in the video clips to enable fine-grained spatio-temporal reasoning. The second stage sees verb-role queries attend and pool information from object embeddings, localising answers to questions posed about the action. The final stage generates these answers as captions to describe each verb-role pair present in the video. Our model operates on a group of events (clips) simultaneously and predicts verbs, verb-role pairs, their nouns, and their grounding on-the-fly. When evaluated on a grounding-augmented version of the VidSitu dataset, we observe a large improvement in entity captioning accuracy, as well as the ability to localize verb-roles without grounding annotations at training time.Dense video understanding requires answering several questions such as who is doing what to whom, with what, how, why, and where. Recently, Video Situation recognition (VidSitu) is framed as a task for structured prediction of multiple events, their relationships, their actions and various verb-role pairs attached to descriptive entities. This task poses several challenges in identifying, disambiguating, and co-referencing entities across multiple verb-role pairs, but also faces some challenges…
Organizer
NeurIPS 2022
Account · 952 followers
Like the format? Trust SlidesLive to capture your next event!
Professional recording and live streaming, delivered globally.
Sharing
Recommended Videos
Presentations on similar topic, category or speaker
Re-Analyze Gauss: Bounds for Private Matrix Approximation via Dyson Brownian Motion
Watch later
Total of 0 viewers voted for saving the presentation to eternal vault which is 0.0%
Improving Generative Adversarial Networks via Adversarial Learning in Latent Space
Watch later
Yang Li, …
Total of 0 viewers voted for saving the presentation to eternal vault which is 0.0%
SAGDA: Achieving 𝒪(ϵ^-2) Communication Complexity in Federated Min-Max Learning
Watch later
Haibo Yang, …
Total of 0 viewers voted for saving the presentation to eternal vault which is 0.0%
CoPur: Robust Collaborative Inference via Feature Purification
Watch later
Jing Liu, …
Total of 0 viewers voted for saving the presentation to eternal vault which is 0.0%
Quartic Polynomial Sub-problem Solutions in Tensor Methods for Nonconvex Optimization
Watch later
Wenqi Zhu, …
Total of 0 viewers voted for saving the presentation to eternal vault which is 0.0%
A Near-Optimal Best-of-Both-Worlds Algorithm for Online Learning with Feedback Graphs
Watch later
Chloé Rouyer, …
Total of 0 viewers voted for saving the presentation to eternal vault which is 0.0%