How Does it Sound? Generation of Rhythmic Soundtrack for Human Movements
Dec 6, 2021
Speakers
Xiulong Liu
Speaker · 0 followers
Eli Shlizerman
Speaker · 0 followers
About
One of the primary purposes of video is to capture people and their unique activities. It is often the case that the experience of watching the video can be enhanced by adding a musical soundtrack that is in-sync with the rhythmic features of these activities. How would this soundtrack sound? Such a problem is challenging since little is known about capturing the rhythmic nature of free body movements. In this work, we explore this problem and propose a novel system, called `RhythmicNet', which takes as an input a video which includes human movements and generates a soundtrack for it. RhythmicNet works directly with human movements by extracting skeleton keypoints and implements a sequence of models which translate the keypoints to rhythmic sounds.RhythmicNet follows the natural process of music improvisation which includes the prescription of streams of the beat, the rhythm and the melody. In particular, RhythmicNet first infers the music beat and the style pattern from body keypoints per each frame to produce rhythm. Next, it implements a transformer-based model to generate the hits of drum instruments and implements a U-net based model to generate the velocity and the offsets of the instruments. Additional types of instruments are added to the soundtrack by further conditioning on the generated drum sounds. We evaluate RhythmicNet on large scale datasets of videos that include body movements with inherit sound association, such as dance, as well as 'in the wild' internet videos of various movements and actions. We show that the method can generate plausible music that aligns well with different types of human movements.One of the primary purposes of video is to capture people and their unique activities. It is often the case that the experience of watching the video can be enhanced by adding a musical soundtrack that is in-sync with the rhythmic features of these activities. How would this soundtrack sound? Such a problem is challenging since little is known about capturing the rhythmic nature of free body movements. In this work, we explore this problem and propose a novel system, called `RhythmicNet', which…
Organizer
NeurIPS 2021
Account · 1.9k followers
About NeurIPS 2021
Neural Information Processing Systems (NeurIPS) is a multi-track machine learning and computational neuroscience conference that includes invited talks, demonstrations, symposia and oral and poster presentations of refereed papers. Following the conference, there are workshops which provide a less formal setting.
Like the format? Trust SlidesLive to capture your next event!
Professional recording and live streaming, delivered globally.
Sharing
Recommended Videos
Presentations on similar topic, category or speaker
The Equity Framework
Watch later
Total of 0 viewers voted for saving the presentation to eternal vault which is 0.0%
Learning to Schedule Heuristics in Branch and Bound
Watch later
Total of 0 viewers voted for saving the presentation to eternal vault which is 0.0%
Hit and Lead Discovery with Explorative RL and Fragment-based Molecule Generation
Watch later
Soojung Yang, …
Total of 0 viewers voted for saving the presentation to eternal vault which is 0.0%
Targeted Neural Dynamical Modeling
Watch later
Cole Hurwitz, …
Total of 0 viewers voted for saving the presentation to eternal vault which is 0.0%
AutoBalance: Optimized Loss Functions for Imbalanced Data
Watch later
Mingchen Li, …
Total of 0 viewers voted for saving the presentation to eternal vault which is 0.0%
XAI:: Beware of the Ostrich Policy: End-Users’ Perceptions Towards Data Transparency and Control
Watch later
Total of 0 viewers voted for saving the presentation to eternal vault which is 0.0%