Learning How Actions Sound from Narrated Egocentric Videos

Core Concepts

We propose a novel self-supervised multimodal embedding approach that can discover how a wide range of human actions sound from narrated in-the-wild egocentric videos, without relying on curated datasets or predefined action categories.

Abstract

The paper proposes a novel self-supervised multimodal embedding approach called Multimodal Contrastive-Consensus Coding (MC3) to learn how actions sound from narrated in-the-wild egocentric videos. Key highlights: Existing methods rely on curated datasets with known audio-visual correspondence, limiting the scope of sounding actions that can be learned. In contrast, the proposed approach aims to discover sounding actions from a broader set of everyday human activities captured in egocentric videos. The key idea is to seek video samples where there is semantic agreement between all three modalities - audio, visual, and language - while distancing those that do not. This intersection of the modalities with language assures that correspondences in the audio and visual streams stem from alignment on the sounding action. The model first aligns a preliminary embedding from contrastive losses imposed per instance on each pair of modalities. It then refines those embeddings with a consensus objective that targets a minimum (bottleneck) pairwise similarity, pushing all pairs of inter-modality agreement towards this consensus. Experiments on the Ego4D and EPIC-Sounds datasets show the proposed model can successfully discover sounding actions that agree with ground truth labels, outperforming existing multimodal embedding approaches on sounding action discovery, retrieval, and audio classification tasks.

Stats

"Closing a door, chopping vegetables, typing on a keyboard, talking with a friend—our interactions with the objects and people around us generate audio that reveals our physical behaviors." "In total, among the 33,000 resulting ground truth clips, 17,693 are positive and 15,307 are negative." "We observe that actions involving more significant human motions (wash, close, cut) are more often sounding, whereas more subtle movements (lift, hold) are often not."

Quotes

"Understanding the link between sounds and actions is valuable for a number of applications, such as multimodal activity recognition, cross-modal retrieval, content generation, or forecasting the physical effects of a person's actions." "Importantly, there may be other events in the video, too (e.g., a TV is playing), but these are not narrated. This is significant: the language specifically addresses near-field human interactions with objects, people, and the environment."

Key Insights Distilled From

SoundingActions

by Changan Chen... at arxiv.org 04-09-2024

https://arxiv.org/pdf/2404.05206.pdf

Deeper Inquiries

How can the proposed approach be extended to discover sounding actions in multi-person egocentric videos

To extend the proposed approach to discover sounding actions in multi-person egocentric videos, we can modify the model to handle multiple individuals in the scene. This can be achieved by incorporating additional modalities such as depth information or skeleton data to track the movements and interactions of each person. By including these modalities, the model can learn to associate specific sounds with the actions of individual persons in the video. Additionally, the model can be trained to differentiate between sounds produced by different individuals in the scene, enabling the discovery of sounding actions in a multi-person context.

What other modalities beyond audio, video, and language could be incorporated to further improve the learning of sounding actions

Beyond audio, video, and language, other modalities that could be incorporated to further improve the learning of sounding actions include: Depth Information: Depth data can provide spatial information about the scene, allowing the model to understand the distance between objects and individuals. This can help in associating specific sounds with actions based on their spatial relationships. Inertial Sensors: Incorporating data from inertial sensors worn by individuals can provide information about their movements and gestures. This data can help in capturing subtle actions that may not be clearly visible in the video. Environmental Sensors: Including data from environmental sensors such as temperature or humidity sensors can provide context about the surroundings in which the actions are taking place. This information can help in understanding how the environment influences the sounds produced during actions. By integrating these additional modalities, the model can create a more comprehensive representation of sounding actions, capturing a wider range of contextual information to improve the learning process.

How can the discovered sounding action representations be leveraged to enable applications like audio-visual content generation or robotic manipulation of objects that produce specific sounds

The discovered sounding action representations can be leveraged in various applications such as audio-visual content generation or robotic manipulation of objects that produce specific sounds: Audio-Visual Content Generation: The learned representations can be used to generate synchronized audio-visual content. By associating specific sounds with visual actions, the model can generate realistic audio effects for silent videos or enhance existing videos with appropriate sound effects. Robotic Manipulation: In robotics, the discovered sounding action representations can be utilized to enable robots to interact with objects in a more human-like manner. By recognizing the sounds produced during specific actions, robots can adapt their manipulation strategies based on the auditory feedback, improving their ability to handle objects and perform tasks effectively. Interactive Systems: The representations can also be applied in interactive systems where user actions are accompanied by sound feedback. By understanding how actions sound, these systems can provide audio cues or responses based on user interactions, enhancing the overall user experience and engagement.