[Paper Review] See, Hear, Explore: Curiosity via Audio-Visual Association

Victoria Dean, Shubham Tulsiani+1 more|arXiv (Cornell University)|Jul 7, 2020

Cognitive Science and Education ResearchNeuroscience42 references18 citations

TL;DR

This paper proposes See, Hear, Explore (SHE), a novel curiosity-driven exploration method in reinforcement learning that rewards agents for discovering novel audio-visual associations rather than predicting future states. By leveraging multimodal sensory input, SHE achieves more efficient and robust exploration in both Atari games and the Habitat navigation simulator, outperforming future-prediction-based baselines, especially under noisy conditions.

ABSTRACT

Exploration is one of the core challenges in reinforcement learning. A common formulation of curiosity-driven exploration uses the difference between the real future and the future predicted by a learned model. However, predicting the future is an inherently difficult task which can be ill-posed in the face of stochasticity. In this paper, we introduce an alternative form of curiosity that rewards novel associations between different senses. Our approach exploits multiple modalities to provide a stronger signal for more efficient exploration. Our method is inspired by the fact that, for humans, both sight and sound play a critical role in exploration. We present results on several Atari environments and Habitat (a photorealistic navigation simulator), showing the benefits of using an audio-visual association model for intrinsically guiding learning agents in the absence of external rewards. For videos and code, see https://vdean.github.io/audio-curiosity.html.

Motivation & Objective

• To address the challenge of efficient exploration in reinforcement learning without external rewards.
• To overcome the limitations of future prediction-based curiosity, which is sensitive to noise and difficult to train in high-dimensional spaces.
• To explore whether multimodal associations—specifically between vision and audio—can provide a stronger, more robust intrinsic reward signal.
• To evaluate the method in both standard Atari environments and a realistic photorealistic navigation simulator (Habitat).
• To demonstrate that audio-visual association-based curiosity leads to more sample-efficient and noise-robust learning than existing baselines.

Proposed method

• The method uses a contrastive learning framework to train a multimodal discriminator that classifies whether audio and visual features are from the same event (positive pair) or not (negative pair).
• The intrinsic reward is defined as the cross-entropy loss of the audio-visual contrastive prediction head, which measures the model's ability to associate correct sensory pairs.
• The agent is rewarded for actions that lead to high uncertainty or low confidence in the audio-visual association classifier, indicating novel associations.
• The audio and visual features are extracted using separate CNNs (e.g., ResNet-18 for vision, a 1D CNN for audio), and then concatenated for the contrastive head.
• The approach avoids learning a full future state predictor, instead focusing on learning shared representations across modalities.
• The method is trained end-to-end with the policy, using the contrastive loss as the intrinsic reward signal to guide exploration.

Experimental results

Research questions

• RQ1 Can audio-visual associations serve as a more robust and effective intrinsic reward signal than future prediction in reinforcement learning?
• RQ2 Does multimodal curiosity lead to more efficient exploration in environments with high stochasticity or sensor noise?
• RQ3 How does audio-visual curiosity perform in complex, realistic environments such as the Habitat simulator compared to standard curiosity baselines?
• RQ4 Can audio-visual association-based curiosity generalize across diverse tasks and environments, including both simple Atari games and photorealistic 3D navigation?
• RQ5 Is the proposed method more sample-efficient and robust to input noise than future prediction-based curiosity?

Key findings

• SHE outperforms standard curiosity baselines on 10 out of 12 Atari games, achieving higher sample efficiency and more consistent exploration.
• In the Habitat simulator, SHE significantly outperforms baselines in area coverage and exploration efficiency, demonstrating strong performance in realistic, complex environments.
• The method is more robust to input noise: performance degrades much less than future-prediction baselines when Gaussian noise is added to visual and audio features.
• On noisy versions of MsPacman, SpaceInvaders, and Asterix, SHE maintains stable performance, while the future prediction baseline shows significant performance drops.
• A joint method combining future prediction and audio-visual contrastive loss outperforms both individual components in 10 of 12 Atari games.
• The ablation study confirms that using both audio and visual features together improves performance over using only visual features, validating the importance of multimodal association.