[Paper Review] Knowledge-driven Scene Priors for Semantic Audio-Visual Embodied Navigation

Gyan Tatiya, Jonathan Francis+5 more|arXiv (Cornell University)|Dec 21, 2022

Multimodal Machine Learning Applications4 citations

TL;DR

This paper introduces knowledge-driven scene priors for semantic audio-visual embodied navigation by integrating a novel knowledge graph encoding object-region and region-region relationships, dual graph encoder networks for spatial reasoning, and pre-trained visual and audio encoders within a reinforcement learning framework. The approach achieves significant improvements in generalization to unseen regions and truly novel sounding objects, outperforming strong baselines on unseen contexts in the Habitat-Matterport3D environment.

ABSTRACT

Generalisation to unseen contexts remains a challenge for embodied navigation agents. In the context of semantic audio-visual navigation (SAVi) tasks, the notion of generalisation should include both generalising to unseen indoor visual scenes as well as generalising to unheard sounding objects. However, previous SAVi task definitions do not include evaluation conditions on truly novel sounding objects, resorting instead to evaluating agents on unheard sound clips of known objects; meanwhile, previous SAVi methods do not include explicit mechanisms for incorporating domain knowledge about object and region semantics. These weaknesses limit the development and assessment of models' abilities to generalise their learned experience. In this work, we introduce the use of knowledge-driven scene priors in the semantic audio-visual embodied navigation task: we combine semantic information from our novel knowledge graph that encodes object-region relations, spatial knowledge from dual Graph Encoder Networks, and background knowledge from a series of pre-training tasks -- all within a reinforcement learning framework for audio-visual navigation. We also define a new audio-visual navigation sub-task, where agents are evaluated on novel sounding objects, as opposed to unheard clips of known objects. We show improvements over strong baselines in generalisation to unseen regions and novel sounding objects, within the Habitat-Matterport3D simulation environment, under the SoundSpaces task.

Motivation & Objective

• To improve generalization of embodied agents in semantic audio-visual navigation beyond seen environments and known sounds.
• To address the lack of evaluation on truly novel sounding objects in prior SAVi benchmarks.
• To incorporate domain knowledge about object-region semantics and spatial relationships into navigation policies.
• To develop a modular, knowledge-enhanced framework that improves sample efficiency and decision interpretability.
• To define a new benchmark task evaluating agents on novel sounding objects, not just unheard clips of known objects.

Proposed method

• Constructs a novel knowledge graph encoding object-object, object-region, and region-region relations in indoor environments.
• Employs dual graph encoder networks to model spatial and semantic relationships between objects and regions.
• Pre-trains visual and audio encoders on a curated multimodal dataset to enhance object-awareness and sound understanding.
• Integrates knowledge-driven scene priors into a reinforcement learning framework using memory-based attention mechanisms.
• Uses a modular training paradigm to improve cross-domain generalization and optimize reasoning over vision and audio modalities.
• Applies a multi-stream attention mechanism (SMT) to fuse visual, audio, and knowledge-based reasoning for navigation decisions.

Experimental results

Research questions

• RQ1 Can knowledge-driven scene priors improve generalization to unseen indoor environments in semantic audio-visual navigation?
• RQ2 Does incorporating explicit object-region and region-region semantic relationships enhance agent performance on novel sounding objects?
• RQ3 How does pre-training on a curated visual dataset affect object-awareness and navigation success?
• RQ4 Can a modular, knowledge-augmented RL framework outperform strong baselines in zero-shot generalization to unseen regions and unheard sounds?
• RQ5 To what extent does knowledge integration improve sample efficiency and decision interpretability in multimodal navigation?

Key findings

• The proposed K-SAVEN model achieves a success rate of 73.2% on unseen house scenes (UH/HS) and 31.9% on unheard sounds in unseen houses (UH/US), outperforming strong baselines.
• In the UH/HS setting, only-GENV and both-GENs models achieve 73.2% and 73.0% success rates, respectively, significantly outperforming only-GENB at 64.4%.
• In the UH/US setting, only-GENB achieves 23.3% success rate, outperforming only-GENV at 21.2%, indicating audio reasoning is more critical for novel sounds.
• The full model with both vision and audio knowledge streams (both-GENs) achieves higher SPL and SNA scores than SAVi, indicating shorter, more efficient paths.
• Qualitative results show K-SAVEN trajectories are more direct and reach goals faster than SAVi, with fewer steps and less wandering.
• The ablation study confirms that combining vision and audio reasoning with knowledge priors leads to better generalization than using either modality alone.