[Paper Review] Object-oriented Targets for Visual Navigation using Rich Semantic Representations

Jean-Benoit Delbrouck, Stéphane Dupont|arXiv (Cornell University)|Nov 22, 2018

Multimodal Machine Learning Applications9 references4 citations

TL;DR

This paper proposes an object-oriented visual navigation framework using rich semantic representations to improve generalization in unseen environments. By training agents on object-centric targets with natural language and spatial grounding, the method achieves faster convergence and superior zero-shot generalization across new targets and scenes compared to baseline models, especially when fine-tuned on high-confidence semantic frames.

ABSTRACT

When searching for an object humans navigate through a scene using semantic information and spatial relationships. We look for an object using our knowledge of its attributes and relationships with other objects to infer the probable location. In this paper, we propose to tackle the visual navigation problem using rich semantic representations of the observed scene and object-oriented targets to train an agent. We show that both allows the agent to generalize to new targets and unseen scene in a short amount of training time.

Motivation & Objective

• To improve visual navigation generalization to unseen targets and environments using semantic knowledge.
• To address the limitation of prior models that rely on scene-specific fine-tuning by introducing scene-type-specific policy heads.
• To leverage natural language descriptions and object localization for richer environmental understanding.
• To evaluate whether semantic priors improve zero-shot transfer to new targets and new scene types.

Proposed method

• The model uses a Siamese network (SN) with shared visual encoders (ResNet-50) and scene-type-specific policy heads for kitchen, bedroom, bathroom, and living room.
• Object-oriented targets are used during training, where the agent learns to associate target objects with their semantic attributes and spatial relationships.
• A semantic component is introduced, combining visual features with dense captioning outputs (DenseCap) to produce a 345-dimensional semantic vector per frame.
• The semantic input is projected via a learned matrix $\bm{W}_{1^\prime}$ into a 512-dimensional embedding space, fused with visual features for joint representation.
• The model is trained end-to-end using reinforcement learning with sparse dense rewards, where success is defined by reaching the target within 1000 actions.
• A variant (SSN_S) is trained on the five highest-confidence DenseCap frames per scene, improving semantic grounding quality.

Experimental results

Research questions

• RQ1 Can object-oriented targets with rich semantic representations improve zero-shot generalization to unseen targets in known scenes?
• RQ2 Does incorporating semantic grounding (via DenseCap) enhance generalization to unseen scenes compared to standard visual-only baselines?
• RQ3 How does training on high-confidence semantic frames (SSN_S) compare to training on standard object-oriented targets (SSN) in terms of generalization performance?
• RQ4 To what extent does scene-type-specific policy sharing improve transferability across different instances of the same scene type?
• RQ5 Can the model achieve faster convergence and higher success rates with semantic priors compared to random or object-agnostic targets?

Key findings

• The semantic model (SSN) achieved a 94.1% success rate in the bathroom for task T2 (generalization to unseen scenes), outperforming the baseline SN (17.4%) and prior work [12] (71.4%).
• For task T1 (generalization to unseen targets in seen scenes), SSN achieved 61.9% success in bedrooms, 92.4% in bathrooms, 31.5% in kitchens, and 18.1% in living rooms, significantly outperforming the SN baseline.
• Training on top-semantic targets (SSN_S) improved success rates across all scene types, with 66.2% in bedrooms and 94.1% in bathrooms, indicating that high-confidence semantic frames enhance generalization.
• The model trained on object-oriented targets (SSN) generalized better than random targets, with faster convergence and higher success on unseen targets, even when the target was object-less (e.g., a wall or corridor).
• The SSN_S variant, trained on the five most confident DenseCap frames per scene, achieved the highest success rate in bathrooms (94.1%) and showed consistent improvement over standard SSN across all scene types.
• The model demonstrated robustness in zero-shot transfer: even with only 10 million training frames excluding one instance per scene type, it achieved 90.1% success in living rooms (T2), showing strong generalization potential.