[Paper Review] Context-Aware Planning and Environment-Aware Memory for Instruction Following Embodied Agents

Byeonghwi Kim, Jinyeon Kim+3 more|arXiv (Cornell University)|Aug 14, 2023

Multimodal Machine Learning Applications4 citations

TL;DR

This paper proposes CAPEAM, a novel framework for embodied agents that enhances instruction-following performance by integrating context-aware planning (CAP) to prioritize task-relevant objects and environment-aware memory (EAM) to track object states and masks across time. The method achieves state-of-the-art results on the ALFRED benchmark, improving success rates by up to 10.70% in unseen environments.

ABSTRACT

Accomplishing household tasks requires to plan step-by-step actions considering the consequences of previous actions. However, the state-of-the-art embodied agents often make mistakes in navigating the environment and interacting with proper objects due to imperfect learning by imitating experts or algorithmic planners without such knowledge. To improve both visual navigation and object interaction, we propose to consider the consequence of taken actions by CAPEAM (Context-Aware Planning and Environment-Aware Memory) that incorporates semantic context (e.g., appropriate objects to interact with) in a sequence of actions, and the changed spatial arrangement and states of interacted objects (e.g., location that the object has been moved to) in inferring the subsequent actions. We empirically show that the agent with the proposed CAPEAM achieves state-of-the-art performance in various metrics using a challenging interactive instruction following benchmark in both seen and unseen environments by large margins (up to +10.70% in unseen env.).

Motivation & Objective

• Address the challenge of poor generalization in embodied agents due to inadequate modeling of action consequences and environmental state changes.
• Improve long-horizon task planning by incorporating semantic context (e.g., task-relevant objects) into the planning process.
• Overcome failures caused by occlusion and dynamic object states by maintaining persistent memory of object locations and visual masks.
• Enable robust object interaction in unseen environments without relying on human-designed plan templates.
• Enhance agent efficiency by reducing unnecessary exploration and incorrect object interactions through contextual and environmental awareness.

Proposed method

• Introduce Context-Aware Planning (CAP), which first predicts task-relevant objects (context) from natural language instructions before generating a sequence of sub-goals.
• Use a sub-goal planner in CAP to generate executable action sequences with placeholders for context objects, ensuring all actions are aligned with the task’s core objects.
• Implement Environment-Aware Memory (EAM) to store object states, including spatial locations and instance masks, even when objects are occluded.
• Enable retrospective object recognition in EAM by preserving object masks from prior observations, allowing interaction even when current perception fails.
• Integrate object relocation tracking in EAM to prevent repeated interactions with already manipulated objects, reducing task failure from redundant actions.
• Combine CAP and EAM into an end-to-end trainable agent architecture that leverages both contextual planning and persistent environmental memory for improved decision-making.

Experimental results

Research questions

• RQ1 Can integrating semantic context (task-relevant objects) into the planning phase significantly improve long-horizon task success in embodied agents?
• RQ2 To what extent does maintaining persistent memory of object states and visual masks reduce failures due to occlusion and dynamic environments?
• RQ3 How does the proposed method generalize to unseen environments compared to existing baselines without human-designed templates?
• RQ4 Can environment-aware memory prevent redundant interactions with already manipulated objects, thereby improving task efficiency?
• RQ5 Does the combination of context-aware planning and environment-aware memory lead to measurable gains in success rate and goal-conditioned success across diverse instruction-following benchmarks?

Key findings

• CAPEAM achieves state-of-the-art performance on the ALFRED benchmark, with a success rate improvement of up to 10.70% in unseen environments compared to prior methods.
• The agent with CAP significantly reduces incorrect object interactions—e.g., avoiding irrelevant objects like a knife in a task requiring a watch and bowl—by focusing on task-relevant context.
• The EAM module enables successful interaction with occluded objects (e.g., a bowl covered by a watch) by preserving and reusing previously stored masks, preventing failure due to visual occlusion.
• Object relocation tracking in EAM prevents repeated interactions with already moved objects (e.g., a tissue box), reducing task-destroying errors common in baseline agents.
• CAPEAM outperforms existing methods in both seen and unseen environments, demonstrating strong zero-shot generalization without relying on human-designed plan templates.
• The method secured first place in the generalist language grounding agents challenge at the Embodied AI Workshop in CVPR 2023, validating its robustness and generalization capability.