[Paper Review] LEBP -- Language Expectation & Binding Policy: A Two-Stream Framework for Embodied Vision-and-Language Interaction Task Learning Agents

Haoyu Liu, Yang Liu+2 more|arXiv (Cornell University)|Mar 9, 2022

Multimodal Machine Learning Applications4 citations

TL;DR

LEBP proposes a two-stream framework for embodied vision-and-language tasks that first generates a language-based task expectation (e.g., 'Pick an apple') before binding it to actions, improving interpretability and generalization. It achieves competitive performance on ALFRED, with a 3.99% absolute SR gain on unseen environments and reduced drop from seen to unseen scenarios.

ABSTRACT

People always desire an embodied agent that can perform a task by understanding language instruction. Moreover, they also want to monitor and expect agents to understand commands the way they expected. But, how to build such an embodied agent is still unclear. Recently, people can explore this problem with the Vision-and-Language Interaction benchmark ALFRED, which requires an agent to perform complicated daily household tasks following natural language instructions in unseen scenes. In this paper, we propose LEBP -- Language Expectation and Binding Policy Module to tackle the ALFRED. The LEBP contains a two-stream process: 1) It first conducts a language expectation module to generate an expectation describing how to perform tasks by understanding the language instruction. The expectation consists of a sequence of sub-steps for the task (e.g., Pick an apple). The expectation allows people to access and check the understanding results of instructions before the agent takes actual actions, in case the task might go wrong. 2) Then, it uses the binding policy module to bind sub-steps in expectation to actual actions to specific scenarios. Actual actions include navigation and object manipulation. Experimental results suggest our approach achieves comparable performance to currently published SOTA methods and can avoid large decay from seen scenarios to unseen scenarios.

Motivation & Objective

• To develop an embodied agent that can understand and plan language instructions before executing actions, enabling human monitoring of task intent.
• To reduce performance decay from seen to unseen environments by decoupling language understanding from action execution.
• To improve interpretability in human-robot interaction by exposing the agent’s internal task planning as a sequence of sub-steps.
• To enhance generalization by minimizing overfitting to visual details (e.g., texture, camera angle) in training scenarios.
• To facilitate integration of environment prior knowledge and spatial structure into the planning process.

Proposed method

• The language expectation module decodes a natural language instruction into a sequence of high-level sub-steps (e.g., 'Walk to the table', 'Pick an apple'), representing the agent’s internal plan.
• The binding policy module maps each sub-step in the expectation to concrete navigation and object manipulation actions based on the current visual and scene context.
• The framework uses a two-stream architecture: one stream processes language for expectation generation, and the other maps expectations to actions in real-time scenarios.
• The model is trained end-to-end using cross-attention mechanisms and supervised learning on expert demonstrations from the ALFRED benchmark.
• Ablation studies use ground-truth depth and segmentation maps to isolate the impact of visual perception modules on performance.
• The method leverages HuggingFace Transformers for language encoding and PyTorch for training, with separate training schedules for language, depth, and segmentation heads.

Experimental results

Research questions

• RQ1 Can a two-stream framework that decouples language understanding from action execution improve generalization in embodied vision-and-language tasks?
• RQ2 Does exposing an agent’s internal task plan (via language expectation) enhance interpretability and human monitoring of agent behavior?
• RQ3 To what extent does the LEBP framework reduce performance drop from seen to unseen environments compared to end-to-end models?
• RQ4 How do visual perception modules (depth, segmentation) impact the overall performance of the LEBP framework?
• RQ5 Can the language expectation module improve planning accuracy without requiring step-by-step ground-truth instructions?

Key findings

• LEBP achieves a 3.99% absolute improvement in Success Rate (SR) on the unseen test split compared to HLSM, with a 19.68% relative gain.
• The model shows strong generalization, with performance on the unseen split close to that on the seen split, indicating reduced overfitting to training scenarios.
• Compared to FILM, LEBP achieves a 2.26% absolute SR improvement on the validation unseen split, demonstrating competitive performance.
• Ground-truth segmentation improves SR by 18.0% on seen and 11.8% on unseen splits, indicating that segmentation is a key bottleneck in the pipeline.
• Ground-truth depth provides a smaller but consistent 1.8% improvement on seen and 1.7% on unseen splits, suggesting depth is less critical than segmentation.
• The method performs particularly well on 'Examine' tasks, with high SR and GC, but struggles with 'Clean & Place', 'Cool & Place', and 'Heat & Place' due to subtask failures.