[Paper Review] Multimodal Procedural Planning via Dual Text-Image Prompting

Yujie Lu, Pan Lu+4 more|arXiv (Cornell University)|May 2, 2023

Multimodal Machine Learning Applications4 citations

TL;DR

This paper introduces Multimodal Procedural Planning (MPP), a novel task that generates paired text-image plans for human-guided task completion. It proposes Text-Image Prompting (TIP), a dual-modality framework using LLMs and diffusion models with bidirectional bridges (T2I-B and I2T-B) to improve cross-modal grounding, resulting in more informative, temporally coherent, and accurate plans than unimodal or baseline multimodal methods on the newly collected WikiPlan and RecipePlan datasets.

ABSTRACT

Embodied agents have achieved prominent performance in following human instructions to complete tasks. However, the potential of providing instructions informed by texts and images to assist humans in completing tasks remains underexplored. To uncover this capability, we present the multimodal procedural planning (MPP) task, in which models are given a high-level goal and generate plans of paired text-image steps, providing more complementary and informative guidance than unimodal plans. The key challenges of MPP are to ensure the informativeness, temporal coherence,and accuracy of plans across modalities. To tackle this, we propose Text-Image Prompting (TIP), a dual-modality prompting method that jointly leverages zero-shot reasoning ability in large language models (LLMs) and compelling text-to-image generation ability from diffusion-based models. TIP improves the interaction in the dual modalities using Text-to-Image Bridge and Image-to-Text Bridge, allowing LLMs to guide the textual-grounded image plan generation and leveraging the descriptions of image plans to ground the textual plan reversely. To address the lack of relevant datasets, we collect WIKIPLAN and RECIPEPLAN as a testbed for MPP. Our results show compelling human preferences and automatic scores against unimodal and multimodal baselines on WIKIPLAN and RECIPEPLAN in terms of informativeness, temporal coherence, and plan accuracy. Our code and data: https://github.com/YujieLu10/MPP.

Motivation & Objective

• To address the underexplored potential of multimodal instructions (text and image) in guiding human task completion.
• To identify key challenges in multimodal procedural planning: informativeness, temporal coherence, and cross-modal accuracy.
• To propose a novel dual-modality prompting framework (TIP) that integrates LLMs and text-to-image diffusion models.
• To create and release two new benchmark datasets, WikiPlan and RecipePlan, for evaluating multimodal procedural planning.
• To demonstrate that TIP significantly improves plan quality over unimodal and multimodal baselines in both automatic and human evaluations.

Proposed method

• Propose Text-Image Prompting (TIP), a dual-modality prompting framework that jointly leverages LLMs and text-to-image (T2I) diffusion models for procedural planning.
• Implement a Text-to-Image Bridge (T2I-B) that uses LLMs to generate complex, grounded textual prompts for T2I models to produce informative image plans.
• Introduce an Image-to-Text Bridge (I2T-B) that verbalizes generated image plans and feeds them back into LLMs to refine and ground the textual plans.
• Use zero-shot reasoning in LLMs to generate initial text plans, then iteratively improve them using visual feedback from the I2T-B.
• Ensure temporal coherence by jointly considering context from both text and image modalities during plan generation.
• Collect and release two new datasets, WikiPlan and RecipePlan, to benchmark MPP with human-annotated text-image plan pairs.

Figure 3: T2I-B elicits visual imagination in LLM to generate explicit scene description (imagination prompt) for text-to-image model conditioned on the vanilla plan.

Experimental results

Research questions

• RQ1 Can multimodal procedural plans (paired text and image steps) provide more informative and accurate guidance than unimodal plans for human task completion?
• RQ2 How can LLMs and T2I models be effectively coordinated to generate cross-modally grounded, temporally coherent plans?
• RQ3 What is the impact of bidirectional bridging (T2I-B and I2T-B) on improving the informativeness and alignment of multimodal plans?
• RQ4 How does TIP compare to unimodal and multimodal baselines in terms of plan quality, especially in zero-shot settings?
• RQ5 To what extent do the proposed WikiPlan and RecipePlan datasets support reliable evaluation of multimodal procedural planning?

Key findings

• TIP achieved a 58% win rate over the step-based baseline (S-Base) in head-to-head human evaluations on plan accuracy, demonstrating consistent improvement.
• The Image-to-Text Bridge (I2T-B) reduced text plan quality by 37.5% on WikiPlan and 35.4% on RecipePlan when ablated, proving its critical role in grounding textual plans.
• The Text-to-Image Bridge (T2I-B) improved FID scores by 1.7% and alignment by 2.6% on average, showing its effectiveness in generating more accurate and grounded image plans.
• The procedure-based TIP method achieved a 60% win rate over the step-based TIP, indicating that holistic planning improves coherence and intent understanding.
• Human evaluations showed strong preference for TIP-generated plans over baselines in terms of informativeness, coherence, and accuracy.
• The new WikiPlan and RecipePlan datasets provide a reliable benchmark for evaluating multimodal procedural planning, with high-quality, human-annotated text-image plan pairs.

(a) w. T2I-B