[Paper Review] Language-driven Scene Synthesis using Multi-conditional Diffusion Model

An Vuong, Minh Nhat Vu+5 more|arXiv (Cornell University)|Oct 24, 2023

Human Motion and Animation4 citations

TL;DR

This paper introduces a language-driven scene synthesis task that generates 3D scenes using text prompts, human motion, and existing objects, leveraging a novel multi-conditional diffusion model with explicit guiding points for denoising. The method achieves state-of-the-art performance on both PRO-teXt and HUMANISE datasets, significantly outperforming baselines in FID and 3D IP metrics, and enables advanced editing applications like object replacement and shape alternation via text commands.

ABSTRACT

Scene synthesis is a challenging problem with several industrial applications. Recently, substantial efforts have been directed to synthesize the scene using human motions, room layouts, or spatial graphs as the input. However, few studies have addressed this problem from multiple modalities, especially combining text prompts. In this paper, we propose a language-driven scene synthesis task, which is a new task that integrates text prompts, human motion, and existing objects for scene synthesis. Unlike other single-condition synthesis tasks, our problem involves multiple conditions and requires a strategy for processing and encoding them into a unified space. To address the challenge, we present a multi-conditional diffusion model, which differs from the implicit unification approach of other diffusion literature by explicitly predicting the guiding points for the original data distribution. We demonstrate that our approach is theoretically supportive. The intensive experiment results illustrate that our method outperforms state-of-the-art benchmarks and enables natural scene editing applications. The source code and dataset can be accessed at https://lang-scene-synth.github.io/.

Motivation & Objective

• To address the gap in scene synthesis that integrates multiple modalities—text, human motion, and existing objects—by proposing a new language-driven scene synthesis task.
• To overcome limitations of single-condition synthesis methods that rely on implicit fusion of conditions in latent space.
• To develop a 3D point cloud-based scene representation that captures both spatial position and shape for more accurate and detailed scene generation.
• To enable practical, intuitive scene editing by allowing users to modify scenes through natural language commands.

Proposed method

• Proposes a multi-conditional diffusion model for 3D point cloud generation that explicitly models guiding points to steer the denoising process, differing from implicit unification in prior work.
• Introduces a novel concept of 'guiding points' that are predicted from text, motion, and object inputs and directly influence the denoising step in the diffusion process.
• Employs a unified encoder architecture combining CLIP for text, PoseNet for human motion, and PointNet++/DGCNN for 3D point clouds, with a cross-attention mechanism to fuse conditions.
• Uses a denoising U-Net with conditional injection of guiding points to guide the reverse diffusion process toward semantically and physically plausible scene generation.
• Employs a contrastive learning-based loss to improve alignment between text prompts and generated 3D scenes.
• Validates the model with ablation studies and user studies to confirm the effectiveness of guiding points and component choices.

Experimental results

Research questions

• RQ1 Can a multi-conditional diffusion model effectively synthesize 3D scenes by jointly conditioning on text prompts, human motion, and existing objects?
• RQ2 How does explicit guiding point prediction improve generation quality compared to implicit latent space unification in multi-condition diffusion models?
• RQ3 To what extent can language-driven scene editing—such as object replacement, shape alternation, and displacement—be achieved through text commands?
• RQ4 How do different backbone architectures (e.g., PointNet++, DGCNN) and text encoders (e.g., CLIP, BERT) affect the performance of the model?
• RQ5 Does the proposed method generalize across diverse datasets and user preferences, as validated by user studies?

Key findings

• The proposed LSDM model achieves a 3D IP of 0.0402 and FID of 161.05 on the PRO-teXt dataset, outperforming all baselines including ATISS, SUMMON, and MIME.
• On the HUMANISE dataset, LSDM reduces FID by at least 35% compared to state-of-the-art methods, demonstrating significant improvement in scene realism.
• The combination of CLIP, POSA, and PointNet++ achieves the best EMD and F1 scores, with only a 0.1866 unit degradation in CD compared to the best-performing configuration.
• The ablation study confirms that POSA outperforms P2R by nearly three times in F1 score when used with CLIP and PointNet++.
• User studies show that LSDM-generated scenes are rated significantly higher in naturalness, non-collision, and intention matching compared to baselines.
• The model’s performance is stable across different numbers of human pose frames, with minimal degradation when using only a single frame, indicating robustness and efficiency.