[Paper Review] SceneX: Procedural Controllable Large-scale Scene Generation

Mengqi Zhou, Yuxi Wang+6 more|arXiv (Cornell University)|Mar 23, 2024

Multimodal Machine Learning Applications4 citations

TL;DR

Scene𝒳 introduces a large-scale, procedural, and controllable 3D scene generation framework driven by large language models (LLMs), enabling non-experts to generate high-fidelity, photorealistic scenes—such as 2.5 km × 2.5 km cities—via natural language instructions. The system leverages PCGBench (a curated library of procedural assets and API docs) and PCGPlanner (an LLM agent that generates executable Blender actions), reducing scene creation time from weeks to hours while supporting fine-grained editing and style transfer.

ABSTRACT

Developing comprehensive explicit world models is crucial for understanding and simulating real-world scenarios. Recently, Procedural Controllable Generation (PCG) has gained significant attention in large-scale scene generation by enabling the creation of scalable, high-quality assets. However, PCG faces challenges such as limited modular diversity, high expertise requirements, and challenges in managing the diverse elements and structures in complex scenes. In this paper, we introduce a large-scale scene generation framework, SceneX, which can automatically produce high-quality procedural models according to designers' textual descriptions. Specifically, the proposed method comprises two components, PCGHub and PCGPlanner. The former encompasses an extensive collection of accessible procedural assets and thousands of hand-craft API documents to perform as a standard protocol for PCG controller. The latter aims to generate executable actions for Blender to produce controllable and precise 3D assets guided by the user's instructions. Extensive experiments demonstrated the capability of our method in controllable large-scale scene generation, including nature scenes and unbounded cities, as well as scene editing such as asset placement and season translation.

Motivation & Objective

• To bridge the gap between academic 3D scene generation and industrial deployment by using procedural modeling instead of incompatible 3D primitives like point clouds or radiance fields.
• To reduce the steep learning curve of procedural content generation (PCG) by enabling non-experts to generate complex scenes using natural language instructions.
• To enable high-fidelity, controllable, and editable large-scale scene generation, including asset placement and seasonal style transfer, through an LLM-agent pipeline.
• To develop a scalable, reusable framework that integrates LLMs with procedural generation tools like Blender, enhancing both efficiency and usability.

Proposed method

• The framework consists of two core components: PCGBench, a curated repository of procedural assets (e.g., terrains, cities, vegetation) and thousands of hand-crafted API documentation entries for procedural modeling.
• PCGPlanner is an LLM-based agent that generates executable, step-by-step actions for Blender based on user instructions, using a prompt template with role, task, examples, and documentation components.
• The prompt engineering strategy incorporates structured templates with role definition, task specification, examples, and API documentation to improve plan accuracy and executability.
• The system uses LLMs (e.g., GPT-4, Mistral) to interpret natural language prompts and map them to valid procedural generation commands in Blender, ensuring correct and executable action sequences.
• The framework supports iterative editing and style transfer (e.g., season translation) by re-planning actions based on modified user instructions.
• Evaluation is conducted via success rate (SR@1) and executability rate (ER@1) on 50 scene and 50 asset generation tasks, with quantitative benchmarking across multiple LLMs.

Experimental results

Research questions

• RQ1 Can an LLM-driven agent generate executable, high-fidelity procedural 3D scenes from natural language descriptions without requiring expert-level procedural modeling knowledge?
• RQ2 How does the inclusion of structured API documentation and prompt components (e.g., examples, role) affect the accuracy and executability of generated procedural plans?
• RQ3 To what extent can the framework generate large-scale, realistic scenes (e.g., 2.5 km × 2.5 km cities) with detailed geometry, textures, and lighting using only textual input?
• RQ4 How does the performance of different LLMs (e.g., GPT-4, Mistral, Llama2) vary in generating correct and executable procedural generation plans for complex 3D scenes?
• RQ5 Can the system support fine-grained editing tasks such as asset repositioning and seasonal style transfer while maintaining high plan accuracy and scene quality?

Key findings

• The proposed Scene𝒳 framework reduces the time required to generate a 2.5 km × 2.5 km city from over two weeks (by professional PCG engineers) to just a few hours for a non-expert user.
• GPT-4 achieved the highest success rate (SR@1 = 86.05%) and executability rate (ER@1 = 86.00%) in natural scene generation, outperforming other LLMs including Mistral and Llama2.
• The use of structured prompt components—especially examples and documentation—significantly improved plan success, with ER@1 increasing from 16.00% (no components) to 94.00% (full template).
• Mistral emerged as a strong open-source alternative, achieving 76.00% ER@1 and 68.42% SR@1 in scene generation, with lower hardware requirements than GPT-4.
• The framework demonstrated robust performance in editing tasks, including asset placement and season translation, maintaining high plan accuracy across diverse user instructions.
• The system’s success is highly dependent on the quality of the underlying PCGBench and API documentation, with performance declining when LLMs are tasked with complex, multi-component scenes due to increased reasoning demands.