[Paper Review] Generative AI Models for Different Steps in Architectural Design: A Literature Review

Chengyuan Li, Tianyu Zhang+3 more|arXiv (Cornell University)|Mar 30, 2024

Architecture and Computational Design9 citations

TL;DR

A literature review surveying how generative AI, including diffusion models, 3D generative models, and foundation models, is applied across six architectural design steps from 2020–2023.

ABSTRACT

Recent advances in generative artificial intelligence (AI) technologies have been significantly driven by models such as generative adversarial networks (GANs), variational autoencoders (VAEs), and denoising diffusion probabilistic models (DDPMs). Although architects recognize the potential of generative AI in design, personal barriers often restrict their access to the latest technological developments, thereby causing the application of generative AI in architectural design to lag behind. Therefore, it is essential to comprehend the principles and advancements of generative AI models and analyze their relevance in architecture applications. This paper first provides an overview of generative AI technologies, with a focus on probabilistic diffusion models (DDPMs), 3D generative models, and foundation models, highlighting their recent developments and main application scenarios. Then, the paper explains how the abovementioned models could be utilized in architecture. We subdivide the architectural design process into six steps and review related research projects in each step from 2020 to the present. Lastly, this paper discusses potential future directions for applying generative AI in the architectural design steps. This research can help architects quickly understand the development and latest progress of generative AI and contribute to the further development of intelligent architecture.

Motivation & Objective

• Explore how generative AI technologies are applied at different architectural design steps (preliminary 3D forms, layout, structural system, 3D form refinement, facade, and imagery) to enhance innovation and efficiency.
• Analyze the progression and trends of AI model usage in architecture from 2020 to 2023.
• Identify barriers, data needs, and interdisciplinary gaps hindering broader adoption of advanced generative models in architecture.
• Summarize foundational AI concepts relevant to architecture (GANs, VAEs, diffusion models, 3D representations, NeRF, and foundation models) and their architectural applications.
• Propose directions for future research and practical integration of generative AI in architectural practice.

Proposed method

• Survey of literature from 2020–2023 using databases such as Cumincad and Web of Science, supplemented by Litmaps.
• Structured analysis around six architectural design steps to map AI applications to design outputs.
• Explanation of core generative AI principles (GANs, VAEs, diffusion models, Latent Diffusion Models, 3D representations like voxels, point clouds, meshes, and implicit functions).
• Discussion of foundation models including Large Language Models and Large Vision Models and their relevance to architecture.
• Presentation of applications in image, video, and 3D model generation, including representative methods (e.g., CGANs, pix2pix, DDPM, LDM, NeRF, DreamFusion, DreamCraft3D, CLIP-NeRF).
• Illustration of an architectural case example (Bo-DAA apartment) to demonstrate practical integration of AI into preliminary 3D forms design.

Figure 1 : Examples of architecture design using generative AI techniques: (a) church design [ 1 ] ; (b) matrix of cuboid shapes [ 2 ] ; (c) Frank Gehry’s Walt Disney concert hall [ 3 ] ; (d) Bangkok urban design [ 4 ] ; (e) foresting architecture [ 4 ] ; (f) Urban interiors [ 4 ] and (g) text-to-ar

Experimental results

Research questions

• RQ1 How do generative AI techniques contribute at each of the six architectural design steps?
• RQ2 What model families (GANs/VAEs/diffusion/3D models/foundation models) are most effective for 2D/3D architectural outputs and why?
• RQ3 What are the main professional barriers and data-related challenges hindering adoption in architectural practice?
• RQ4 What future directions and research trajectories can advance the integration of generative AI in architecture?

Key findings

• Research output on generative AI in architecture grew significantly from 2020–2023, with most work centered on architectural plan/design (layout).
• GANs and VAEs are still widely used, while diffusion models (DDPM, LDM) and 3D generative models are increasingly integrated, especially for higher-quality 2D/3D outputs.
• 3D representations explored include voxels, point clouds, meshes, and implicit functions (SDF/UDF/NeRF), with diffusion-based and neural implicit methods enhancing 3D generation.
• Foundation models (LLMs and large vision models) enable text-to-image, image-to-text, and cross-modal capabilities that support architectural workflows.
• Applications span image generation, video generation, and 3D model generation, including text-to-3D and image-to-3D pipelines, often with multi-stage refinement processes.
• Challenges include professional barriers between architecture and computer science, and data-related issues such as unstructured architectural data and GPU resource demands.

Figure 2 : Overview of generative AI applications in architectural design: statistics on research paper numbers and generative models.