[Paper Review] How Can Large Language Models Help Humans in Design and Manufacturing?

Liane Makatura, Michael Foshey+16 more|arXiv (Cornell University)|Jul 25, 2023

BIM and Construction Integration24 citations

TL;DR

The paper analyzes how GPT-4 can augment the entire design and manufacturing (CDaM) workflow, exploring text-to-design, design-space generation, design-for-manufacturing, performance prediction, and inverse design, with a focus on capabilities and limitations.

ABSTRACT

The advancement of Large Language Models (LLMs), including GPT-4, provides exciting new opportunities for generative design. We investigate the application of this tool across the entire design and manufacturing workflow. Specifically, we scrutinize the utility of LLMs in tasks such as: converting a text-based prompt into a design specification, transforming a design into manufacturing instructions, producing a design space and design variations, computing the performance of a design, and searching for designs predicated on performance. Through a series of examples, we highlight both the benefits and the limitations of the current LLMs. By exposing these limitations, we aspire to catalyze the continued improvement and progression of these models.

Motivation & Objective

• Motivate the use of Large Language Models to lower barriers in computational design and manufacturing workflows.
• Systematically examine GPT-4's ability to perform tasks across the five CDaM phases: design generation, design space generation, manufacturing preparation, performance evaluation, and inverse design.
• Identify practical strategies for prompting, human-in-the-loop feedback, and tooling to integrate LLMs with existing solvers and visualizers.
• Document limitations and dualities of GPT-4 in CDaM to guide future model improvements and workflow design.

Proposed method

• Represent CDaM components (design, design space, manufacturing instructions, performance metrics) as compact programs to enable LLM-style translation between input and output DSLs.
• Conduct a broad suite of experiments across design representations (2D vector, 3D parametric, robotics URDF/graphs) and manufacturing processes (CNC, additive/subtractive) using GPT-4.
• Evaluate text-to-design, text/design-to-design-space, bi-directional design-for-manufacturing, design-to-performance, and inverse design tasks.
• Assess prompting strategies, human feedback integration, and external tool usage to leverage GPT-4 effectively.
• Fabricate end-to-end examples (cabinet and quadcopter) to demonstrate practical CDaM workflows with GPT-4 support.

Figure 1 . Opportunities for LLM Integration within the CDaM Workflow. Each technical section of our paper covers opportunities for LLM integration in one of the tasks depicted above: text to design, text/design to design space, bi-directional design for manufacturing, design to performance, and inv

Experimental results

Research questions

• RQ1 Can GPT-4 generate meaningful designs from high-level goals across multiple modeling representations?
• RQ2 To what extent can GPT-4 respect user constraints and modular design intentions in generated designs?
• RQ3 How effectively can GPT-4 generate design spaces and variations suitable for performance-driven exploration?
• RQ4 Can GPT-4 assist in predicting performance and guiding inverse design within a CDaM pipeline?
• RQ5 What are the practical limitations and dualities of GPT-4 when integrated into end-to-end CDaM workflows?

Key findings

• GPT-4 demonstrates capability to generate designs and supporting code for SVG and DXF vector formats with iterative refinement.
• GPT-4 can generate 3D constructs using CSG and sketch-based CAD representations but faces spatial and positional challenges that improve with targeted prompting.
• GPT-4 can produce simple table-like structures in 3D using predefined primitives, though alignment and placement issues persist without explicit corrections.
• GPT-4 supports manufacturing process selection, design modifications for manufacturability, and creation of machine-readable manufacturing instructions across additive and subtractive processes.
• The study reveals notable limitations in reasoning, verification, and multi-request scalability, along with dualities related to context dependence and unprompted inferences.
• End-to-end demonstrations (cabinet and quadcopter) illustrate both the potential and the iterative nature of GPT-4-enabled CDaM workflows.

Figure 2 . The prompts used for generating files in the format of SVG and DXF using GPT-4. The visualization of the graphs are shown on the right side. It is clear that GPT-4 can accomplish this task after several iterations.