[Paper Review] From Concept to Manufacturing: Evaluating Vision-Language Models for Engineering Design

Cyril Picard, Kristen M. Edwards+5 more|arXiv (Cornell University)|Nov 21, 2023

Machine Learning in Materials Science17 citations

TL;DR

The paper systematically evaluates GPT-4V and LLaVA 1.6 34B on engineering design tasks spanning conceptual to manufacturing stages, and releases benchmark datasets and prompts for future VLM assessments.

ABSTRACT

Engineering design is undergoing a transformative shift with the advent of AI, marking a new era in how we approach product, system, and service planning. Large language models have demonstrated impressive capabilities in enabling this shift. Yet, with text as their only input modality, they cannot leverage the large body of visual artifacts that engineers have used for centuries and are accustomed to. This gap is addressed with the release of multimodal vision-language models (VLMs), such as GPT-4V, enabling AI to impact many more types of tasks. Our work presents a comprehensive evaluation of VLMs across a spectrum of engineering design tasks, categorized into four main areas: Conceptual Design, System-Level and Detailed Design, Manufacturing and Inspection, and Engineering Education Tasks. Specifically in this paper, we assess the capabilities of two VLMs, GPT-4V and LLaVA 1.6 34B, in design tasks such as sketch similarity analysis, CAD generation, topology optimization, manufacturability assessment, and engineering textbook problems. Through this structured evaluation, we not only explore VLMs' proficiency in handling complex design challenges but also identify their limitations in complex engineering design applications. Our research establishes a foundation for future assessments of vision language models. It also contributes a set of benchmark testing datasets, with more than 1000 queries, for ongoing advancements and applications in this field.

Motivation & Objective

• Assess how vision-language models handle multimodal engineering design tasks that combine sketches, drawings, and text.
• Create standardized benchmarks and datasets to evaluate VLMs in engineering design.
• Provide qualitative and quantitative analyses to identify capabilities and limitations of VLMs in design contexts.
• Offer a baseline evaluation to guide future VLM developments in engineering design.

Proposed method

• Developed prompts and experiments around images as primary input with short text prompts.
• Performed over 1000 queries to evaluate GPT-4V across tasks including design similarity, early-stage sketch descriptions, CAD generation, topology optimization understanding, manufacturability assessment, machining feature identification, defect identification, textbook problems, and spatial reasoning.
• Compared GPT-4V with an open-source VLM, LLaVA 1.6 34B, using the same tasks and datasets.
• Provided exact prompts and model responses to enable reproducibility of benchmark tasks.

Experimental results

Research questions

• RQ1 Can vision-language models effectively perform engineering design tasks that use both visual and textual inputs?
• RQ2 How do VLMs perform on conceptual design, detailed design, manufacturing/inspection, and education-related tasks compared to human baselines?
• RQ3 What are the limitations and failure modes of VLMs in engineering design contexts, and how can benchmarks drive future improvements?
• RQ4 Can standardized datasets and prompts enable fair comparison across VLMs for engineering design tasks?

Key findings

• GPT-4V achieves high self-consistency in design similarity tasks (94.0%) and minimizes transitive violations (5) across 360 triplets, comparable to or better than human raters.
• GPT-4V produces idea maps that cluster logically by design features (e.g., milk-frother cups vs. bicycles) similarly to human-generated maps.
• Description-matching tasks show perfect accuracy (10/10) when handwritten text is present in sketches; without handwritten text, performance declines but remains above chance, and improves when the option “None of the above” is removed.
• For description generation from sketches, GPT-4V can generate descriptive text that aligns with design content, with results influenced by sketch quality; qualitative prompts can yield informative descriptions.
• The study provides datasets with over 1000 queries and releases inputs/prompts/answers publicly to enable future benchmarking of VLMs in engineering design.