[Paper Review] Knowledge Sharing in Manufacturing using Large Language Models: User Evaluation and Model Benchmarking

Samuel Kernan Freire, Chaofan Wang+4 more|arXiv (Cornell University)|Jan 10, 2024

Semantic Web and Ontologies4 citations

TL;DR

This paper presents a Large Language Model (LLM)-based system that retrieves and answers operator queries using factory documentation and issue reports, improving knowledge sharing in manufacturing. GPT-4 outperformed other models in factuality and conciseness, while open-source models like StableBeluga2 offered strong performance with better data privacy and customization, though users still preferred human experts when available.

ABSTRACT

Recent advances in natural language processing enable more intelligent ways to support knowledge sharing in factories. In manufacturing, operating production lines has become increasingly knowledge-intensive, putting strain on a factory's capacity to train and support new operators. This paper introduces a Large Language Model (LLM)-based system designed to retrieve information from the extensive knowledge contained in factory documentation and knowledge shared by expert operators. The system aims to efficiently answer queries from operators and facilitate the sharing of new knowledge. We conducted a user study at a factory to assess its potential impact and adoption, eliciting several perceived benefits, namely, enabling quicker information retrieval and more efficient resolution of issues. However, the study also highlighted a preference for learning from a human expert when such an option is available. Furthermore, we benchmarked several commercial and open-sourced LLMs for this system. The current state-of-the-art model, GPT-4, consistently outperformed its counterparts, with open-source models trailing closely, presenting an attractive option given their data privacy and customization benefits. In summary, this work offers preliminary insights and a system design for factories considering using LLM tools for knowledge management.

Motivation & Objective

• To address the challenge of inefficient knowledge transfer in knowledge-intensive manufacturing environments.
• To develop and evaluate an LLM-powered system that retrieves relevant information from factory documentation and issue reports to answer operator queries.
• To benchmark closed-source (e.g., GPT-4, GPT-3.5) and open-source (e.g., StableBeluga2, Llama2) LLMs for domain-specific knowledge retrieval in manufacturing.
• To assess user perceptions and usability of the system in a real factory setting, focusing on trust, efficiency, and safety.

Proposed method

• The system ingests factory documentation and issue reports, then uses vector embeddings to retrieve relevant context for user queries.
• A retrieval-augmented generation (RAG) approach is employed, where retrieved context is injected into the LLM prompt to generate answers.
• Multiple LLMs—including GPT-4, GPT-3.5, StableBeluga2, and Llama2—were evaluated using a standardized prompt across 20 queries.
• User evaluation was conducted with factory managers who simulated operator queries, assessing response quality, speed, and usability.
• Benchmarking used a single coder to evaluate responses on factuality, completeness, and hallucination, with scores normalized across models.
• The study used a fixed prompt and similar hyperparameters (e.g., temperature) across models to ensure comparability, though local hosting was not feasible for all models.

Experimental results

Research questions

• RQ1 How effective are different LLMs—both closed and open-source—in retrieving and generating accurate, contextually relevant answers from manufacturing documentation?
• RQ2 How do factory personnel perceive the usability, reliability, and safety of an LLM-based knowledge system compared to consulting human experts?
• RQ3 What are the key usability and functional improvements needed to enhance operator adoption and trust in LLM-powered tools in manufacturing settings?
• RQ4 To what extent do open-source LLMs like StableBeluga2 match the performance of proprietary models like GPT-4 in a real-world manufacturing knowledge retrieval task?
• RQ5 How does the system’s performance compare to human expert consultation in terms of response speed, completeness, and factual accuracy?

Key findings

• GPT-4 demonstrated the highest performance across all evaluation criteria, including factuality, completeness, and minimal hallucinations.
• StableBeluga2 emerged as a strong open-source alternative, closely matching GPT-4 in performance while offering advantages in data privacy and local deployment.
• GPT-3.5 produced more verbose responses than GPT-4 but at a significantly lower cost per input token, though it was less concise and slightly less accurate.
• Users appreciated the system’s speed and modernization potential but consistently preferred human experts when available, citing concerns about information accuracy and safety.
• The user study revealed that improvements in content specificity, user interface, and operator training are critical for increasing trust and adoption.
• Limitations include a small sample size of factory managers, a narrow set of 20 benchmark questions, and a single coder for evaluation, suggesting caution in generalizing results.