[Paper Review] LLMs with Industrial Lens: Deciphering the Challenges and Prospects -- A Survey

Ashok Urlana, Charaka Vinayak Kumar+4 more|arXiv (Cornell University)|Feb 22, 2024

ERP Systems Implementation and Impact7 citations

TL;DR

This survey analyzes industrial use of LLMs by interviewing practitioners and reviewing 68 industry papers across eight domains to identify adoption challenges, datasets, evaluation methods, and deployment issues. It highlights compute, privacy, and open-access concerns, and outlines future directions.

ABSTRACT

Large language models (LLMs) have become the secret ingredient driving numerous industrial applications, showcasing their remarkable versatility across a diverse spectrum of tasks. From natural language processing and sentiment analysis to content generation and personalized recommendations, their unparalleled adaptability has facilitated widespread adoption across industries. This transformative shift driven by LLMs underscores the need to explore the underlying associated challenges and avenues for enhancement in their utilization. In this paper, our objective is to unravel and evaluate the obstacles and opportunities inherent in leveraging LLMs within an industrial context. To this end, we conduct a survey involving a group of industry practitioners, develop four research questions derived from the insights gathered, and examine 68 industry papers to address these questions and derive meaningful conclusions. We maintain the Github repository with the most recent papers in the field.

Motivation & Objective

• Understand how LLMs assist industrial applications and the ways they are used.
• Identify the primary industrial applications, datasets, and evaluation metrics in practice.
• Characterize deployment challenges including privacy, compute, and regulatory issues.
• Suggest directions to maximize industrial utility of LLMs in future work.

Proposed method

• Conduct a two-stage study: a practitioner case study via questionnaire and a survey of 68 industry papers.
• Categorize papers into eight application domains to map usage patterns.
• Derive four research questions from practitioner insights and literature review.
• Analyze datasets, models, evaluation metrics, and limitations relevant to industry.
• Synthesize deployment challenges and propose future directions.
• Provide a structured taxonomy of industrial LLM applications and evaluation approaches.

Experimental results

Research questions

• RQ1 RQ1. How do LLMs assist industrial applications, and in what ways?
• RQ2 RQ2. What are the primary applications that industries are focusing on, including the associated datasets and evaluation metrics?
• RQ3 RQ3. What are the deployment challenges, if any?
• RQ4 RQ4. What are the potential directions to maximize the utility of LLMs in industrial applications?

Key findings

• The survey analyzes 68 industry papers and pipelines LLM use across eight broad application domains.
• Industrial LLM adoption faces critical challenges such as compute requirements, privacy concerns, and open access.
• A practitioner case study yielded 26 responses from mid-sized companies to ground the analysis.
• less than 15% of studies conducted human evaluations, indicating a need for more rigorous validation.
• The paper discusses deployment challenges (privacy, infrastructure, regulation) and outlines future directions including multilingual models and privacy/security considerations.
• The study provides a first industrial perspective on LLM utilization, datasets, evaluation metrics, and limitations.