[Paper Review] Exploring the Potential of Large Language Models for Automation in Technical Customer Service

Jochen Wulf, Juerg Meierhofer|arXiv (Cornell University)|May 15, 2024

Technology and Data Analysis4 citations

TL;DR

This study investigates the automation of cognitive tasks in technical customer service using Large Language Models (LLMs), demonstrating that lower-level tasks like translation and summarization can be effectively automated with models such as GPT-4, while higher-level reasoning requires advanced techniques like Retrieval-Augmented Generation (RAG) or fine-tuning. The research highlights the critical role of data ecosystems in enabling complex LLM applications across service organizations.

ABSTRACT

Purpose: The purpose of this study is to investigate the potential of Large Language Models (LLMs) in transforming technical customer service (TCS) through the automation of cognitive tasks. Design/Methodology/Approach: Using a prototyping approach, the research assesses the feasibility of automating cognitive tasks in TCS with LLMs, employing real-world technical incident data from a Swiss telecommunications operator. Findings: Lower-level cognitive tasks such as translation, summarization, and content generation can be effectively automated with LLMs like GPT-4, while higher-level tasks such as reasoning require more advanced technological approaches such as Retrieval-Augmented Generation (RAG) or finetuning ; furthermore, the study underscores the significance of data ecosystems in enabling more complex cognitive tasks by fostering data sharing among various actors involved. Originality/Value: This study contributes to the emerging theory on LLM potential and technical feasibility in service management, providing concrete insights for operators of TCS units and highlighting the need for further research to address limitations and validate the applicability of LLMs across different domains.

Motivation & Objective

• To assess the feasibility of automating cognitive tasks in technical customer service using LLMs.
• To evaluate the performance of LLMs on real-world technical incident data from a Swiss telecom operator.
• To identify which cognitive tasks are automatable with off-the-shelf LLMs versus those requiring advanced methods like RAG or fine-tuning.
• To examine the role of data ecosystems in enabling more complex LLM-driven service automation.
• To contribute practical and theoretical insights for service management and future research on LLM applications in technical support.

Proposed method

• A prototyping approach was used to implement and test LLM-based automation on real technical incident data from a Swiss telecommunications operator.
• Lower-level cognitive tasks such as translation, summarization, and content generation were evaluated using GPT-4 and similar LLMs.
• Higher-level reasoning tasks were tested using Retrieval-Augmented Generation (RAG) and fine-tuning techniques to improve factual accuracy and contextual reasoning.
• The study analyzed the impact of data sharing and data ecosystems on the performance and scalability of LLM-based automation in service contexts.
• The research compared LLM outputs against human-generated benchmarks to assess quality, relevance, and accuracy.

Experimental results

Research questions

• RQ1 To what extent can off-the-shelf LLMs automate lower-level cognitive tasks such as translation, summarization, and content generation in technical customer service?
• RQ2 What are the limitations of standard LLMs in handling higher-level reasoning tasks in technical support scenarios?
• RQ3 How do advanced techniques like Retrieval-Augmented Generation (RAG) and fine-tuning improve the performance of LLMs on complex reasoning tasks?
• RQ4 What role do data ecosystems and data sharing play in enabling effective LLM-based automation in technical customer service?
• RQ5 How do LLM-generated responses compare to human-generated responses in terms of accuracy and relevance for real-world technical incidents?

Key findings

• Lower-level cognitive tasks such as translation, summarization, and content generation can be effectively automated using standard LLMs like GPT-4 with high accuracy and relevance.
• Higher-level reasoning tasks, such as diagnosing complex technical issues, require advanced techniques like Retrieval-Augmented Generation (RAG) or fine-tuning to achieve acceptable performance.
• The integration of external knowledge retrieval through RAG significantly improves factual consistency and contextual accuracy in reasoning tasks.
• Data ecosystems that enable data sharing among service actors are essential for scaling and enhancing the capabilities of LLM-based automation in technical customer service.
• The study confirms that LLMs can significantly reduce manual effort in technical service workflows, especially when combined with structured data access and retrieval mechanisms.