[Paper Review] Observations on LLMs for Telecom Domain: Capabilities and Limitations

Sumit Soman, H. G. Ranjani|arXiv (Cornell University)|May 22, 2023

Topic Modeling16 references4 citations

TL;DR

This paper evaluates the capabilities and limitations of generative LLMs—GPT-3.5, GPT-4, Bard, and LLaMA—for enterprise wireless telecom chatbots using Cradlepoint’s domain-specific data. It finds that while models like GPT-4 and Bard show strong performance in domain understanding and robustness to input perturbations, hallucination, inconsistent context retention, and sensitivity to input errors remain critical issues, necessitating fine-tuning and guardrails for production use.

ABSTRACT

The landscape for building conversational interfaces (chatbots) has witnessed a paradigm shift with recent developments in generative Artificial Intelligence (AI) based Large Language Models (LLMs), such as ChatGPT by OpenAI (GPT3.5 and GPT4), Google's Bard, Large Language Model Meta AI (LLaMA), among others. In this paper, we analyze capabilities and limitations of incorporating such models in conversational interfaces for the telecommunication domain, specifically for enterprise wireless products and services. Using Cradlepoint's publicly available data for our experiments, we present a comparative analysis of the responses from such models for multiple use-cases including domain adaptation for terminology and product taxonomy, context continuity, robustness to input perturbations and errors. We believe this evaluation would provide useful insights to data scientists engaged in building customized conversational interfaces for domain-specific requirements.

Motivation & Objective

• To assess the suitability of generative LLMs for building domain-specific conversational interfaces in the enterprise wireless telecom domain.
• To identify key limitations such as hallucination, context retention failure, and sensitivity to input perturbations in real-world telecom use cases.
• To evaluate the effectiveness of prompt-based guardrails in constraining model behavior for product-specific queries.
• To compare model robustness across variations in input spelling, grammar, and terminology perturbations.
• To provide actionable insights for data scientists on model selection, fine-tuning necessity, and deployment constraints in telecom applications.

Proposed method

• Conducted comparative experiments using publicly available LLM interfaces (GPT-3.5, GPT-4, Bard, LLaMA via HuggingChat) on Cradlepoint’s enterprise wireless product and service data.
• Designed domain-specific question-answering tasks covering 5G technology, product features, and configuration steps to evaluate terminology understanding and accuracy.
• Evaluated context continuity through multi-turn dialogues simulating product information lookup and troubleshooting workflows.
• Introduced controlled input perturbations (spelling, grammatical, and domain-specific typos) to test robustness and error recovery capabilities.
• Applied prompt-based guardrails to constrain responses to known product and terminology sets, assessing their effectiveness under perturbed inputs.
• Used qualitative and comparative analysis of model responses to assess hallucination, co-reference resolution, and reliability of step-by-step instructions.

Experimental results

Research questions

• RQ1 Can conversational interfaces using generative LLMs accurately adapt to domain-specific terminology in the telecom space, such as 5G, enterprise wireless products, and network configurations?
• RQ2 How well do these models maintain context across multi-turn conversations, particularly in co-reference resolution and long-term context retention?
• RQ3 To what extent do LLMs hallucinate in generating configuration steps or product-specific information, and how reliable are their recipe-like responses?
• RQ4 How robust are LLMs to input perturbations such as spelling errors, grammatical mistakes, or domain-specific typos in user queries?

Key findings

• GPT-4 demonstrated the strongest performance in long-term context retention and co-reference resolution, outperforming GPT-3.5, Bard, and LLaMA in multi-turn dialogues.
• Bard showed the highest robustness to input perturbations, correctly interpreting and resolving typos in both general and domain-specific terminology without requiring clarification.
• GPT-3.5 failed to resolve certain spelling errors and required user clarification, while GPT-4 corrected errors but expressed uncertainty, prompting users for confirmation.
• LLaMA was highly sensitive to input perturbations; it misinterpreted queries even when guardrails were applied, and hallucinated product names or configurations in response to typos.
• All models, except GPT-4, exhibited spurious correlations—e.g., incorrectly linking 4G and 5G to dual connectivity—indicating potential distributional biases in training data.
• Prompt-based guardrails were ineffective for LLaMA under perturbed inputs, suggesting that model behavior cannot be reliably constrained without fine-tuning for domain-specific robustness.