[Paper Review] TSpec-LLM: An Open-source Dataset for LLM Understanding of 3GPP Specifications

Rasoul Nikbakht, Mohamed Benzaghta+1 more|arXiv (Cornell University)|Jun 3, 2024

IPv6, Mobility, Handover, Networks, Security4 citations

TL;DR

This paper introduces TSpec-LLM, an open-source, comprehensive dataset of all 3GPP specifications from Release 8 to Release 19 (1999–2023), preserving original tables, formulas, and document structure. By integrating a retrieval-augmented generation (RAG) framework, the dataset boosts LLM accuracy on telecom-specific questions from 44–51% to 71–75%, significantly enhancing understanding of complex 3GPP standards.

ABSTRACT

Understanding telecom standards involves sorting through numerous technical documents, such as those produced by the 3rd Generation Partnership Project (3GPP), which is time-consuming and labor-intensive. While large language models (LLMs) can assist with the extensive 3GPP knowledge base, an inclusive dataset is crucial for their effective pre-training and fine-tuning. In this paper, we introduce extit{TSpec-LLM}, an open-source comprehensive dataset covering all 3GPP documents from Release 8 to Release 19 (1999--2023). To evaluate its efficacy, we first select a representative sample of 3GPP documents, create corresponding technical questions, and assess the baseline performance of various LLMs. We then incorporate a retrieval-augmented generation (RAG) framework to enhance LLM capabilities by retrieving relevant context from the extit{TSpec-LLM} dataset. Our evaluation shows that using a naive-RAG framework on extit{TSpec-LLM} improves the accuracy of GPT-3.5, Gemini 1.0 Pro, and GPT-4 from 44\%, 46\%, and 51\% to 71\%, 75\%, and 72\%, respectively.

Motivation & Objective

• Address the lack of comprehensive, structured datasets for training LLMs on 3GPP technical specifications.
• Overcome limitations of existing datasets like SPEC5G, which discard tables and formulas and cover only partial releases.
• Enable accurate, automated question-answering on complex 3GPP standards using LLMs through a curated, open dataset.
• Evaluate the effectiveness of retrieval-augmented generation (RAG) in improving LLM performance on telecom-specific queries.
• Provide a foundation for fine-tuning smaller, deployable LLMs for offline, privacy-preserving telecom applications.

Proposed method

• Curated a complete collection of 30,137 3GPP documents from Release 8 to Release 19, preserving original structure, tables, and formulas.
• Converted documents into both .docx and markdown (.md) formats, with LaTeX rendering for equations to ensure LLM processability.
• Generated a technical questionnaire from 3GPP Releases 15–17 (series 36 and 38) using a three-step pipeline: GPT-4 prompt engineering, Mistral 7B validation, and human verification.
• Implemented a naive-RAG framework that retrieves relevant context from TSpec-LLM before feeding it to LLMs for question answering.
• Evaluated state-of-the-art LLMs (GPT-3.5, GPT-4, Gemini 1.0 Pro) on the questionnaire with and without RAG retrieval.
• Used a multi-stage validation process to ensure question difficulty and correctness, including category and difficulty labeling.

Experimental results

Research questions

• RQ1 Can a comprehensive, open-source dataset of 3GPP specifications improve LLM performance on telecom-specific question-answering tasks?
• RQ2 To what extent does retrieval-augmented generation (RAG) enhance LLM accuracy when retrieving context from a domain-specific knowledge base like TSpec-LLM?
• RQ3 How does the performance of LLMs vary across different question difficulty levels (easy, intermediate, hard) when using TSpec-LLM with RAG?
• RQ4 What is the impact of preserving original technical content—especially tables and formulas—on LLM understanding of 3GPP standards?
• RQ5 Can TSpec-LLM serve as a reliable foundation for fine-tuning smaller, open-source LLMs for offline, local deployment in telecom applications?

Key findings

• The TSpec-LLM dataset contains 13.5 GB of text across 30,137 documents from 3GPP Releases 8 to 19, preserving original tables, formulas, and document structure.
• Using a naive-RAG framework, the accuracy of GPT-3.5 on 3GPP questions increased from 44% to 71%, Gemini 1.0 Pro from 46% to 75%, and GPT-4 from 51% to 72%.
• For the most difficult questions requiring deep standard understanding, accuracy improved from 16–36% to 66% with RAG, demonstrating significant gains on complex queries.
• The dataset's retention of technical content like tables and equations is critical, as filtering such content (as in SPEC5G) leads to loss of essential system parameters and configurations.
• The current naive-RAG framework suffers from suboptimal retrieval due to lack of advanced indexing, indicating room for improvement with techniques like sliding windows or metadata augmentation.
• Future work will focus on optimizing RAG with better indexing and developing a task-specific questionnaire integrated with TSpec-LLM for fine-tuning smaller models like Phi3 for offline deployment.