[Paper Review] Optimizing RAG Techniques for Automotive Industry PDF Chatbots: A Case Study with Locally Deployed Ollama Models

Fei Liu, Zejun Kang+1 more|arXiv (Cornell University)|Aug 12, 2024

AI in Service Interactions4 citations

TL;DR

This paper proposes a multi-dimensional optimization of Retrieval-Augmented Generation (RAG) for local, Ollama-deployed LLMs in automotive PDF chatbots, enhancing PDF parsing, retrieval, and context compression for technical documents. It achieves significant gains in context precision, recall, and answer faithfulness—especially on proprietary automotive datasets—via custom Langchain components and a self-RAG agent using LangGraph best practices.

ABSTRACT

With the growing demand for offline PDF chatbots in automotive industrial production environments, optimizing the deployment of large language models (LLMs) in local, low-performance settings has become increasingly important. This study focuses on enhancing Retrieval-Augmented Generation (RAG) techniques for processing complex automotive industry documents using locally deployed Ollama models. Based on the Langchain framework, we propose a multi-dimensional optimization approach for Ollama's local RAG implementation. Our method addresses key challenges in automotive document processing, including multi-column layouts and technical specifications. We introduce improvements in PDF processing, retrieval mechanisms, and context compression, tailored to the unique characteristics of automotive industry documents. Additionally, we design custom classes supporting embedding pipelines and an agent supporting self-RAG based on LangGraph best practices. To evaluate our approach, we constructed a proprietary dataset comprising typical automotive industry documents, including technical reports and corporate regulations. We compared our optimized RAG model and self-RAG agent against a naive RAG baseline across three datasets: our automotive industry dataset, QReCC, and CoQA. Results demonstrate significant improvements in context precision, context recall, answer relevancy, and faithfulness, with particularly notable performance on the automotive industry dataset. Our optimization scheme provides an effective solution for deploying local RAG systems in the automotive sector, addressing the specific needs of PDF chatbots in industrial production environments. This research has important implications for advancing information processing and intelligent production in the automotive industry.

Motivation & Objective

• Address the challenge of deploying reliable, offline LLM-powered chatbots in automotive production environments with limited computational resources.
• Overcome domain-specific document complexities such as multi-column layouts and technical specifications in automotive PDFs.
• Improve retrieval and generation quality in RAG pipelines tailored for industrial documentation using locally deployed LLMs.
• Design and evaluate a self-RAG agent based on LangGraph best practices to enhance contextual reasoning and response accuracy.
• Establish a benchmark for local RAG performance in the automotive domain using a proprietary dataset of technical reports and regulations.

Proposed method

• Utilized the Langchain framework to implement a modular RAG pipeline with custom components for PDF processing and embedding pipelines.
• Enhanced PDF parsing using layout-aware techniques to preserve multi-column structures and technical content in automotive documents.
• Optimized retrieval with vector similarity search augmented by hybrid retrieval strategies tailored to technical terminology and document structure.
• Applied context compression techniques to reduce token overhead while preserving critical information for LLM generation.
• Designed a self-RAG agent using LangGraph to enable iterative reasoning and dynamic context refinement during query resolution.
• Integrated Ollama-hosted LLMs in a local deployment to ensure data privacy and low-latency inference in industrial settings.

Experimental results

Research questions

• RQ1 How can RAG systems be optimized for complex, multi-column automotive PDFs with technical specifications using locally deployed LLMs?
• RQ2 To what extent do context compression and retrieval enhancements improve answer precision and recall in industrial document chatbots?
• RQ3 How does a self-RAG agent based on LangGraph outperform standard RAG baselines in multi-turn, contextually dense queries?
• RQ4 What performance gains are achievable in faithfulness and relevancy when applying domain-specific optimizations to RAG pipelines?
• RQ5 How do the optimized RAG and self-RAG models compare across diverse benchmarks, including proprietary automotive, QReCC, and CoQA datasets?

Key findings

• The optimized RAG model achieved significantly higher context precision and recall on the proprietary automotive dataset compared to the naive RAG baseline.
• Answer relevancy and faithfulness metrics improved substantially, particularly on complex queries involving technical specifications and multi-part document structures.
• The self-RAG agent demonstrated enhanced reasoning capabilities, with measurable improvements in handling multi-step and contextually dependent queries.
• Performance gains were most pronounced on the automotive-specific dataset, indicating strong domain adaptation through layout-aware parsing and retrieval.
• The system maintained high efficiency and low latency in local deployment, confirming feasibility for real-time use in industrial production environments.
• Quantitative results across QReCC and CoQA confirmed consistent improvements, though the largest gains were observed on the domain-specific dataset.