[Paper Review] Evaluating the Efficacy of Open-Source LLMs in Enterprise-Specific RAG Systems: A Comparative Study of Performance and Scalability

B Gautam, Anupam Purwar|arXiv (Cornell University)|Jun 17, 2024

Semantic Web and Ontologies4 citations

TL;DR

This study evaluates open-source LLMs (Llama3 and Mistral) in enterprise-specific RAG systems using proprietary data scraped from company websites. It finds that open-source models achieve performance comparable to GPT-4 using cosine similarity with ground truth (CSGA) as a stable metric, and that context length (top-k) has minimal impact on answer quality, suggesting smaller models like Llama3-8B can outperform larger ones like Mistral-8x7B in domain-specific QA tasks.

ABSTRACT

This paper presents an analysis of open-source large language models (LLMs) and their application in Retrieval-Augmented Generation (RAG) tasks, specific for enterprise-specific data sets scraped from their websites. With the increasing reliance on LLMs in natural language processing, it is crucial to evaluate their performance, accessibility, and integration within specific organizational contexts. This study examines various open-source LLMs, explores their integration into RAG frameworks using enterprise-specific data, and assesses the performance of different open-source embeddings in enhancing the retrieval and generation process. Our findings indicate that open-source LLMs, combined with effective embedding techniques, can significantly improve the accuracy and efficiency of RAG systems, offering a viable alternative to proprietary solutions for enterprises.

Motivation & Objective

• To assess the performance and scalability of open-source LLMs in enterprise-specific RAG systems using proprietary data.
• To compare open-source LLMs and embeddings against proprietary models like GPT-4 in terms of accuracy and efficiency.
• To identify the most effective evaluation metrics for RAG answer quality in enterprise contexts.
• To determine optimal hyperparameters (e.g., top-k) for RAG systems using enterprise datasets.
• To demonstrate that open-source LLMs can deliver cost-effective, scalable, and accurate RAG solutions without high-end GPU infrastructure.

Proposed method

• Collected enterprise-specific data from company websites to build a domain-specific vector database using open-source embeddings.
• Implemented a RAG framework integrating retrieval (via dense vector search) with generative LLMs (Llama3-8B and Mistral-8x7B).
• Evaluated answer quality using multiple metrics: Cosine Similarity with Groundtruth Answer (CSGA), unigram precision/recall, and Deepeval’s contextual recall/precision.
• Systematically varied the top-k parameter to assess retrieval context length impact on answer quality.
• Used a four-category question classification (sparse factual, sparse reasoning, dense factual, dense reasoning) to evaluate model behavior across different question types.
• Measured inference latency and computational cost using perplexity API to compare open-source models with GPT-3.5.

Experimental results

Research questions

• RQ1 How do open-source LLMs compare to proprietary models like GPT-4 in accuracy and efficiency for enterprise-specific RAG tasks?
• RQ2 Which evaluation metrics (e.g., CSGA, unigram precision/recall, Deepeval) are most reliable for measuring RAG answer quality?
• RQ3 What is the optimal top-k value for retrieval in RAG systems when using open-source LLMs on enterprise data?
• RQ4 Does increasing context length (via higher top-k) significantly improve answer quality for different LLMs and question types?
• RQ5 Can open-source LLMs achieve performance comparable to commercial models without requiring expensive GPU infrastructure?

Key findings

• Open-source LLMs such as Llama3-8B achieve answer quality comparable to GPT-4, as measured by Cosine Similarity with Groundtruth Answer (CSGA), which remains stable across varying top-k values.
• CSGA demonstrated minimal variation (typically around 0.5, max 1) across different questions and top-k settings, indicating it is a reliable and consistent metric for RAG evaluation.
• Unigram precision and recall showed significantly higher variability (average changes near 1, max up to 2), suggesting they are less reliable than CSGA for assessing answer quality.
• Increasing top-k did not significantly improve answer quality, as CSGA scores remained largely unchanged, indicating that large context windows are not necessary for effective QA in this setting.
• Llama3-8B outperformed Mistral-8x7B in reasoning-dense and reasoning-sparse questions, despite having fewer parameters, suggesting model architecture and efficiency matter more than size in domain-specific RAG tasks.
• Llama3 and Mistral achieved ~50% lower inference time compared to GPT-3.5 using the perplexity API, demonstrating strong scalability and efficiency for enterprise deployment without high-end GPU costs.