[Paper Review] DomainRAG: A Chinese Benchmark for Evaluating Domain-specific Retrieval-Augmented Generation

Shuting Wang, Jiongnan Liu Shiren Song+7 more|arXiv (Cornell University)|Jun 9, 2024

Domain Adaptation and Few-Shot Learning4 citations

TL;DR

This paper introduces DomainRAG, a Chinese benchmark for evaluating retrieval-augmented generation (RAG) models in domain-specific scenarios, focusing on college enrollment. It evaluates LLMs across six key abilities—conversational understanding, structural analysis, faithfulness, denoising, time-sensitive reasoning, and multi-document interaction—using curated HTML and text corpora, revealing that even advanced LLMs struggle with expert-domain tasks without reliable RAG augmentation.

ABSTRACT

Retrieval-Augmented Generation (RAG) offers a promising solution to address various limitations of Large Language Models (LLMs), such as hallucination and difficulties in keeping up with real-time updates. This approach is particularly critical in expert and domain-specific applications where LLMs struggle to cover expert knowledge. Therefore, evaluating RAG models in such scenarios is crucial, yet current studies often rely on general knowledge sources like Wikipedia to assess the models' abilities in solving common-sense problems. In this paper, we evaluated LLMs by RAG settings in a domain-specific context, college enrollment. We identified six required abilities for RAG models, including the ability in conversational RAG, analyzing structural information, faithfulness to external knowledge, denoising, solving time-sensitive problems, and understanding multi-document interactions. Each ability has an associated dataset with shared corpora to evaluate the RAG models' performance. We evaluated popular LLMs such as Llama, Baichuan, ChatGLM, and GPT models. Experimental results indicate that existing closed-book LLMs struggle with domain-specific questions, highlighting the need for RAG models to solve expert problems. Moreover, there is room for RAG models to improve their abilities in comprehending conversational history, analyzing structural information, denoising, processing multi-document interactions, and faithfulness in expert knowledge. We expect future studies could solve these problems better.

Motivation & Objective

• To address the lack of domain-specific evaluation benchmarks for Retrieval-Augmented Generation (RAG) models in expert scenarios.
• To identify and evaluate six critical abilities essential for effective RAG in specialized domains: conversational understanding, structural analysis, faithfulness, denoising, time-sensitive reasoning, and multi-document interaction.
• To assess the performance of leading LLMs (e.g., Llama, Baichuan, ChatGLM, GPT) in a real-world, expert-driven application—college enrollment—using external knowledge.
• To highlight the limitations of closed-book LLMs and the necessity of RAG for reliable expert knowledge retrieval and generation.
• To provide a reproducible, structured benchmark with shared corpora and datasets to enable standardized evaluation of RAG systems in domain-specific contexts.

Proposed method

• Crawled official college enrollment webpages from Chinese higher education institutions to construct two types of corpora: raw text and HTML-structured documents.
• Designed six sub-datasets targeting specific RAG capabilities: conversational RAG, structural information analysis, faithfulness to external knowledge, denoising, time-sensitive problem solving, and multi-document interaction.
• Evaluated multiple LLMs (Llama, Baichuan, ChatGLM, GPT) under RAG settings using the same external knowledge base to ensure fair comparison.
• Introduced controlled noise settings (varying noise counts) to study the impact of irrelevant documents on model performance and confidence.
• Conducted ablation studies comparing closed-book settings, golden references, and noisy references to assess model reliance on external knowledge.
• Used golden and anti-references to evaluate faithfulness, measuring whether models generate correct answers based on provided knowledge rather than their own prior knowledge.

Experimental results

Research questions

• RQ1 How well do existing LLMs perform in a domain-specific RAG setting for college enrollment, compared to closed-book inference?
• RQ2 To what extent can RAG models effectively understand conversational history and maintain context across multi-turn interactions?
• RQ3 How do structural elements (e.g., HTML formatting) in retrieved documents affect the model’s ability to extract and reason over relevant information?
• RQ4 What is the impact of noisy or redundant external documents on the reliability and faithfulness of LLM-generated answers?
• RQ5 How do RAG models handle time-sensitive questions where knowledge changes rapidly, and can they distinguish current from outdated information?

Key findings

• Closed-book LLMs significantly underperform when answering domain-specific college enrollment questions, confirming the necessity of external knowledge retrieval.
• RAG models show limited ability in comprehending conversational history, indicating room for improvement in multi-turn dialogue understanding.
• Models struggle with denoising, as higher noise counts in retrieved documents often degrade performance, though some benefit from increased reference volume.
• The presence of multiple references (even noisy ones) can improve model confidence and performance compared to single-reference or no-reference settings.
• LLMs often generate correct answers even without external knowledge, suggesting they rely on their own static, potentially outdated knowledge—highlighting a critical need for improved faithfulness to external sources.
• Faithfulness to expert knowledge remains a challenge: models frequently fail to reject incorrect or anti-references, indicating a risk of over-reliance on external data without proper validation.