[論文レビュー] Don't Forget to Connect! Improving RAG with Graph-based Reranking

Jialin Dong, Bahare Fatemi+3 more|arXiv (Cornell University)|May 28, 2024

Context-Aware Activity Recognition Systems被引用数 6

ひとこと要約

G-RAGはAMRベースの文書グラフを介した文書間接続を用いるグラフニューラルネットワークのリランカーを導入し、RAGを改善し、ベースラインをより低い計算量で上回る。零-shotリランキングでPaLM 2はG-RAGと比較して劣る。

ABSTRACT

Retrieval Augmented Generation (RAG) has greatly improved the performance of Large Language Model (LLM) responses by grounding generation with context from existing documents. These systems work well when documents are clearly relevant to a question context. But what about when a document has partial information, or less obvious connections to the context? And how should we reason about connections between documents? In this work, we seek to answer these two core questions about RAG generation. We introduce G-RAG, a reranker based on graph neural networks (GNNs) between the retriever and reader in RAG. Our method combines both connections between documents and semantic information (via Abstract Meaning Representation graphs) to provide a context-informed ranker for RAG. G-RAG outperforms state-of-the-art approaches while having smaller computational footprint. Additionally, we assess the performance of PaLM 2 as a reranker and find it to significantly underperform G-RAG. This result emphasizes the importance of reranking for RAG even when using Large Language Models.

研究の動機と目的

• Motivate improving RAG performance in ODQA by leveraging connections between retrieved documents.
• Investigate graph-based reranking using AMR-informed document graphs to encode cross-document relationships.
• Reduce computational overhead relative to knowledge-graph or heavy AMR-token approaches.
• Assess the effectiveness of PaLM 2 as a reranker in zero-shot settings.
• Propose robust evaluation metrics that handle ranking ties in reranking scenarios.

提案手法

• Construct a document graph over the top-100 retrieved documents where nodes are documents and edges indicate shared AMR concepts.
• Encode documents with a pre-trained encoder and augment node features with AMR-path-based semantic information.
• Define edge features as number of common nodes and common edges between AMR graphs and normalize them for stability.
• Update node representations with a graph neural network using a message-passing scheme that weights neighbor contributions by edge features.
• Compute a reranking score as a dot product between the question embedding and final node representations; train with pairwise ranking loss to optimize document ordering.
• Evaluate using MRR, MHits@10, and their tied-rank variants (MTRR, TMHits@10) to account for score ties.

実験結果

リサーチクエスチョン

• RQ1 Can document-level connections (via AMR-based graphs) improve document reranking in RAG for ODQA?
• RQ2 How do AMR-informed node/edge features affect reranking performance and computational cost?
• RQ3 Are graph-based rerankers superior to pure LM-based rerankers in zero-shot settings, and how do large LLMs perform as rerankers?
• RQ4 What metrics best capture performance when ranking scores contain ties?
• RQ5 What is the impact of different embedding models on G-RAG performance?

主な発見

戦略	MRR_NQ	MH_NQ	MRR_TQA	MH_TQA
リランカーなし	20.2	18.0	37.9	34.6
BART	25.7	23.3	49.3	45.8
BART-GST	28.4	25.0	53.2	48.7
RGCN-S	26.1	23.1	49.5	46.0
MLP	19.2	17.8	40.0	38.8
GCN	22.6	22.4	47.6	44.2
G-RAG	25.1	24.2	49.1	47.2
G-RAG-RL	27.3	25.7	49.2	47.4
PaLM 2 XS	14.9	14.0	34.1	34.2
PaLM 2 L	18.6	17.9	40.7	39.7
G-RAG-RL (PaLM 2)	27.3	25.7	49.2	47.4

• G-RAG-RL (graph-based reranker with pairwise loss) achieves strong MRR and MHits@10 gains over baselines on Natural Questions and TriviaQA without fine-tuning.
• Incorporating AMR-derived cross-document connections yields performance improvements up to ~7 percentage points over non-graph baselines.
• Among embedding models, Ember, GTE, and BGE provide strong gains, with Ember (HPs-T) offering robust performance after hyperparameter tuning.
• PaLM 2 as a zero-shot reranker underperforms relative to G-RAG variants, highlighting reranker design importance even with strong LLMs.