[Paper Review] STaRK: Benchmarking LLM Retrieval on Textual and Relational Knowledge Bases

Shirley Wu, Shiyu Zhao+8 more|arXiv (Cornell University)|Apr 19, 2024

Natural Language Processing Techniques4 citations

TL;DR

STaRK introduces the first comprehensive benchmark for evaluating large language model (LLM)-driven retrieval on semi-structured knowledge bases (SKBs) that integrate textual and relational data. The authors develop a synthetic pipeline to generate realistic, complex queries across e-commerce, academic, and biomedical domains, revealing that current retrieval systems—especially those relying on embeddings—struggle with both accuracy and reasoning, highlighting the need for LLM-augmented reranking to improve performance.

ABSTRACT

Answering real-world complex queries, such as complex product search, often requires accurate retrieval from semi-structured knowledge bases that involve blend of unstructured (e.g., textual descriptions of products) and structured (e.g., entity relations of products) information. However, many previous works studied textual and relational retrieval tasks as separate topics. To address the gap, we develop STARK, a large-scale Semi-structure retrieval benchmark on Textual and Relational Knowledge Bases. Our benchmark covers three domains: product search, academic paper search, and queries in precision medicine. We design a novel pipeline to synthesize realistic user queries that integrate diverse relational information and complex textual properties, together with their ground-truth answers (items). We conduct rigorous human evaluation to validate the quality of our synthesized queries. We further enhance the benchmark with high-quality human-generated queries to provide an authentic reference. STARK serves as a comprehensive testbed for evaluating the performance of retrieval systems driven by large language models (LLMs). Our experiments suggest that STARK presents significant challenges to the current retrieval and LLM systems, highlighting the need for more capable semi-structured retrieval systems. The benchmark data and code are available on https://github.com/snap-stanford/STaRK.

Motivation & Objective

• To address the gap in evaluating retrieval systems that handle both textual and relational knowledge in real-world queries.
• To create a realistic, large-scale benchmark for semi-structured knowledge base (SKB) retrieval that reflects practical applications.
• To evaluate the performance of LLM-driven retrieval systems on complex queries requiring reasoning over both unstructured text and structured relations.
• To assess the trade-offs between retrieval accuracy, latency, and reasoning capability in LLM-augmented retrieval pipelines.
• To provide a standardized testbed for future research in multimodal, hybrid retrieval systems.

Proposed method

• Design a novel, automated pipeline to synthesize natural-sounding, complex user queries that integrate textual descriptions and relational constraints from real SKBs.
• Construct three diverse SKBs from public datasets: STaRK-Amazon (e-commerce), STaRK-MAG (academic publications), and STaRK-Prime (biomedical knowledge).
• Use human evaluation to validate query quality and ensure relevance, realism, and diversity across domains.
• Implement retrieval baselines using dense retrieval, graph neural networks (QAGNN), and vector-similarity search (VSS), with LLM reranking (Claude, GPT-4) to improve accuracy.
• Apply metrics including Recall@20, MRR, and Hit@1 to evaluate retrieval effectiveness across all datasets.
• Measure system latency on a single A100 GPU to assess practical deployment trade-offs.

Experimental results

Research questions

• RQ1 How well do current LLM-based retrieval systems perform on complex, hybrid queries involving both textual and relational knowledge in semi-structured knowledge bases?
• RQ2 To what extent do embedding-based retrieval methods fail to capture the interplay between textual and relational components in queries?
• RQ3 How much does LLM reranking improve retrieval accuracy, especially in cases requiring reasoning over structured relations and unstructured text?
• RQ4 What is the latency-performance trade-off when integrating LLMs into the retrieval pipeline?
• RQ5 Can synthetic query generation produce realistic, diverse, and high-quality benchmarks for evaluating hybrid retrieval systems?

Key findings

• Recall@20 scores across all STaRK datasets are below 60%, indicating significant room for improvement in retrieval comprehensiveness.
• Hit@1 and MRR scores remain low, especially on STaRK-Prime, demonstrating that current models struggle to retrieve the correct answer even in the top-1 position.
• The VSS model with LLM reranking (e.g., VSS+Claude) achieves the highest accuracy, showing that reasoning-enhanced reranking significantly improves results.
• Latency increases substantially when using LLM rerankers: VSS+Claude averages 26.33 seconds on STaRK, compared to 1.40 seconds for the Dense Retriever.
• Case studies show that pure embedding models often misrank documents due to keyword repetition, while LLM rerankers correct these errors by reasoning over query-context relationships.
• The benchmark reveals that current retrieval systems are not robust to queries requiring joint reasoning over textual and relational information, underscoring the need for better integration of LLM reasoning in retrieval pipelines.