[Paper Review] INTERS: Unlocking the Power of Large Language Models in Search with Instruction Tuning

Yutao Zhu, Peitian Zhang+6 more|arXiv (Cornell University)|Jan 12, 2024

Topic Modeling4 citations

TL;DR

This paper introduces INTERS, a new instruction tuning dataset specifically designed to enhance large language models (LLMs) for information retrieval (IR) tasks. By curating 21 search-related tasks across query understanding, document understanding, and query-document relationship understanding from 43 datasets, and using manually crafted templates, INTERS significantly improves LLM performance on both in-domain and out-of-domain IR tasks, with consistent gains across models like LLaMA, Mistral, and Phi.

ABSTRACT

Large language models (LLMs) have demonstrated impressive capabilities in various natural language processing tasks. Despite this, their application to information retrieval (IR) tasks is still challenging due to the infrequent occurrence of many IR-specific concepts in natural language. While prompt-based methods can provide task descriptions to LLMs, they often fall short in facilitating a comprehensive understanding and execution of IR tasks, thereby limiting LLMs' applicability. To address this gap, in this work, we explore the potential of instruction tuning to enhance LLMs' proficiency in IR tasks. We introduce a novel instruction tuning dataset, INTERS, encompassing 20 tasks across three fundamental IR categories: query understanding, document understanding, and query-document relationship understanding. The data are derived from 43 distinct datasets with manually written templates. Our empirical results reveal that INTERS significantly boosts the performance of various publicly available LLMs, such as LLaMA, Mistral, and Phi, in IR tasks. Furthermore, we conduct extensive experiments to analyze the effects of instruction design, template diversity, few-shot demonstrations, and the volume of instructions on performance. We make our dataset and the fine-tuned models publicly accessible at https://github.com/DaoD/INTERS.

Motivation & Objective

• To address the limited effectiveness of LLMs in information retrieval (IR) tasks due to infrequent exposure to IR-specific concepts like relevance and user intent.
• To close the gap in instruction tuning datasets tailored for IR by creating a specialized, comprehensive dataset for search-related tasks.
• To investigate how instruction design, data volume, and few-shot learning impact LLM performance in IR through systematic experiments.
• To enable better generalization of LLMs to unseen IR tasks through instruction tuning on diverse, task-specific templates.

Proposed method

• The authors collect 43 existing datasets spanning 21 search-related tasks, grouped into three categories: query understanding, document understanding, and query-document relationship understanding.
• For each task, they manually design a task description and 12 unique input templates to guide model behavior and improve task comprehension.
• The dataset is constructed by mixing all examples from the 43 datasets, with each example formatted using the corresponding instruction and template.
• Fine-tuning is performed on multiple open-source LLMs (e.g., LLaMA, Mistral, Phi) using the INTERS dataset to adapt them for IR tasks.
• Ablation studies are conducted to evaluate the impact of templates, data volume, and few-shot learning by comparing configurations with and without these components.
• Zero-shot and few-shot inference is evaluated on unseen tasks to assess generalization, with performance measured using standard IR metrics.

Experimental results

Research questions

• RQ1 How does instruction tuning on a task-specific dataset like INTERS improve LLM performance across diverse IR tasks?
• RQ2 What is the impact of manually designed instruction templates on model performance and task understanding?
• RQ3 How does the volume of training data affect the generalization and performance of instruction-fine-tuned LLMs in IR?
• RQ4 To what extent do few-shot examples enhance the model’s ability to generalize to out-of-domain IR tasks?
• RQ5 How do different base models (e.g., LLaMA, Mistral, Phi) benefit from fine-tuning on INTERS across varying task types?

Key findings

• INTERS significantly improves the performance of multiple open-source LLMs across both in-domain and out-of-domain IR tasks, with consistent gains observed across LLaMA, Mistral, and Phi models.
• The use of instructional templates during training leads to a measurable performance boost, as ablation without templates results in inferior results, confirming their importance in task comprehension.
• Increasing the volume of training data consistently improves model performance, though the sensitivity to data quantity varies by task, with reranking tasks benefiting most from more data.
• Few-shot examples consistently improve performance across all tasks, particularly in complex output spaces such as reading comprehension (BoolQ), where they help clarify task and output format.
• The model fine-tuned on INTERS outperforms models fine-tuned on FLAN, especially in reranking tasks, demonstrating the advantage of task-specific instruction tuning for IR.
• The query reformulation task shows stable performance across different data volumes, suggesting it is less data-sensitive due to its straightforward nature.