[Paper Review] Integrating Large Language Models into Recommendation via Mutual Augmentation and Adaptive Aggregation

Sichun Luo, Yuxuan Yao+10 more|arXiv (Cornell University)|Jan 25, 2024

Recommender Systems and Techniques4 citations

TL;DR

This paper proposes Llama4Rec, a model-agnostic framework that enhances recommendation performance by jointly leveraging conventional collaborative/sequential models and large language models (LLMs) through mutual data and prompt augmentation, followed by adaptive aggregation of predictions. Empirical results on three real-world datasets show consistent and significant improvements over strong baselines, demonstrating the effectiveness of synergistic integration between traditional and LLM-based recommendation approaches.

ABSTRACT

Conventional recommendation methods have achieved notable advancements by harnessing collaborative or sequential information from user behavior. Recently, large language models (LLMs) have gained prominence for their capabilities in understanding and reasoning over textual semantics, and have found utility in various domains, including recommendation. Conventional recommendation methods and LLMs each have their strengths and weaknesses. While conventional methods excel at mining collaborative information and modeling sequential behavior, they struggle with data sparsity and the long-tail problem. LLMs, on the other hand, are proficient at utilizing rich textual contexts but face challenges in mining collaborative or sequential information. Despite their individual successes, there is a significant gap in leveraging their combined potential to enhance recommendation performance. In this paper, we introduce a general and model-agnostic framework known as extbf{L}arge extbf{la}nguage model with extbf{m}utual augmentation and extbf{a}daptive aggregation for extbf{Rec}ommendation ( extbf{Llama4Rec}). Llama4Rec synergistically combines conventional and LLM-based recommendation models. Llama4Rec proposes data augmentation and prompt augmentation strategies tailored to enhance the conventional model and LLM respectively. An adaptive aggregation module is adopted to combine the predictions of both kinds of models to refine the final recommendation results. Empirical studies on three real-world datasets validate the superiority of Llama4Rec, demonstrating its consistent outperformance of baseline methods and significant improvements in recommendation performance.

Motivation & Objective

• To address data sparsity and long-tail problems in recommendation systems by enhancing conventional collaborative/sequential models with LLM-powered data augmentation.
• To improve LLM-based recommendation by injecting collaborative and sequential signals through prompt engineering, overcoming the LLMs' lack of user-item interaction awareness.
• To develop a general, model-agnostic framework that enables bidirectional enhancement between conventional models and LLMs, rather than unidirectional integration.
• To explore result-level fusion via adaptive aggregation to combine predictions from both model types, optimizing final recommendation quality.
• To provide a comprehensive solution that integrates data-level, prompt-level, and result-level enhancements in a single, unified framework.

Proposed method

• Llama4Rec employs data augmentation to generate synthetic user-item interaction data for conventional recommendation models, improving robustness and generalization.
• It applies prompt augmentation by injecting user behavior sequences and item metadata into LLM prompts, enabling the LLM to reason over collaborative and sequential patterns.
• The framework uses a learnable adaptive aggregation module that dynamically combines predictions from the conventional model and the LLM based on confidence and reliability scores.
• The method is model-agnostic, allowing integration with various backbone models such as LightGCN and BERT-based sequential models.
• Instruction tuning is applied to LLaMA-2 models using task-specific prompts to align LLMs with recommendation objectives.
• Training involves fine-tuning LLMs on augmented instruction datasets, with hyperparameters tuned for model size and data scale.

Experimental results

Research questions

• RQ1 Can mutual augmentation between conventional recommendation models and LLMs lead to consistent performance gains across diverse recommendation tasks?
• RQ2 How does the integration of collaborative and sequential signals via prompt engineering improve LLM-based recommendation quality?
• RQ3 To what extent does adaptive aggregation of predictions from conventional and LLM-based models outperform individual models or simple averaging?
• RQ4 How do model size and instruction data scale affect the performance and efficiency of the LLM component in the framework?
• RQ5 Can the proposed framework be generalized across different datasets and recommendation scenarios without requiring task-specific architectural modifications?

Key findings

• Llama4Rec achieves significant and consistent improvements over strong baselines across three real-world datasets, including Bookcrossing, ML-1M, and Yelp, demonstrating its generalizability.
• The LLaMA-2 (13B) model outperforms the 7B variant, though the performance gain is marginal, indicating diminishing returns with larger model sizes.
• Increasing the number of instruction-tuning examples from 2.5K to 12.5K leads to measurable performance gains, especially for direct recommendation tasks, highlighting the importance of data scale in instruction tuning.
• Training the LLaMA-2 7B model with ~25K instructions on 16 A800 GPUs took approximately 1.94 hours, with inference time of ~0.059 seconds per item, indicating high computational cost.
• The adaptive aggregation module effectively fuses predictions from both model types, leading to better performance than either model alone or simple averaging strategies.
• Despite high computational demands, the framework shows promise for future deployment with advances in efficient inference and parallelization.