[Paper Review] A Survey of Graph Meets Large Language Model: Progress and Future Directions

Yuhan Li, Zhixun Li+5 more|arXiv (Cornell University)|Nov 21, 2023

Advanced Graph Neural Networks4 citations

TL;DR

This survey introduces a novel taxonomy categorizing LLM-graph integration into three roles: LLM as enhancer, predictor, and alignment component. It systematically reviews representative methods, identifies key challenges in efficiency, explainability, and expressive power, and outlines future research directions for advancing graph learning with LLMs.

ABSTRACT

Graph plays a significant role in representing and analyzing complex relationships in real-world applications such as citation networks, social networks, and biological data. Recently, Large Language Models (LLMs), which have achieved tremendous success in various domains, have also been leveraged in graph-related tasks to surpass traditional Graph Neural Networks (GNNs) based methods and yield state-of-the-art performance. In this survey, we first present a comprehensive review and analysis of existing methods that integrate LLMs with graphs. First of all, we propose a new taxonomy, which organizes existing methods into three categories based on the role (i.e., enhancer, predictor, and alignment component) played by LLMs in graph-related tasks. Then we systematically survey the representative methods along the three categories of the taxonomy. Finally, we discuss the remaining limitations of existing studies and highlight promising avenues for future research. The relevant papers are summarized and will be consistently updated at: https://github.com/yhLeeee/Awesome-LLMs-in-Graph-tasks.

Motivation & Objective

• To provide a comprehensive review of existing methods integrating large language models (LLMs) with graph data.
• To address the limitations of traditional Graph Neural Networks (GNNs) in capturing semantic and contextual node features by leveraging LLMs.
• To propose a new taxonomy based on the functional role of LLMs in graph tasks—enhancer, predictor, and alignment component.
• To identify key challenges such as efficiency, explainability, and expressive power in LLM-graph integration.
• To highlight promising future research directions, including LLMs as agents and improved parameter-efficient adaptation.

Proposed method

• Proposes a novel three-category taxonomy: LLM as enhancer (improving node features), predictor (replacing GNN heads), and alignment component (unifying modalities).
• Reviews representative methods under each category, including TAPE (LLM-enhanced node embeddings), InstructGLM (LLM-based prediction via instruction prompting), and MoleculeSTM (cross-modal alignment).
• Analyzes techniques like prompt engineering, fine-tuning (e.g., LoRA, prefix tuning), and graph flattening to adapt LLMs to graph structures.
• Examines methods that align GNN and LLM representations in a shared vector space to improve reasoning and knowledge infusion.
• Explores the use of in-context learning and chain-of-thought prompting to enhance model interpretability and reasoning in graph tasks.
• Discusses architectural and training strategies to improve efficiency, such as parameter-efficient fine-tuning for large-scale graph inference.

Experimental results

Research questions

• RQ1 How can LLMs be systematically categorized based on their functional roles in graph-related tasks?
• RQ2 What are the most effective techniques for integrating LLMs with graph neural networks to improve performance beyond GNNs?
• RQ3 How do LLMs enhance explainability in graph reasoning tasks through natural language generation?
• RQ4 What are the key limitations in efficiency, scalability, and expressive power when applying LLMs to graph data?
• RQ5 Can LLMs be reimagined as agents with planning, memory, and tool-use capabilities to solve complex graph tasks?

Key findings

• LLM integration significantly improves performance on graph tasks compared to traditional GNNs, especially in scenarios requiring rich semantic understanding.
• Methods like TAPE and LLM4Mol demonstrate that LLM-generated semantic descriptions enhance initial node embeddings, leading to improved downstream accuracy.
• InstructGLM achieves state-of-the-art results by replacing GNN predictors with LLMs via instruction prompting and graph flattening, enabling zero-shot generalization.
• Cross-modal alignment techniques such as MoleculeSTM and G-Prompt show that aligning LLM and GNN representations improves reasoning in molecular and knowledge graph tasks.
• Parameter-efficient fine-tuning methods like LoRA and prefix tuning reduce computational costs and enable efficient adaptation of LLMs to graph data.
• Despite progress, challenges remain in model efficiency, especially for large-scale graphs, due to high inference costs and hardware demands when using API-based LLMs.