[Paper Review] GraphText: Graph Reasoning in Text Space

Jianan Zhao, Le Zhuo+6 more|arXiv (Cornell University)|Oct 2, 2023

Topic Modeling4 citations

TL;DR

GRAPHTEXT is a novel framework that enables graph reasoning in text space by converting graphs into natural language prompts via a graph-syntax tree traversal. It achieves training-free graph reasoning with state-of-the-art performance on par with or exceeding supervised GNNs using in-context learning, while enabling interactive, explainable AI-human collaboration in natural language.

ABSTRACT

Large Language Models (LLMs) have gained the ability to assimilate human knowledge and facilitate natural language interactions with both humans and other LLMs. However, despite their impressive achievements, LLMs have not made significant advancements in the realm of graph machine learning. This limitation arises because graphs encapsulate distinct relational data, making it challenging to transform them into natural language that LLMs understand. In this paper, we bridge this gap with a novel framework, GraphText, that translates graphs into natural language. GraphText derives a graph-syntax tree for each graph that encapsulates both the node attributes and inter-node relationships. Traversal of the tree yields a graph text sequence, which is then processed by an LLM to treat graph tasks as text generation tasks. Notably, GraphText offers multiple advantages. It introduces training-free graph reasoning: even without training on graph data, GraphText with ChatGPT can achieve on par with, or even surpassing, the performance of supervised-trained graph neural networks through in-context learning (ICL). Furthermore, GraphText paves the way for interactive graph reasoning, allowing both humans and LLMs to communicate with the model seamlessly using natural language. These capabilities underscore the vast, yet-to-be-explored potential of LLMs in the domain of graph machine learning.

Motivation & Objective

• To bridge the gap between graph-structured data and large language models (LLMs), which traditionally lack direct support for graph reasoning.
• To enable training-free graph reasoning by leveraging in-context learning in LLMs without fine-tuning on graph data.
• To facilitate interactive and explainable graph reasoning by allowing humans and LLMs to communicate in natural language.
• To generalize across diverse graph types, including both continuous-feature and text-attributed graphs, using a unified framework.

Proposed method

• Construct a graph-syntax tree that encodes node attributes (features, labels) and inter-node relationships (e.g., center-node, 1st-hop, 2nd-hop) in a hierarchical structure.
• Traverse the graph-syntax tree in a depth-first manner to generate a sequential text prompt representing the graph in natural language.
• Feed the generated graph prompt into a pre-trained LLM to perform graph reasoning as a text generation task.
• Support both in-context learning (ICL) and instruction tuning (e.g., LoRA fine-tuning on Llama-2) to adapt the LLM for graph tasks.
• Enable multi-modal input handling by mapping continuous node features into the LLM’s embedding space, while treating raw text attributes as discrete tokens.
• Leverage the LLM’s reasoning capabilities to generate step-by-step explanations and predictions, enhancing interpretability.

Experimental results

Research questions

• RQ1 Can graph reasoning be effectively performed in natural language space without fine-tuning on graph data?
• RQ2 How does the performance of LLMs using GRAPHTEXT compare to supervised GNNs in graph classification tasks?
• RQ3 Can GRAPHTEXT support interactive reasoning between humans and LLMs using natural language?
• RQ4 How does the framework generalize across different graph types, including text-attributed and continuous-feature graphs?
• RQ5 What is the impact of input modality (raw text vs. continuous features) on LLM performance in the GRAPHTEXT framework?

Key findings

• GRAPHTEXT with ChatGPT achieves node classification accuracy of 67.77% on Cora and 68.98% on Citeseer using in-context learning, outperforming GNN baselines in some settings.
• Instruction tuning on Llama-2-7B with LoRA achieves 77.53% accuracy on Cora and 73.83% on Citeseer, approaching GNN performance despite using a smaller open-source model.
• Closed-source LLMs like ChatGPT struggle with continuous features, achieving only 10.68% and 16.14% accuracy on Cora and Citeseer, respectively, due to inherent limitations in processing continuous embeddings.
• Open-source LLMs such as Llama-2-7B show improved performance when fine-tuned on continuous features (87.11% on Cora), indicating their adaptability to graph data when properly integrated.
• The framework enables explainable reasoning: LLMs generate step-by-step justifications, such as identifying 1st-hop labels as robust predictors, enhancing interpretability.
• GRAPHTEXT demonstrates that training-free graph reasoning is feasible and effective, with performance matching or surpassing supervised GNNs through in-context learning alone.