[Paper Review] Harnessing Explanations: LLM-to-LM Interpreter for Enhanced Text-Attributed Graph Representation Learning

Xiaoxin He, Xavier Bresson+4 more|arXiv (Cornell University)|May 31, 2023

Topic Modeling16 citations

TL;DR

The paper introduces TAPE, an LMaaS-compatible framework that uses LLMs to generate predictions and explanations for text attributes on text-attributed graphs, then a fine-tuned LM interpreter converts these explanations into enriched node features for downstream GNNs, achieving SOTA results with faster training.

ABSTRACT

Representation learning on text-attributed graphs (TAGs) has become a critical research problem in recent years. A typical example of a TAG is a paper citation graph, where the text of each paper serves as node attributes. Initial graph neural network (GNN) pipelines handled these text attributes by transforming them into shallow or hand-crafted features, such as skip-gram or bag-of-words features. Recent efforts have focused on enhancing these pipelines with language models (LMs), which typically demand intricate designs and substantial computational resources. With the advent of powerful large language models (LLMs) such as GPT or Llama2, which demonstrate an ability to reason and to utilize general knowledge, there is a growing need for techniques which combine the textual modelling abilities of LLMs with the structural learning capabilities of GNNs. Hence, in this work, we focus on leveraging LLMs to capture textual information as features, which can be used to boost GNN performance on downstream tasks. A key innovation is our use of explanations as features: we prompt an LLM to perform zero-shot classification, request textual explanations for its decision-making process, and design an LLM-to-LM interpreter to translate these explanations into informative features for downstream GNNs. Our experiments demonstrate that our method achieves state-of-the-art results on well-established TAG datasets, including Cora, PubMed, ogbn-arxiv, as well as our newly introduced dataset, tape-arxiv23. Furthermore, our method significantly speeds up training, achieving a 2.88 times improvement over the closest baseline on ogbn-arxiv. Lastly, we believe the versatility of the proposed method extends beyond TAGs and holds the potential to enhance other tasks involving graph-text data. Our codes and datasets are available at: https://github.com/XiaoxinHe/TAPE.

Motivation & Objective

• Motivate improved TAG representation learning by combining LLM reasoning with GNNs to leverage rich textual information.
• Propose a modular LMaaS-compatible pipeline that uses LLM explanations as features for downstream models.
• Demonstrate state-of-the-art performance on TAG benchmarks and show training efficiency gains over prior LM-GNN approaches.
• Provide open-source datasets and code to support reproducible TAG research.

Proposed method

• Query LLMs in a prompt-based, LMaaS-compatible manner to obtain ranked predictions and textual explanations for each node’s text (title and abstract).
• Fine-tune a smaller LM as an interpreter for the LLM explanations, producing enriched fixed-length node features from original text and explanations.
• Extract top-k LLM predictions and transform them into a prediction feature vector for each node.
• Train a GNN on the augmented features in an ensemble fashion by averaging predictions from models trained on original, explanation, and prediction features.

Experimental results

Research questions

• RQ1 Can explanations generated by LLMs serve as informative features to improve text-attributed graph representations?
• RQ2 Does an LM-to-LM interpreter effectively translate LLM explanations into discriminative node embeddings for GNNs?
• RQ3 How does the proposed LMaaS-compatible pipeline compare to existing LM-based and GNN-based TAG methods in accuracy and training time?
• RQ4 What is the impact of including ranked predictions and explanations on downstream node classification across standard TAG datasets?

Key findings

• The TAPE framework achieves state-of-the-art node classification performance on Cora, PubMed, ogbn-arxiv, and tape-arxiv23 TAG datasets.
• Using explanations as features provides measurable gains over shallow features and prior LM-based features.
• The approach yields substantial training-time speedups (e.g., 2.88× faster than the GLEM baseline on ogbn-arxiv) by decoupling LM and GNN training.
• An ensemble fusion of orig, expl, and pred feature-based GNNs consistently improves performance over individual components.
• A new tape-arxiv23 dataset demonstrates generalization beyond the LLM knowledge cutoff (GPT-3.5) with strong results.