[Paper Review] Faithful Path Language Modeling for Explainable Recommendation over Knowledge Graph

Giacomo Balloccu, Ludovico Boratto+3 more|arXiv (Cornell University)|Oct 25, 2023

Advanced Graph Neural Networks4 citations

TL;DR

PEARLM introduces a knowledge graph-constrained decoding mechanism (KGCD) that ensures faithful, hallucination-free path generation in explainable recommendation systems by enforcing valid KG connections during decoding. By combining direct token embedding learning with KGCD, PEARLM achieves state-of-the-art recommendation accuracy and explanation faithfulness, eliminating corrupted paths while remaining model-agnostic and deployable in resource-constrained settings.

ABSTRACT

The integration of path reasoning with language modeling in recommender systems has shown promise for enhancing explainability but often struggles with the authenticity of the explanations provided. Traditional models modify their architecture to produce entities and relations alternately--for example, employing separate heads for each in the model--which does not ensure the authenticity of paths reflective of actual Knowledge Graph (KG) connections. This misalignment can lead to user distrust due to the generation of corrupted paths. Addressing this, we introduce PEARLM (Path-based Explainable-Accurate Recommender based on Language Modelling), which innovates with a Knowledge Graph Constraint Decoding (KGCD) mechanism. This mechanism ensures zero incidence of corrupted paths by enforcing adherence to valid KG connections at the decoding level, agnostic of the underlying model architecture. By integrating direct token embedding learning from KG paths, PEARLM not only guarantees the generation of plausible and verifiable explanations but also highly enhances recommendation accuracy. We validate the effectiveness of our approach through a rigorous empirical assessment, employing a newly proposed metric that quantifies the integrity of explanation paths. Our results demonstrate a significant improvement over existing methods, effectively eliminating the generation of inaccurate paths and advancing the state-of-the-art in explainable recommender systems.

Motivation & Objective

• To address the problem of path hallucination in language model-based explainable recommendation systems.
• To ensure generated explanation paths are factually grounded in the underlying knowledge graph, improving user trust and system reliability.
• To develop a model-agnostic decoding mechanism that enforces KG validity without altering the model architecture.
• To improve recommendation accuracy by directly learning embeddings from KG paths, enhancing both utility and faithfulness.
• To validate the effectiveness of the approach through a novel metric measuring path integrity and user trust.

Proposed method

• Introduces Knowledge Graph-Constrained Decoding (KGCD), a decoding-level mechanism that enforces valid KG transitions (entity-relation alternation) during sequence generation.
• Employs a causal language modeling framework that generates sequences alternating between entities and relations, structurally mirroring KG paths.
• Leverages direct token embedding learning for KG entities and relations, improving representation quality and model performance.
• Uses a novel metric to quantify path integrity, measuring the prevalence of corrupted paths in generated explanations.
• Applies KGCD at inference time, ensuring all generated paths are valid regardless of model architecture or training process.
• Validates the method using a user survey to assess the impact of path hallucinations on trust and satisfaction.

Experimental results

Research questions

• RQ1 To what extent does KGCD eliminate the generation of corrupted or hallucinated paths in explainable recommendation systems?
• RQ2 How does direct token embedding learning from KG paths affect recommendation accuracy and explanation faithfulness?
• RQ3 What is the impact of model size and sample size on the performance of PEARLM, particularly in relation to path length (3-hop vs. 5-hop)?
• RQ4 How do path hallucinations affect user trust and satisfaction in recommendation systems?
• RQ5 Can the KGCD mechanism be applied to existing path-based language models without architectural changes?

Key findings

• PEARLM successfully eliminates the generation of corrupted paths by enforcing valid KG transitions during decoding, as measured by the new path integrity metric.
• The model achieves state-of-the-art recommendation performance, with distilPEARLM outperforming larger variants on both ML1M and LFM1M datasets at optimal sample sizes.
• Shorter 3-hop paths yield better performance than 5-hop paths, indicating that increased path length introduces noise and reduces generalization.
• Direct token embedding learning is the primary driver of performance improvement, significantly outperforming traditional methods.
• User survey results confirm that hallucinated paths severely undermine user trust, validating the necessity of faithfulness in explanations.
• KGCD is model-agnostic and can be applied to any path-based language model to ensure explanation reliability without architectural modifications.