[Paper Review] Collaborative Policy Learning for Open Knowledge Graph Reasoning

Cong Fu, Tong Chen+3 more|arXiv (Cornell University)|Aug 31, 2019

Advanced Graph Neural Networks19 references4 citations

TL;DR

This paper proposes Collaborative Policy Learning (CPL), a reinforcement learning framework that jointly trains a reasoning agent and a fact extraction agent to improve open knowledge graph reasoning. The fact extractor dynamically enriches the knowledge graph with relevant facts from text corpora, while the reasoning agent provides feedback to guide the extractor toward useful facts, significantly improving path-finding performance and interpretability on sparse graphs.

ABSTRACT

In recent years, there has been a surge of interests in interpretable graph reasoning methods. However, these models often suffer from limited performance when working on sparse and incomplete graphs, due to the lack of evidential paths that can reach target entities. Here we study open knowledge graph reasoning---a task that aims to reason for missing facts over a graph augmented by a background text corpus. A key challenge of the task is to filter out "irrelevant" facts extracted from corpus, in order to maintain an effective search space during path inference. We propose a novel reinforcement learning framework to train two collaborative agents jointly, i.e., a multi-hop graph reasoner and a fact extractor. The fact extraction agent generates fact triples from corpora to enrich the graph on the fly; while the reasoning agent provides feedback to the fact extractor and guides it towards promoting facts that are helpful for the interpretable reasoning. Experiments on two public datasets demonstrate the effectiveness of the proposed approach. Source code and datasets used in this paper can be downloaded at https://github.com/shanzhenren/CPL

Motivation & Objective

• Address the challenge of low performance in sparse and incomplete knowledge graphs due to lack of evidential paths.
• Enable dynamic knowledge graph augmentation using facts extracted from background text corpora during reasoning.
• Reduce search space and scalability issues by filtering out irrelevant or noisy facts during graph enrichment.
• Improve interpretability of reasoning paths by ensuring only semantically relevant facts are added to the graph.
• Develop a collaborative learning framework where reasoning feedback guides fact extraction policy to enhance overall reasoning performance.

Proposed method

• Train two reinforcement learning agents: a reasoning agent for path finding and a fact extraction agent for selecting relevant facts from text corpora.
• Use a shared environment where the fact extraction agent dynamically adds edges to the knowledge graph based on context and reasoning progress.
• Provide dense reward signals to the reasoning agent when it successfully reaches target entities, which are then backpropagated to the fact extraction agent.
• Employ adaptive sampling during training to generate high-quality positive experiences early, stabilizing policy learning.
• Use a policy gradient-based optimization to jointly train both agents, with the fact extraction policy guided by the reasoning agent's success.
• Integrate the framework with existing path-based reasoning models (e.g., Multi-Hop, ConvE, RotatE) to enhance performance without requiring architectural changes.

Experimental results

Research questions

• RQ1 Can a collaborative reinforcement learning framework improve reasoning performance on sparse and incomplete knowledge graphs by dynamically enriching them with relevant facts from text corpora?
• RQ2 How can the fact extraction policy be trained to filter out irrelevant or noisy facts while still supporting effective path inference?
• RQ3 To what extent does the feedback from the reasoning agent improve the quality of extracted facts over time?
• RQ4 Does the proposed framework maintain interpretability of reasoning paths while improving prediction accuracy?
• RQ5 How does the adaptive sampling strategy affect the convergence and stability of the collaborative policy learning process?

Key findings

• The proposed CPL framework significantly improves knowledge graph reasoning performance on two public datasets, UMLS-PubMed and OpenKE, outperforming strong baselines.
• The fact extraction agent learns to suggest relevant facts with a high sug_edge/pos_path ratio, indicating effective filtering of useful edges.
• The performance curve shows a valley shape during training due to premature stopping of adaptive sampling, but the model recovers and improves as the agent learns a stable exploration policy.
• Abllation studies show that removing adaptive sampling or freezing the extractor degrades performance, confirming the importance of joint training and dynamic adaptation.
• Case studies demonstrate that CPL enables interpretable reasoning by learning path patterns and extracting facts that align with these patterns, enhancing model transparency.
• The framework is generalizable and can be integrated with various path-finding modules (e.g., Multi-Hop with ConvE or RotatE), improving their performance as the reasoning module evolves.