[Paper Review] Learning Contextualized Knowledge Structures for Commonsense Reasoning

Jun Yan, Mrigank Raman+7 more|arXiv (Cornell University)|Oct 24, 2020

Advanced Graph Neural Networks54 references4 citations

TL;DR

This paper proposes Hybrid Graph Network (HGN), a knowledge graph-augmented model that jointly reasons over both extracted and generated commonsense facts within a unified graph structure. By learning to reweight edges and filter out context-irrelevant ones, HGN achieves state-of-the-art performance on four commonsense reasoning benchmarks, including first place on CommonsenseQA and OpenbookQA leaderboards, with human evaluations confirming improved edge validity and helpfulness.

ABSTRACT

Recently, knowledge graph (KG) augmented models have achieved noteworthy success on various commonsense reasoning tasks. However, KG edge (fact) sparsity and noisy edge extraction/generation often hinder models from obtaining useful knowledge to reason over. To address these issues, we propose a new KG-augmented model: Hybrid Graph Network (HGN). Unlike prior methods, HGN learns to jointly contextualize extracted and generated knowledge by reasoning over both within a unified graph structure. Given the task input context and an extracted KG subgraph, HGN is trained to generate embeddings for the subgraph's missing edges to form a "hybrid" graph, then reason over the hybrid graph while filtering out context-irrelevant edges. We demonstrate HGN's effectiveness through considerable performance gains across four commonsense reasoning benchmarks, plus a user study on edge validness and helpfulness.

Motivation & Objective

• To address the limitations of sparse and noisy knowledge graph edges in commonsense reasoning tasks.
• To improve model performance by jointly contextualizing both extracted and generated knowledge facts.
• To reduce reliance on static, heuristic KG subgraphs by learning dynamic, context-aware edge representations.
• To enhance reasoning quality through edge reweighting that filters out irrelevant facts.
• To validate that hybrid knowledge structures yield more valid and helpful edges than static extraction alone.

Proposed method

• HGN constructs a hybrid graph by combining a task-specific KG subgraph (extracted) with PLM-generated edges.
• It learns edge embeddings for missing edges in the subgraph, forming a unified 'hybrid' graph structure.
• Edge reweighting is applied to downweight context-irrelevant edges, with weights learned via attention over node, edge, and text features.
• Message passing is performed with edge weights to attenuate the influence of noisy or irrelevant edges during reasoning.
• The model is trained end-to-end to update graph and text encoders, with a scoring function for prediction.
• Soft pruning is measured via a 'prune rate' metric, indicating the proportion of edges with low weights during training.

Experimental results

Research questions

• RQ1 Can jointly reasoning over both extracted and generated knowledge improve commonsense reasoning performance?
• RQ2 Does edge reweighting in a hybrid graph structure lead to better filtering of irrelevant or noisy facts?
• RQ3 Are the edges produced by HGN more valid and helpful than those from heuristic extraction methods?
• RQ4 How does HGN compare to late-fusion or separate processing of extracted and generated facts?
• RQ5 Can HGN generalize across diverse commonsense reasoning benchmarks with improved robustness?

Key findings

• HGN achieves state-of-the-art performance on four commonsense reasoning benchmarks, outperforming all baselines.
• HGN ranks first on the CommonsenseQA leaderboard and first on the OpenbookQA leaderboard among comparable methods.
• Human evaluation shows HGN-filtered edges are rated significantly higher in validity and helpfulness than heuristically extracted edges.
• The prune rate metric indicates that HGN effectively soft-prunes a substantial portion of irrelevant edges during training.
• The Fleiss’ Kappa scores of 0.51 (validness) and 0.36 (helpfulness) confirm moderate to fair inter-annotator agreement on edge quality.
• The refined graph produced by HGN contains more valid edges and is denser in useful facts compared to the original extracted graph.