[Paper Review] Lexically-constrained Text Generation through Commonsense Knowledge Extraction and Injection

Yikang Li, Pulkit Goel+6 more|arXiv (Cornell University)|Dec 19, 2020

Topic Modeling30 references6 citations

TL;DR

This paper proposes a method to improve lexically-constrained text generation by injecting commonsense knowledge from ConceptNet into the Unified Language Model (UniLM) via attention mechanisms and enforcing lexical constraints through modified beam search. The approach significantly enhances semantic correctness and concept coverage, reducing missing concepts by up to 15% and improving alignment with human common sense.

ABSTRACT

Conditional text generation has been a challenging task that is yet to see human-level performance from state-of-the-art models. In this work, we specifically focus on the Commongen benchmark, wherein the aim is to generate a plausible sentence for a given set of input concepts. Despite advances in other tasks, large pre-trained language models that are fine-tuned on this dataset often produce sentences that are syntactically correct but qualitatively deviate from a human understanding of common sense. Furthermore, generated sequences are unable to fulfill such lexical requirements as matching part-of-speech and full concept coverage. In this paper, we explore how commonsense knowledge graphs can enhance model performance, with respect to commonsense reasoning and lexically-constrained decoding. We propose strategies for enhancing the semantic correctness of the generated text, which we accomplish through: extracting commonsense relations from Conceptnet, injecting these relations into the Unified Language Model (UniLM) through attention mechanisms, and enforcing the aforementioned lexical requirements through output constraints. By performing several ablations, we find that commonsense injection enables the generation of sentences that are more aligned with human understanding, while remaining compliant with lexical requirements.

Motivation & Objective

• Address the failure of pre-trained language models to generate semantically coherent, commonsense-compliant sentences under lexical constraints.
• Overcome limitations in concept coverage and part-of-speech (POS) accuracy in text generation tasks like CommonGen.
• Enhance model performance by integrating external commonsense knowledge from ConceptNet into the language model architecture.
• Ensure generated sentences strictly include all input concepts with correct POS tags through constrained decoding strategies.

Proposed method

• Extract commonsense relations between input concepts from ConceptNet, prioritizing high-frequency, POS-aligned relations.
• Inject extracted relations into UniLM using attention mechanisms, particularly in early layers, to enrich contextual representations.
• Implement a modified beam search algorithm that scores candidate sequences based on both generation likelihood and lexical constraint compliance.
• Apply Best-N beam scoring to favor longer, concept-rich sequences without sacrificing fluency or coherence.
• Use knowledge selection strategies to filter irrelevant or noisy expansion concepts, reducing interference during generation.
• Integrate POS tagging into the knowledge injection and decoding processes to maintain grammatical correctness of generated tokens.

Experimental results

Research questions

• RQ1 Can injecting commonsense knowledge from ConceptNet improve the semantic plausibility of lexically-constrained text generation?
• RQ2 How does attention-based knowledge injection compare to non-attention methods in enhancing generation quality?
• RQ3 To what extent can constrained beam search decoding reduce missing concepts and concept repetition?
• RQ4 Does selecting high-frequency, POS-aligned relations from ConceptNet lead to better performance than unfiltered injection?
• RQ5 Can a combination of knowledge injection and constrained decoding significantly improve alignment with human common sense?

Key findings

• Knowledge injection via attention mechanisms improved semantic correctness, producing sentences more aligned with human understanding of common sense.
• Best-N beam scoring reduced missing concept rates by up to 15% compared to standard beam search, especially benefiting concatenation-based injection methods.
• Models with POS-aware knowledge selection achieved over 2 percentage points higher accuracy in matching required part-of-speech tags.
• Filtering expansion concepts by frequency reduced noise and improved generation quality, indicating that not all extracted relations are beneficial.
• The combination of attention-based injection and constrained decoding eliminated concept repetition in final outputs, unlike the baseline UniLM.
• Ablation studies confirmed that both knowledge injection and constrained decoding independently contribute to performance gains, with synergistic effects when combined.