[Paper Review] Neural Rule-Execution Tracking Machine For Transformer-Based Text Generation

Yufei Wang, Can Xu+6 more|arXiv (Cornell University)|Jul 27, 2021

Topic Modeling30 references4 citations

TL;DR

This paper proposes Neural Rule-Execution Tracking Machine (NRETM), a novel module that enables transformer-based text generators to dynamically track and enforce multiple predicate logic constraints during decoding. By integrating a state matrix and logical tracker that inject context-aware representations into cross-attention, NRETM achieves superior performance in both controllable and general text generation tasks, outperforming strong baselines in zero-shot rule generalization and constraint adherence.

ABSTRACT

Sequence-to-Sequence (S2S) neural text generation models, especially the pre-trained ones (e.g., BART and T5), have exhibited compelling performance on various natural language generation tasks. However, the black-box nature of these models limits their application in tasks where specific rules (e.g., controllable constraints, prior knowledge) need to be executed. Previous works either design specific model structure (e.g., Copy Mechanism corresponding to the rule "the generated output should include certain words in the source input") or implement specialized inference algorithm (e.g., Constrained Beam Search) to execute particular rules through the text generation. These methods require careful design case-by-case and are difficult to support multiple rules concurrently. In this paper, we propose a novel module named Neural Rule-Execution Tracking Machine that can be equipped into various transformer-based generators to leverage multiple rules simultaneously to guide the neural generation model for superior generation performance in a unified and scalable way. Extensive experimental results on several benchmarks verify the effectiveness of our proposed model in both controllable and general text generation.

Motivation & Objective

• To address the limitations of black-box neural text generators in adhering to fine-grained, arbitrary logical constraints.
• To unify constrained generation and prior knowledge integration into a single, scalable framework using predicate logic.
• To design a mechanism that dynamically tracks rule execution progress during decoding, enabling real-time constraint enforcement.
• To overcome the limitations of case-specific model architectures or training objectives in prior work.
• To enable zero-shot generalization to unseen rule configurations, such as altering sentence positions of required keywords.

Proposed method

• NRETM introduces a state matrix to record the progress of each constraint's expression during decoding.
• A logical tracker updates the state matrix based on the latest generated tokens, using executable programs as logic operators.
• State representations are aggregated and injected into the decoder's cross-attention module as relative positional embeddings.
• The framework supports arbitrary predicate logic formulas, enabling flexible and extensible rule definitions.
• The dynamic tracking mechanism allows real-time monitoring and correction of constraint violations during generation.
• The model is integrated into pre-trained sequence-to-sequence architectures like T5, enabling end-to-end training and inference.

Experimental results

Research questions

• RQ1 Can a unified framework enforce multiple, arbitrary predicate logic constraints simultaneously during neural text generation?
• RQ2 How does dynamic rule tracking compare to static constraint enforcement in terms of accuracy and generalization?
• RQ3 Can the model generalize to zero-shot rule configurations not seen during training?
• RQ4 To what extent can NRETM improve generation quality while strictly adhering to complex, fine-grained constraints?
• RQ5 How effective is NRETM in integrating prior knowledge (e.g., coverage, lead bias) into sequence-to-sequence generation?

Key findings

• NRETM achieved 97.7% accuracy in placing required keywords in the correct sentence positions during zero-shot testing, compared to only 19.7% for standard T5.
• In story generation, NRETM maintained 98.3% coverage of required story elements while ensuring 97.7% of them appeared in the correct sentences.
• On the Commonsense Generation benchmark, NRETM significantly outperformed baseline models in generating commonsense-consistent responses under complex constraints.
• On the TED15 Zh-En translation benchmark, NRETM improved generation quality while enforcing document-level coverage constraints.
• The dynamic tracking mechanism significantly outperformed static strategies in both constraint satisfaction and fluency.
• The model demonstrated strong zero-shot generalization, correctly handling novel rule configurations such as reordering required keywords to earlier sentences.