[Paper Review] PTR: Prompt Tuning with Rules for Text Classification

Xu Han, Weilin Zhao+3 more|arXiv (Cornell University)|May 24, 2021

Topic Modeling36 references27 citations

TL;DR

PTR introduces a rules-based prompt-tuning framework for many-class text classification, combining sub-prompts with logic rules to encode prior knowledge and improve performance on relation classification benchmarks.

ABSTRACT

Fine-tuned pre-trained language models (PLMs) have achieved awesome performance on almost all NLP tasks. By using additional prompts to fine-tune PLMs, we can further stimulate the rich knowledge distributed in PLMs to better serve downstream tasks. Prompt tuning has achieved promising results on some few-class classification tasks such as sentiment classification and natural language inference. However, manually designing lots of language prompts is cumbersome and fallible. For those auto-generated prompts, it is also expensive and time-consuming to verify their effectiveness in non-few-shot scenarios. Hence, it is still challenging for prompt tuning to address many-class classification tasks. To this end, we propose prompt tuning with rules (PTR) for many-class text classification and apply logic rules to construct prompts with several sub-prompts. In this way, PTR is able to encode prior knowledge of each class into prompt tuning. We conduct experiments on relation classification, a typical and complicated many-class classification task, and the results show that PTR can significantly and consistently outperform existing state-of-the-art baselines. This indicates that PTR is a promising approach to take advantage of both human prior knowledge and PLMs for those complicated classification tasks.

Motivation & Objective

• Motivate prompt tuning for many-class classification tasks where manual prompts are hard to design.
• Propose a PTR framework that encodes prior knowledge via logic-rule-based sub-prompts.
• Show that composing sub-prompts with conjunctions yields effective task-specific prompts.
• Demonstrate PTR's performance gains on relation classification benchmarks over strong baselines.

Proposed method

• Define a conditional-function set F of unary/binary/multi-variable predicates to capture task priors.
• Design sub-prompts T_f(·) and label word sets V_f for each conditional function f ∈ F.
• Aggregate sub-prompts into a final prompt T(x) using conjunctions in a rule-based manner.
• Use multiple masked positions and token-level verbalizers φ to map [MASK] tokens to class labels.
• Train the model by maximizing the likelihood of correct labels given the composite prompt: maximize (1/|X|) Σ_x log ∏_j p([MASK]_j = φ_j(y) | T(x)).
• Experiment with relation classification datasets TACRED, TACREV, ReTACRED, and SemEval 2010 Task 8 to evaluate effectiveness.

Experimental results

Research questions

• RQ1 Can PTR improve performance on many-class classification tasks by encoding prior knowledge through rules?
• RQ2 How does PTR compare to standard fine-tuning and knowledge-enhanced PLMs on relation classification benchmarks?
• RQ3 What is the impact of reversing relations on PTR performance?
• RQ4 How does PTR perform in few-shot scenarios compared to other prompt-based methods?

Key findings

• PTR significantly and consistently outperforms state-of-the-art baselines on TACRED, TACREV, ReTACRED, and SemEval when using the normal (non-reversed) relation settings.
• Reversing a subset of relations yields substantial improvements for PTR across TACRED, TACREV, and ReTACRED datasets.
• PTR achieves competitive or superior results compared to marker-based prompts and other prompt tuning approaches, while not requiring extra human annotations or neural layers.
• In few-shot settings, PTR can be competitive or superior to some baselines, highlighting the efficiency of rule-based prompt design.
• PTR also shows faster convergence compared to some prompt-based alternatives.