[Paper Review] Structured Prediction as Translation between Augmented Natural Languages

Giovanni Paolini, Ben Athiwaratkun+7 more|arXiv (Cornell University)|Jan 14, 2021

Topic ModelingComputer Science69 references36 citations

TL;DR

TANL reframes diverse structured prediction tasks as translation between augmented natural languages, enabling a single generative model to achieve state-of-the-art results across multiple NLP tasks with shared architecture and inputs.

ABSTRACT

We propose a new framework, Translation between Augmented Natural Languages (TANL), to solve many structured prediction language tasks including joint entity and relation extraction, nested named entity recognition, relation classification, semantic role labeling, event extraction, coreference resolution, and dialogue state tracking. Instead of tackling the problem by training task-specific discriminative classifiers, we frame it as a translation task between augmented natural languages, from which the task-relevant information can be easily extracted. Our approach can match or outperform task-specific models on all tasks, and in particular, achieves new state-of-the-art results on joint entity and relation extraction (CoNLL04, ADE, NYT, and ACE2005 datasets), relation classification (FewRel and TACRED), and semantic role labeling (CoNLL-2005 and CoNLL-2012). We accomplish this while using the same architecture and hyperparameters for all tasks and even when training a single model to solve all tasks at the same time (multi-task learning). Finally, we show that our framework can also significantly improve the performance in a low-resource regime, thanks to better use of label semantics.

Motivation & Objective

• Motivation to unify diverse structured prediction tasks under a single generative framework.
• Leverage latent label semantics by using augmented natural languages rather than task-specific discriminators.
• Demonstrate competitive or state-of-the-art performance across multiple SP tasks with the same architecture and hyperparameters.
• Show benefits in low-resource regimes via improved label-semantic transfer.
• Explore multi-task and single-model training across several SP tasks.

Proposed method

• Design augmented natural languages that encode structured information into the input/output text.
• Frame SP tasks as text-to-text translation using a pre-trained Transformer model (T5-base) with the same architecture across tasks.
• Implement a DP-based alignment (Needleman-Wunsch) to robustly map output structure back to the input tokens.
• Provide task-specific output formats (e.g., for NER, joint entity/relation extraction, SRL, coreference, DST) within a unified translation paradigm.
• Allow multi-task learning by prefixing inputs with dataset/task identifiers (e.g., ‘ade:’) to train a single model on multiple datasets.

Experimental results

Research questions

• RQ1 Can a single generative model solve multiple structured prediction tasks without task-specific modules?
• RQ2 Does encoding task semantics via augmented natural language improve transfer, especially in low-resource settings?
• RQ3 How does multi-task training on diverse SP tasks compare to single-task models in performance?
• RQ4 Can the generative TANL approach support complex structures like nested entities and arbitrary relation graphs effectively?

Key findings

• TANL achieves state-of-the-art results on joint entity and relation extraction (CoNLL04, ADE, NYT, ACE2005).
• TANL achieves state-of-the-art results on relation classification (FewRel and TACRED) and semantic role labeling (CoNLL-2005 and CoNLL-2012).
• A single model trained across multiple tasks (multi-task) performs comparably or better than single-task models on the evaluated datasets.
• In low-resource regimes, TANL shows significant improvements due to better use of label semantics.
• Output likelihood of the generated sequence can serve as a robust proxy for class scores in discriminative evaluation scenarios.