[Paper Review] Low Resource Multi-Task Sequence Tagging -- Revisiting Dynamic Conditional Random Fields

Jonas Pfeiffer, Edwin Simpson+1 more|arXiv (Cornell University)|May 1, 2020

Topic Modeling50 references4 citations

TL;DR

This paper proposes neural factorial conditional random fields (CRFs) to explicitly model inter-task dependencies in low-resource multi-task sequence tagging, outperforming single-task and standard multi-task learning setups. By integrating deep neural networks with dynamic CRFs that capture label correlations across tasks—especially using silver POS tags as auxiliary tasks—it achieves significant performance gains in low-resource settings, particularly for rare or overlapping spans.

ABSTRACT

We compare different models for low resource multi-task sequence tagging that leverage dependencies between label sequences for different tasks. Our analysis is aimed at datasets where each example has labels for multiple tasks. Current approaches use either a separate model for each task or standard multi-task learning to learn shared feature representations. However, these approaches ignore correlations between label sequences, which can provide important information in settings with small training datasets. To analyze which scenarios can profit from modeling dependencies between labels in different tasks, we revisit dynamic conditional random fields (CRFs) and combine them with deep neural networks. We compare single-task, multi-task and dynamic CRF setups for three diverse datasets at both sentence and document levels in English and German low resource scenarios. We show that including silver labels from pretrained part-of-speech taggers as auxiliary tasks can improve performance on downstream tasks. We find that especially in low-resource scenarios, the explicit modeling of inter-dependencies between task predictions outperforms single-task as well as standard multi-task models.

Motivation & Objective

• To address performance degradation in low-resource multi-task sequence tagging due to ignored inter-task label dependencies.
• To investigate whether explicitly modeling dependencies between label sequences of different tasks improves performance in low-resource settings.
• To evaluate the effectiveness of using silver POS tags as auxiliary tasks in multi-task learning for sequence tagging.
• To compare single-task, standard multi-task, and dynamic CRF-based models in diverse low-resource scenarios across English and German datasets.
• To provide a practical, reproducible framework for multi-task sequence tagging with improved generalization in data-scarce environments.

Proposed method

• Proposes a neural factorial CRF model that extends standard linear-chain CRFs to jointly model multiple label sequences across tasks, enabling explicit dependency modeling between task-specific labels.
• Integrates deep neural networks (DNNs) with factorial CRFs to allow end-to-end training of both feature representations and inter-task label dependencies.
• Employs a weighted factorial CRF (WFac) architecture that learns a joint probability matrix C to encode task-specific label correlations, including positive and negative dependencies.
• Uses pre-trained part-of-speech taggers to generate low-cost silver labels as auxiliary tasks, improving performance on downstream tasks without requiring manual annotation.
• Employs a multi-head architecture with shared DNN encoders and task-specific CRF layers, comparing it against the proposed factorial CRF setup.
• Conducts extensive experiments on three diverse datasets (Streusle, FAMULUS, Malware) at both sentence and document levels in English and German, under low-resource conditions.

Experimental results

Research questions

• RQ1 Can explicit modeling of inter-task label dependencies improve performance in low-resource multi-task sequence tagging?
• RQ2 How does the performance of a neural factorial CRF compare to single-task and standard multi-task learning setups in low-resource settings?
• RQ3 To what extent do silver POS tags from pre-trained models serve as effective auxiliary tasks in multi-task sequence tagging?
• RQ4 In which scenarios—especially regarding rare or overlapping spans—does modeling inter-task dependencies yield the greatest performance gains?
• RQ5 Does the inclusion of explicit dependency modeling reduce interference between tasks compared to standard multi-task learning?

Key findings

• The weighted factorial CRF (WFac) model outperforms both single-task and standard multi-task learning models in low-resource settings, particularly for tasks with rare or overlapping spans such as MWE and SSC in the Streusle dataset.
• Explicit modeling of label dependencies via the joint probability matrix C enables the model to downscale unlikely label combinations and upscale likely ones, improving prediction accuracy.
• Performance gains are most significant when tasks are highly interdependent, such as between POS and MWE in the Streusle dataset, where the model learns strong positive and negative correlations between specific tag pairs.
• The use of silver POS tags as auxiliary tasks leads to notable performance improvements on target tasks, even when performance on the auxiliary task itself degrades, due to the transfer of useful dependency signals.
• Standard multi-task learning often causes interference, where performance on some tasks declines despite gains on others, highlighting the limitations of shared representations alone.
• The factorial CRF approach is especially beneficial during early-stage dataset creation, where it can guide annotation through reliable label suggestions, improving speed and inter-annotator agreement.