[Paper Review] Multitask Learning for Class-Imbalanced Discourse Classification

Alexander Spangher, Jonathan May+2 more|arXiv (Cornell University)|Jan 2, 2021

Topic Modeling61 references4 citations

TL;DR

This paper proposes a multitask learning framework that improves discourse classification on the class-imbalanced NewsDiscourse dataset by leveraging complementary signals from six diverse discourse datasets, including a novel 67-sample dataset based on Van Dijk's schema. The approach boosts Micro F1-score by 7% (from 62.8% to 67.7%), with significant gains for underrepresented classes due to cross-task label correlations, while data augmentation and other standard techniques fail to improve performance.

ABSTRACT

Small class-imbalanced datasets, common in many high-level semantic tasks like discourse analysis, present a particular challenge to current deep-learning architectures. In this work, we perform an extensive analysis on sentence-level classification approaches for the News Discourse dataset, one of the largest high-level semantic discourse datasets recently published. We show that a multitask approach can improve 7% Micro F1-score upon current state-of-the-art benchmarks, due in part to label corrections across tasks, which improve performance for underrepresented classes. We also offer a comparative review of additional techniques proposed to address resource-poor problems in NLP, and show that none of these approaches can improve classification accuracy in such a setting.

Motivation & Objective

• To address the challenge of class imbalance in sentence-level discourse classification, particularly on small, high-level semantic datasets like NewsDiscourse.
• To investigate whether combining multiple discourse datasets with different schemas can improve performance on a primary task through multitask learning.
• To evaluate whether established techniques for handling class imbalance—such as hierarchical labeling, modified loss functions, and CRF modeling—can improve performance in low-resource, imbalanced settings.
• To demonstrate that labeled signal from auxiliary tasks is more effective than data expansion alone in such scenarios.
• To provide a unified framework that reuses existing discourse annotations across divergent schemas without discarding prior annotation work.

Proposed method

• A multitask neural architecture is trained jointly on seven discourse datasets, including NewsDiscourse (VD1), a novel 67-sample dataset (VD3), and five others with varying schemas and label types (multiclass/multilabel).
• The model shares a shared encoder (e.g., BERT-based) across all tasks and uses task-specific heads for classification, enabling parameter sharing and cross-task knowledge transfer.
• Task-specific losses are weighted adaptively, and the framework supports both multiclass and multilabel classification tasks across different discourse schemas.
• Inter-task prediction correlations are analyzed to understand how auxiliary tasks support underrepresented classes in the primary task.
• The framework incorporates both labeled and, where applicable, filtered subsets of existing datasets (e.g., PDTB, RST, KBP) to maximize signal reuse.
• Ablation studies compare the multitask model against baselines using data augmentation, hierarchical labeling, CRF decoding, and alternative loss functions.

Experimental results

Research questions

• RQ1 Can multitask learning improve performance on a primary discourse classification task when the dataset is small and class-imbalanced?
• RQ2 Do complementary signals from multiple discourse datasets with different schemas improve classification accuracy, especially for underrepresented classes?
• RQ3 Why does simple data augmentation fail to improve performance in this setting, despite increasing training size?
• RQ4 Do standard techniques for handling class imbalance—such as hierarchical labeling, CRF modeling, or focal loss—improve performance on this specific dataset?
• RQ5 To what extent do prediction correlations between tasks explain the performance gains observed in the multitask framework?

Key findings

• The multitask learning approach achieves a 7% absolute improvement in Micro F1-score on the NewsDiscourse dataset, rising from 62.8% to 67.7%.
• The largest performance gains are observed for underrepresented discourse classes, indicating that multitask learning effectively mitigates class imbalance.
• Data augmentation alone fails to improve performance and even degrades overall accuracy, as it overfits to overrepresented classes.
• None of the standard techniques—hierarchical labeling, CRF-based sequence modeling, or modified loss functions—improve performance over the baseline.
• High inter-task prediction correlations are observed between the primary task and auxiliary tasks, suggesting that label signals from diverse schemas provide meaningful inductive bias for underrepresented classes.
• The framework successfully reuses existing discourse annotations across different schemas, enabling knowledge transfer without discarding prior annotation work.