[Paper Review] Learning Robust Representations for Continual Relation Extraction via Adversarial Class Augmentation

Peiyi Wang, Yifan Song+5 more|arXiv (Cornell University)|Oct 10, 2022

Data Stream Mining Techniques4 citations

TL;DR

This paper proposes Adversarial Class Augmentation (ACA), a model-agnostic method that improves continual relation extraction (CRE) by enhancing representation robustness against analogous relations. By generating hard negative classes via hybrid and reversed class augmentation during training, ACA reduces catastrophic forgetting—especially for relations with similar semantics—consistently boosting state-of-the-art CRE models on two benchmarks.

ABSTRACT

Continual relation extraction (CRE) aims to continually learn new relations from a class-incremental data stream. CRE model usually suffers from catastrophic forgetting problem, i.e., the performance of old relations seriously degrades when the model learns new relations. Most previous work attributes catastrophic forgetting to the corruption of the learned representations as new relations come, with an implicit assumption that the CRE models have adequately learned the old relations. In this paper, through empirical studies we argue that this assumption may not hold, and an important reason for catastrophic forgetting is that the learned representations do not have good robustness against the appearance of analogous relations in the subsequent learning process. To address this issue, we encourage the model to learn more precise and robust representations through a simple yet effective adversarial class augmentation mechanism (ACA), which is easy to implement and model-agnostic. Experimental results show that ACA can consistently improve the performance of state-of-the-art CRE models on two popular benchmarks.

Motivation & Objective

• To investigate the root causes of catastrophic forgetting in continual relation extraction (CRE), particularly focusing on relations with analogous semantics.
• To challenge the common assumption in prior work that models adequately learn old relations before new ones are introduced.
• To address the overlooked issue of non-robust representations caused by shortcut learning during initial training on easy tasks.
• To develop a simple, effective, and model-agnostic method that improves representation robustness before catastrophic forgetting occurs.
• To demonstrate that adversarial class augmentation can mitigate forgetting in subsequent tasks, especially when analogous relations appear.

Proposed method

• ACA introduces two class augmentation strategies: hybrid-class augmentation and reversed-class augmentation, to generate hard negative examples for new relations.
• During training, new relations are jointly learned with adversarially augmented classes to encourage the model to form more discriminative and robust representations.
• The method is applied during the initial training phase of each new task, acting as a preventive measure against future forgetting.
• ACA is orthogonal to existing rehearsal-based methods, which focus on knowledge retention after forgetting has occurred, making it complementary.
• The approach is designed to be lightweight and applicable to any backbone CRE model without architectural modifications.
• Hard negative classes are constructed by combining features or labels from semantically similar but distinct relations, increasing the difficulty of distinguishing them during training.

Experimental results

Research questions

• RQ1 Why does catastrophic forgetting in CRE disproportionately affect certain relations, especially when their analogous relations appear later?
• RQ2 To what extent is the failure of existing CRE models due to non-robust representations formed during early training on easy, non-challenging data?
• RQ3 Can adversarial data augmentation improve representation robustness in CRE, particularly for relations with high semantic similarity?
• RQ4 How does ACA compare to existing rehearsal-based methods in terms of performance gain and forgetting mitigation?
• RQ5 Does the effectiveness of ACA vary with memory size or the similarity of relations in the dataset?

Key findings

• Catastrophic forgetting in CRE is strongly correlated with the presence of analogous relations; performance drops significantly when such relations appear in later tasks.
• The EMAR model suffers a 14-point F1 drop on relation P25 when P26 is introduced, indicating severe forgetting due to non-robust representations.
• With ACA, the same model only experiences a 4-point F1 drop on P25 when P26 is introduced, demonstrating significant forgetting mitigation.
• On FewRel, EMAR+ACA outperforms EMAR by 2.8, 1.1, and 0.7 accuracy points at memory sizes of 5, 10, and 20, respectively, showing consistent gains across settings.
• Error analysis shows ACA most effectively reduces forgetting for relations with high maximum similarity (MS), though performance gaps relative to supervised baselines remain for these hard cases.
• ACA improves F1 scores by up to 0.9 points in the highest similarity group (G3), indicating its effectiveness for semantically similar relations.