[Paper Review] Incremental Prompting: Episodic Memory Prompt for Lifelong Event Detection

Min‐Qian Liu, Shiyu Chang+1 more|arXiv (Cornell University)|Apr 15, 2022

Domain Adaptation and Few-Shot Learning12 citations

TL;DR

This paper proposes Episodic Memory Prompts (EMP), a novel incremental prompting framework for class-incremental event detection that explicitly retains task-specific knowledge using continuous, learnable prompts for each event type. By freezing BERT and fine-tuning only the prompts while preserving previously learned prompts across tasks, EMP effectively mitigates catastrophic forgetting and achieves state-of-the-art performance on the MAVEN benchmark, outperforming strong baselines in both average and incremental performance.

ABSTRACT

Lifelong event detection aims to incrementally update a model with new event types and data while retaining the capability on previously learned old types. One critical challenge is that the model would catastrophically forget old types when continually trained on new data. In this paper, we introduce Episodic Memory Prompts (EMP) to explicitly preserve the learned task-specific knowledge. Our method adopts continuous prompt for each task and they are optimized to instruct the model prediction and learn event-specific representation. The EMPs learned in previous tasks are carried along with the model in subsequent tasks, and can serve as a memory module that keeps the old knowledge and transferring to new tasks. Experiment results demonstrate the effectiveness of our method. Furthermore, we also conduct a comprehensive analysis of the new and old event types in lifelong learning.

Motivation & Objective

• To address catastrophic forgetting in class-incremental event detection, where models degrade on previously learned event types when trained on new data.
• To develop a parameter-efficient lifelong learning method that avoids fine-tuning the entire model, leveraging prompt tuning instead.
• To explicitly store and transfer task-specific knowledge across incremental learning stages using learnable prompts.
• To reduce reliance on stored exemplars by enabling knowledge transfer through prompt-based memory, improving generalization and robustness.
• To demonstrate the effectiveness of prompt-based continual learning in event detection, a task previously unexplored with prompting techniques.

Proposed method

• The method uses continuous prompts initialized with event type names to represent task-specific knowledge for each new event type in a stream of incremental tasks.
• Each prompt is a learnable embedding vector that is optimized during training to guide the model’s prediction and align with the event mention’s representation.
• An entangled prompt optimization (EPO) mechanism is introduced to jointly optimize the prompt and the model’s prediction by encouraging the prompt to reflect the type-specific distribution of event mentions.
• Knowledge distillation is applied to further regularize the model’s predictions using the soft labels from previous tasks, improving generalization.
• The model retains all previously learned prompts across tasks, forming a soft episodic memory that enables knowledge transfer without requiring task identifiers at inference.
• BERT’s parameters are frozen throughout training, and only the prompt embeddings are fine-tuned, ensuring parameter efficiency and scalability.

Experimental results

Research questions

• RQ1 Can continuous prompts be effectively used to store and transfer task-specific knowledge in a class-incremental event detection setting?
• RQ2 How does the proposed Episodic Memory Prompting (EMP) framework compare to standard fine-tuning and experience replay baselines in mitigating catastrophic forgetting?
• RQ3 To what extent does prompt initialization with event type names improve performance in incremental learning?
• RQ4 How effective is the combination of entangled prompt optimization and knowledge distillation in preserving old task performance?
• RQ5 Can EMP reduce reliance on stored exemplars while maintaining high performance in continual learning?

Key findings

• EMP achieves an average F1 score of 60.26 on the MAVEN benchmark across five task permutations, significantly outperforming the best baseline (KT) and showing strong incremental learning capability.
• Ablation studies show that removing prompt initialization (EInit) reduces performance by 1.13 F1 points on average, confirming its importance for effective prompt learning.
• Removing entangled prompt optimization (EPO) leads to a 1.24 F1 point drop on average, indicating that joint optimization of prompts and predictions is crucial for knowledge retention.
• Knowledge distillation (KD) contributes to a 2.79 F1 point improvement on average, demonstrating its effectiveness in reducing forgetting.
• Using discrete prompts instead of continuous ones degrades performance by 7.73 F1 points on average, proving that continuous prompts are more effective than hard prompts in this setting.
• Even with only 10 exemplars per class, EMP maintains strong performance, outperforming the KT baseline and showing reduced overfitting compared to standard replay methods.