[Paper Review] Single Image Unlearning: Efficient Machine Unlearning in Multimodal Large Language Models

Jiaqi Li, Qianshan Wei+6 more|arXiv (Cornell University)|May 21, 2024

Radiomics and Machine Learning in Medical Imaging4 citations

TL;DR

This paper proposes Single Image Unlearning (SIU), an efficient method for forgetting visual concepts in Multimodal Large Language Models (MLLMs) by fine-tuning on just one image per concept. SIU uses four unlearning targets to construct multifaceted fine-tuning data and a novel Dual Masked KL-divergence loss to jointly forget targeted visual knowledge while preserving model utility, outperforming existing methods on a new benchmark, MMUBench, and showing robustness against membership and jailbreak attacks.

ABSTRACT

Machine unlearning empowers individuals with the `right to be forgotten' by removing their private or sensitive information encoded in machine learning models. However, it remains uncertain whether MU can be effectively applied to Multimodal Large Language Models (MLLMs), particularly in scenarios of forgetting the leaked visual data of concepts. To overcome the challenge, we propose an efficient method, Single Image Unlearning (SIU), to unlearn the visual recognition of a concept by fine-tuning a single associated image for few steps. SIU consists of two key aspects: (i) Constructing Multifaceted fine-tuning data. We introduce four targets, based on which we construct fine-tuning data for the concepts to be forgotten; (ii) Jointly training loss. To synchronously forget the visual recognition of concepts and preserve the utility of MLLMs, we fine-tune MLLMs through a novel Dual Masked KL-divergence Loss combined with Cross Entropy loss. Alongside our method, we establish MMUBench, a new benchmark for MU in MLLMs and introduce a collection of metrics for its evaluation. Experimental results on MMUBench show that SIU completely surpasses the performance of existing methods. Furthermore, we surprisingly find that SIU can avoid invasive membership inference attacks and jailbreak attacks. To the best of our knowledge, we are the first to explore MU in MLLMs. We will release the code and benchmark in the near future.

Motivation & Objective

• To address the challenge of forgetting visual concepts in Multimodal Large Language Models (MLLMs) where collecting sufficient training data is impractical.
• To develop an efficient unlearning method that requires only a single image per concept, reducing resource demands.
• To preserve model utility and avoid degradation, such as meaningless or contradictory outputs, during unlearning.
• To establish a comprehensive benchmark, MMUBench, for evaluating machine unlearning in MLLMs.
• To investigate the robustness of unlearning methods against membership inference and jailbreak attacks.

Proposed method

• Constructing multifaceted fine-tuning data based on four unlearning targets: aligning with unseen concepts, assigning new visual descriptions, decoupling factual knowledge, and preserving non-targeted knowledge.
• Introducing a Dual Masked KL-divergence (DMK) loss that applies token-level and vocabulary-level masking to exclude conflicting and targeted concept tokens from KL loss computation.
• Jointly optimizing the DMK loss with cross-entropy loss via gradient descent to balance forgetting and utility preservation.
• Designing a new benchmark, MMUBench, with 20 concepts, each having at least 50 images, where one image is used for unlearning and the rest for generalization evaluation.
• Defining a multi-dimensional evaluation scheme including efficacy, generality, specificity, fluency, and diversity to assess unlearning performance.
• Using gradient-based fine-tuning on a single image per concept, minimizing the need for large-scale forgetting datasets.

Figure 1 : Overview of the Unlearning Process in MLLMs Using SIU. The process starts with a user request to unlearn the visual recognition of concepts, utilizing MMUBench (introduced in Section 5 ) to provide concepts for unlearning. SIU has two elements which are Multifaceted Fine-tuning Data and D

Experimental results

Research questions

• RQ1 Can machine unlearning be effectively applied to Multimodal Large Language Models (MLLMs) for forgetting visual concepts?
• RQ2 Can efficient unlearning be achieved with only a single image per concept, avoiding the need for large-scale forgetting datasets?
• RQ3 Does the proposed SIU method preserve model utility while effectively forgetting targeted visual knowledge?
• RQ4 How does SIU perform compared to existing unlearning methods in terms of efficacy, generality, and robustness?
• RQ5 Can SIU resist invasive attacks such as membership inference and jailbreak attacks?

Key findings

• SIU completely surpasses existing methods across all evaluation metrics on the MMUBench benchmark, demonstrating superior efficacy and generalization.
• The method achieves effective forgetting of visual concepts using only one image per concept, significantly reducing data and computational requirements.
• SIU maintains high fluency and diversity in model responses post-unlearning, avoiding meaningless or contradictory outputs.
• SIU shows robustness against membership inference and jailbreak attacks, suggesting enhanced security properties.
• The proposed Dual Masked KL-divergence loss effectively balances forgetting and utility preservation during fine-tuning.
• The benchmark MMUBench enables comprehensive and standardized evaluation of machine unlearning in MLLMs, setting a new standard for future research.

Figure 6 : Candidate names for targeted unlearning concepts.