[Paper Review] CTP: Towards Vision-Language Continual Pretraining via Compatible Momentum Contrast and Topology Preservation

Hongguang Zhu, Yunchao Wei+3 more|arXiv (Cornell University)|Aug 14, 2023

Multimodal Machine Learning ApplicationsComputer Science3 citations

TL;DR

This paper proposes CTP, a novel method for Vision-Language Continual Pretraining (VLCP) that combines Compatible Momentum Contrast and Topology Preservation to enable efficient, stable, and accurate continual learning on ever-expanding vision-language data. CTP achieves state-of-the-art performance with 50.53% TR@1 and 70.63% Rm on the P9D benchmark, outperforming prior methods by up to 5.76% while avoiding memory overhead.

ABSTRACT

Vision-Language Pretraining (VLP) has shown impressive results on diverse downstream tasks by offline training on large-scale datasets. Regarding the growing nature of real-world data, such an offline training paradigm on ever-expanding data is unsustainable, because models lack the continual learning ability to accumulate knowledge constantly. However, most continual learning studies are limited to uni-modal classification and existing multi-modal datasets cannot simulate continual non-stationary data stream scenarios. To support the study of Vision-Language Continual Pretraining (VLCP), we first contribute a comprehensive and unified benchmark dataset P9D which contains over one million product image-text pairs from 9 industries. The data from each industry as an independent task supports continual learning and conforms to the real-world long-tail nature to simulate pretraining on web data. We comprehensively study the characteristics and challenges of VLCP, and propose a new algorithm: Compatible momentum contrast with Topology Preservation, dubbed CTP. The compatible momentum model absorbs the knowledge of the current and previous-task models to flexibly update the modal feature. Moreover, Topology Preservation transfers the knowledge of embedding across tasks while preserving the flexibility of feature adjustment. The experimental results demonstrate our method not only achieves superior performance compared with other baselines but also does not bring an expensive training burden. Dataset and codes are available at https://github.com/KevinLight831/CTP.

Motivation & Objective

• To address the lack of large-scale, realistic, continual pretraining datasets for vision-language models.
• To identify and analyze new challenges in Vision-Language Continual Pretraining (VLCP), including fixed-dimensional embedding and missing contrast samples.
• To propose a memory-efficient, high-performing method that enables stable knowledge accumulation across tasks without catastrophic forgetting.
• To establish a unified benchmark, P9D, with over 1 million image-text pairs from 9 industries to simulate real-world, long-tailed, non-stationary data streams.

Proposed method

• Introduces a compatible momentum contrast (CMC) module that dynamically updates the momentum encoder by fusing knowledge from both current and previous task models.
• Employs topology preservation to maintain consistent sample relationships across tasks, preserving the structural organization of the embedding space.
• Uses a momentum queue to store negative samples from the current task, stabilizing training and reducing forgetting.
• Applies soft constraints on same-modal similarity to prevent performance degradation from maximum similarity interference.
• Jointly optimizes image, text, and multi-modal encoders through a unified loss that balances plasticity and stability.
• Enables continual pretraining without storing old data, achieving memory efficiency while maintaining high performance.

Experimental results

Research questions

• RQ1 How can vision-language models be effectively and efficiently fine-tuned on continuously arriving data without catastrophic forgetting?
• RQ2 What are the key challenges unique to vision-language continual pretraining compared to traditional class-incremental learning?
• RQ3 Can a memory-free method achieve performance comparable to memory-buffer approaches in VLCP?
• RQ4 How do compatible momentum contrast and topology preservation jointly improve model stability and accuracy in continual vision-language learning?
• RQ5 To what extent does the proposed P9D benchmark reflect real-world data distribution and long-tailed characteristics?

Key findings

• CTP achieves 50.53% TR@1 and 70.63% Rm on the P9D benchmark, outperforming the strongest baseline by 5.76% when combined with experience replay.
• The ablation study confirms that both compatible momentum contrast and topology preservation are essential, with topology preservation contributing 5.64% improvement on Rm.
• The method improves mAP@1 by 0.57% and Rm by 1.88% when both components are used together, demonstrating synergistic benefits.
• The momentum queue improves performance slightly, indicating its role in smoothing training and reducing forgetting.
• Suppressing same-modal maximum similarity is critical—omitting this component causes performance degradation, highlighting its importance in maintaining accurate similarity learning.
• CTP achieves superior results without storing old data, proving its memory efficiency and scalability compared to memory-buffer methods.