[Paper Review] Robust Domain Generalization for Multi-modal Object Recognition

Yuxin Qiao, Keqin Li+5 more|arXiv (Cornell University)|Aug 11, 2024

Advanced Image and Video Retrieval Techniques4 citations

TL;DR

This paper proposes Mixup-CLIPood, a novel domain generalization method for multi-modal object recognition that addresses limitations in CLIPood by inferring the true training loss, evaluating on larger vision-language backbones, and introducing a class-aware mix-up loss. The method achieves superior zero-shot generalization across datasets like PACS, VLCS, and Office-Home, demonstrating robust performance improvements over existing baselines.

ABSTRACT

In multi-label classification, machine learning encounters the challenge of domain generalization when handling tasks with distributions differing from the training data. Existing approaches primarily focus on vision object recognition and neglect the integration of natural language. Recent advancements in vision-language pre-training leverage supervision from extensive visual-language pairs, enabling learning across diverse domains and enhancing recognition in multi-modal scenarios. However, these approaches face limitations in loss function utilization, generality across backbones, and class-aware visual fusion. This paper proposes solutions to these limitations by inferring the actual loss, broadening evaluations to larger vision-language backbones, and introducing Mixup-CLIPood, which incorporates a novel mix-up loss for enhanced class-aware visual fusion. Our method demonstrates superior performance in domain generalization across multiple datasets.

Motivation & Objective

• Address the discrepancy between the reported loss and the actual loss used in the CLIPood implementation.
• Expand evaluation beyond a single backbone by testing on larger vision-language models (ViT-B/32 and ViT-L/14).
• Improve domain generalization by incorporating class-aware visual feature fusion through a novel mix-up loss.
• Demonstrate the effectiveness of the proposed method across diverse multi-modal benchmarks under zero-shot generalization settings.

Proposed method

• Infer the actual training loss used in the official CLIPood implementation by analyzing code, correcting the discrepancy with the originally reported loss.
• Extend evaluation to two larger vision-language backbones: ViT-B/32 and ViT-L/14, to assess generalization across model scales.
• Propose a novel mix-up loss that enables class-aware visual feature fusion during training, enhancing model robustness to domain shifts.
• Apply the mix-up loss during fine-tuning on source domains, generating mixed samples that encourage the model to learn more invariant and generalized representations.
• Adopt standard domain generalization protocols: one domain as target (unseen during training), three domains as source for adaptation.
• Use standard evaluation metrics: top-1 accuracy on each target domain, with results reported across multiple datasets and backbones.

Experimental results

Research questions

• RQ1 What is the actual loss function used in the official CLIPood implementation, and how does it differ from the one described in the paper?
• RQ2 How does the performance of CLIPood generalize across larger vision-language backbones such as ViT-B/32 and ViT-L/14?
• RQ3 To what extent does incorporating class-aware visual fusion via a mix-up loss improve zero-shot domain generalization in multi-modal recognition?
• RQ4 Can the proposed Mixup-CLIPood method achieve consistent performance gains across diverse multi-modal datasets like PACS, VLCS, and Office-Home?

Key findings

• The actual loss used in the CLIPood implementation differs from the one reported in the paper, and this discrepancy is corrected through code analysis.
• Mixup-CLIPood achieves consistent performance gains across all evaluated backbones (ViT-B/16, ViT-B/32, ViT-L/14), demonstrating robust generalization.
• On the VLCS dataset with ViT-L/14, Mixup-CLIPood achieves a starting test accuracy of approximately 89%, surpassing 90% after training, outperforming ViT-B/16’s 81% after 10 epochs.
• The mix-up loss significantly improves generalization, with substantial accuracy gains over CLIPood across most tasks on PACS, VLCS, and Office-Home.
• The method shows stable training dynamics, with test accuracy increasing over epochs and converging to high performance levels.
• The proposed method sets a new benchmark in multi-modal domain generalization by effectively combining class-aware fusion and broader backbone evaluation.