[Paper Review] Addressing Missing and Noisy Modalities in One Solution: Unified Modality-Quality Framework for Low-quality Multimodal Data

Sijie Mai, Shiqin Han+1 more|arXiv (Cornell University)|Mar 3, 2026

Emotion and Mood Recognition0 citations

TL;DR

UMQ jointly addresses missing and noisy modalities in multimodal data by introducing a quality estimator, a quality enhancer, and a modality-quality-aware mixture-of-experts, achieving state-of-the-art results on multiple MAC datasets.

ABSTRACT

Multimodal data encountered in real-world scenarios are typically of low quality, with noisy modalities and missing modalities being typical forms that severely hinder model performance and robustness. However, prior works often handle noisy and missing modalities separately. In contrast, we jointly address missing and noisy modalities to enhance model robustness in low-quality data scenarios. We regard both noisy and missing modalities as a unified low-quality modality problem, and propose a unified modality-quality (UMQ) framework to enhance low-quality representations for multimodal affective computing. Firstly, we train a quality estimator with explicit supervised signals via a rank-guided training strategy that compares the relative quality of different representations by adding a ranking constraint, avoiding training noise caused by inaccurate absolute quality labels. Then, a quality enhancer for each modality is constructed, which uses the sample-specific information provided by other modalities and the modality-specific information provided by the defined modality baseline representation to enhance the quality of unimodal representations. Finally, we propose a quality-aware mixture-of-experts module with particular routing mechanism to enable multiple modality-quality problems to be addressed more specifically. UMQ consistently outperforms state-of-the-art baselines on multiple datasets under the settings of complete, missing, and noisy modalities.

Motivation & Objective

• Motivate robust multimodal learning under realistic low-quality data with both missing and noisy modalities.
• Propose a unified framework that jointly handles missing and noisy modalities as a low-quality modality problem.
• Develop explicit supervision for modality quality estimation via rank-guided training.
• Design a quality enhancer that leverages sample-specific and modality-specific information to improve unimodal representations.
• Introduce a modality-quality-aware mixture-of-experts that specialized routing for different modality-quality configurations.

Proposed method

• Define a unified modality-quality (UMQ) framework to treat missing and noisy modalities as a single low-quality modality problem.
• Train a quality estimator for each modality using supervised signals augmented with a rank-guided training strategy to learn relative quality without relying on absolute quality labels.
• Develop a quality enhancer that uses sample-specific information from other modalities and modality-specific baseline representations to enrich unimodal features.
• Construct a modality baseline representation and fuse enhanced unimodal representations into a multimodal representation via a fusion network.
• Implement a modality-quality-aware mixture-of-experts (MQ-MoE) with routing constraints to handle 2^{|M|} modality-quality configurations efficiently, plus balance and similarity constraints across experts.

Experimental results

Research questions

• RQ1 How can missing and noisy modalities be jointly modeled to improve robustness in low-quality multimodal data?
• RQ2 Can a quality estimator learned with rank-guided supervision provide reliable modality quality signals for enhancement and routing?
• RQ3 Does leveraging both modality-specific baselines and sample-specific information improve unimodal representation fidelity under degradation?
• RQ4 Can a quality-aware mixture-of-experts effectively specialize handling for diverse modality-quality configurations?
• RQ5 What performance gains are achievable across complete, missing, and noisy modality settings on standard MAC datasets?

Key findings

• UMQ improves over state-of-the-art baselines on multiple multimodal affective computing datasets under complete, missing, and noisy modality settings.
• A quality estimator trained with rank-guided supervision effectively identifies modality quality without relying on absolute quality labels.
• Quality enhancer leveraging both sample-specific and modality-specific information yields higher-quality unimodal representations that retain modality-specific characteristics.
• MQ-MoE with quality-aware routing and constrained expert usage provides specialized handling for each modality-quality configuration, improving robustness and accuracy.
• Ablation studies show that removing the quality estimator or rank-guided training substantially degrades performance, underscoring their critical role.