[Paper Review] CaReFlow: Cyclic Adaptive Rectified Flow for Multimodal Fusion

Sijie Mai, Shiqin Han|arXiv (Cornell University)|Feb 22, 2026

Emotion and Mood Recognition0 citations

TL;DR

CaReFlow leverages cyclic adaptive rectified flow to map visual and acoustic modality distributions to language distribution via one-to-many mapping, adaptive relaxed alignment, and cycle-consistency, enabling robust multimodal fusion and state-of-the-art results on multimodal affective computing tasks.

ABSTRACT

Modality gap significantly restricts the effectiveness of multimodal fusion. Previous methods often use techniques such as diffusion models and adversarial learning to reduce the modality gap, but they typically focus on one-to-one alignment without exposing the data points of the source modality to the global distribution information of the target modality. To this end, leveraging the characteristic of rectified flow that can map one distribution to another via a straight trajectory, we extend rectified flow for modality distribution mapping. Specifically, we leverage the `one-to-many mapping' strategy in rectified flow that allows each data point of the source modality to observe the overall target distribution. This also alleviates the issue of insufficient paired data within each sample, enabling a more robust distribution transformation. Moreover, to achieve more accurate distribution mapping and address the ambiguous flow directions in one-to-many mapping, we design `adaptive relaxed alignment', enforcing stricter alignment for modality pairs belonging to the same sample, while applying relaxed mapping for pairs not belonging to the same sample or category. Additionally, to prevent information loss during distribution mapping, we introduce `cyclic rectified flow' to ensure the transferred features can be translated back to the original features, allowing multimodal representations to learn sufficient modality-specific information. After distribution alignment, our approach achieves very competitive results on multiple tasks of multimodal affective computing even with a simple fusion method, and visualizations verify that it can effectively reduce the modality gap.

Motivation & Objective

• Reformulate the modality gap in multimodal affective computing (MAC) as a distribution mapping problem.
• Develop CaReFlow with one-to-many mapping, adaptive relaxed alignment, and cyclic information flow to improve alignment while preserving modality-specific information.
• Enable effective multimodal fusion using a simple fusion module after distribution alignment and demonstrate strong empirical results across MAC benchmarks.
• Show that CaReFlow reduces modality gap visually and quantitatively, and analyze robustness across fusion strategies.

Proposed method

• Adopt rectified flow to map source modality distributions toward the target language distribution with a straight trajectory.
• Employ one-to-many mapping so each source data point observes the broader target distribution during alignment.
• Introduce adaptive relaxed alignment to enforce stricter alignment for pairs from the same sample and relax for different samples/categories with a margin that depends on label similarity.
• Incorporate cyclic rectified flow to ensure forward-transformed features can be mapped back to the original features, preserving modality-specific information.
• Implement drift force V as a time-dependent neural network (MLP with time embeddings) trained via a forward loss that aligns X_m1 to X_m2 and a backward loss to map back, with detach operations to decouple unimodal learning.
• Use two Euler steps (dt=0.5) in practice to map visual/acoustic modalities to language space before fusion, enabling causal information flow from source to target.
• Adaptive relaxed alignment is controlled by a margin η that depends on sample and category relationships, enabling faster and more accurate distribution mapping without extensive iterative rectified-flow training.
• Cyclic information flow includes a backward rectified flow to reconstruct the original modality features from the mapped ones, aiding information preservation for downstream prediction.
• Fusion after distribution alignment is performed with a simple Fusion module (concatenation + MLP), demonstrating that CaReFlow improves performance even with lightweight fusion.

Experimental results

Research questions

• RQ1 Can rectified flow be adapted to multimodal distribution alignment to bridge modality gaps beyond one-to-one mappings?
• RQ2 Does one-to-many mapping with adaptive relaxed alignment improve alignment quality and reduce the need for iterative rectified-flow training?
• RQ3 Is cycle-consistency essential to preserve modality-specific information during distribution transformation for MAC tasks?

Key findings

• CaReFlow achieves competitive and often state-of-the-art results on multimodal sentiment analysis and related MAC tasks across several datasets (e.g., CMU-MOSI, CMU-MOSEI, CH-SIMS-v2, UR-FUNNY, MUStARD).
• On CMU-MOSI, CaReFlow attains Acc7=50.6, Acc2=89.8, F1=89.7, MAE=0.616, Corr=0.858.
• On CMU-MOSEI, CaReFlow attains Acc7=55.7, Acc2=87.9, F1=88.0, MAE=0.504, Corr=0.799.
• On CH-SIMS-v2, CaReFlow attains Acc5=57.9, Acc3=75.9, Acc2=82.9, F1=82.9, MAE=0.277, Corr=0.745.
• Across MHD and MSD tasks (UR-FUNNY and MUStARD), CaReFlow outperforms strong baselines by notable margins, demonstrating generalizability to other MAC tasks.
• Ablation studies show significant performance drops when removing distribution alignment, cyclic information flow, adaptive relaxed alignment, or one-to-many mapping, underscoring their importance.