[Paper Review] AdaLTM: Adaptive Layer-wise Task Vector Merging for Categorical Speech Emotion Recognition with ASR Knowledge Integration

Chia-Yu Lee, Huang-Cheng Chou+5 more|arXiv (Cornell University)|Mar 26, 2026

Emotion and Mood Recognition0 citations

TL;DR

AdaLTM integrates in-domain ASR knowledge into SER by adaptive layer-wise merging of task vectors into a frozen WavLM backbone, mitigating gradient conflicts and domain mismatches; achieves state-of-the-art-like UAR on MSP-Podcast with efficient adaptation.

ABSTRACT

Integrating Automatic Speech Recognition (ASR) into Speech Emotion Recognition (SER) enhances modeling by providing linguistic context. However, conventional feature fusion faces performance bottlenecks, and multi-task learning often suffers from optimization conflicts. While task vectors and model merging have addressed such conflicts in NLP and CV, their potential in speech tasks remains largely unexplored. In this work, we propose an Adaptive Layer-wise Task Vector Merging (AdaLTM) framework based on WavLM-Large. Instead of joint optimization, we extract task vectors from in-domain ASR and SER models fine-tuned on emotion datasets. These vectors are integrated into a frozen base model using layer-wise learnable coefficients. This strategy enables depth-aware balancing of linguistic and paralinguistic knowledge across transformer layers without gradient interference. Experiments on the MSP-Podcast demonstrate that the proposed approach effectively mitigates conflicts between ASR and SER.

Motivation & Objective

• Motivate integrating ASR knowledge into SER to leverage linguistic context while avoiding optimization conflicts of joint training.
• Propose a weight-space, layer-wise task vector merging framework to fuse ASR and SER information without gradient interference.
• Show that in-domain ASR knowledge provides superior alignment for MSP-Podcast SER compared to out-of-domain sources.

Proposed method

• Extract task vectors from in-domain ASR and SER models fine-tuned on MSP-Podcast (ΔW_ASR, ΔW_SER).
• Merge these vectors into a frozen WavLM-Large backbone using layer-wise learnable coefficients λ^(l) to form θ_merged^(l) = θ_base^(l) + λ_ASR^(l)ΔW_ASR^(l) + λ_SER^(l)ΔW_SER^(l).
• Train only the merging coefficients λ^(l), layer-wise feature weights α_l for combining layer representations, and the SER prediction head to avoid catastrophic forgetting.
• Represent final SER features as a weighted sum H_out = Σ_l α_l H^(l) from all transformer layers; freeze base and task vectors during this phase.

Experimental results

Research questions

• RQ1 Can in-domain ASR knowledge be effectively integrated into SER without gradient-based multi-task optimization?
• RQ2 Does layer-wise merging provide depth-wise advantages over global merging for ASR-enhanced SER?
• RQ3 How does domain consistency (in-domain vs out-of-domain ASR) affect task-vector merging performance in MSP-Podcast?
• RQ4 What is the impact of using dual task vectors (ASR and SER) versus a single vector on SER performance?

Key findings

• Using adaptive layer-wise merging, AdaLTM achieves a Unweighted Average Recall (UAR) of 38.94% on MSP-Podcast with Macro-F1 35.20%.
• In-domain ASR task vectors outperform out-of-domain counterparts, confirming the importance of domain consistency for merging knowledge.
• Dual-vector merging (ASR + SER) yields better performance than single-vector setups, demonstrating a synergistic effect between linguistic and paralinguistic cues.
• Layer-wise merging outperforms static global (λ fixed) or adaptive global (one λ for all layers) strategies, confirming the benefit of depth-aware integration.
• Compared to fully trainable multi-task learning baselines, AdaLTM improves UAR by over 8 percentage points while updating less than 1% of parameters, highlighting computational efficiency.