[论文解读] Semantic visually-guided acoustic highlighting with large vision-language models

Junhua Huang, Chao Huang+1 more|arXiv (Cornell University)|Jan 12, 2026

Multimodal Machine Learning Applications被引用 0

一句话总结

论文利用冻结的大型Vision-Language模型提取六个视觉语义线索，并系统性评估哪些线索最能引导视觉对齐的音频混辑，发现相机焦点和场景背景对感知混音质量最有帮助，并在较轻模型上达到现有最佳结果。

ABSTRACT

Balancing dialogue, music, and sound effects with accompanying video is crucial for immersive storytelling, yet current audio mixing workflows remain largely manual and labor-intensive. While recent advancements have introduced the visually guided acoustic highlighting task, which implicitly rebalances audio sources using multimodal guidance, it remains unclear which visual aspects are most effective as conditioning signals.We address this gap through a systematic study of whether deep video understanding improves audio remixing. Using textual descriptions as a proxy for visual analysis, we prompt large vision-language models to extract six types of visual-semantic aspects, including object and character appearance, emotion, camera focus, tone, scene background, and inferred sound-related cues. Through extensive experiments, camera focus, tone, and scene background consistently yield the largest improvements in perceptual mix quality over state-of-the-art baselines. Our findings (i) identify which visual-semantic cues most strongly support coherent and visually aligned audio remixing, and (ii) outline a practical path toward automating cinema-grade sound design using lightweight guidance derived from large vision-language models.

研究动机与目标

• Identify which visual-semantic cues most influence visually guided audio remixing.
• Systematically ablate six LVLM-derived cues to determine effectiveness for remixing quality.
• Show that LVLM-derived cues can outperform audio-only and prior multi-modal baselines with lighter models.
• Provide practical guidance for automating cinema-grade sound design using LVLM-derived signals.

提出的方法

• Adopt a VisAH-style end-to-end remixing framework with a text-based conditioning pathway.
• Prompt-driven conditioning to inject six semantic cues: Emotion, Objects, Scene, Tone, Sound Sources, Camera Focus.
• Compare focused vs minimal prompting strategies to evaluate grounding fidelity and hallucination risk.
• Evaluate with a standard MuddyMix-like setup using MAG, ENV, KLD, ΔIB, and W-dis metrics.
• Systematically vary transformer depth (L=0,3,6) to assess the need for deep self-attention.

Fig. 1 : Overview identical to VisAH [ 6 ] except the text feature module (orange) feeding the context encoder.

实验结果

研究问题

• RQ1 Which visual-semantic cues reliably improve perceptual quality and video-audio alignment in automated remixing?
• RQ2 Do focused prompts outperform minimal prompts in grounding visual information for audio remixing?
• RQ3 What is the impact of transformer depth on remixing performance when conditioning on LVLM-derived cues?
• RQ4 Can LVLM-derived guidance achieve state-of-the-art results with fewer parameters and shallower architectures?

主要发现

• Camera Focus provides the strongest gains across MAG, ENV, and KLD relative to the VisAH baseline.
• Scene (Setting & Time) and Sound Sources (Visible) yield small but consistent improvements.
• Objects (Salient) and Tone (Color & Mood) are less beneficial or can hurt certain metrics.
• Focused prompts generally outperform Minimal prompts, with focused cues improving grounding and reducing hallucinations.
• Three-layer transformers (L=3) with focused prompts achieve strong performance, with diminishing returns at greater depth.
• SemMix achieves better results with 18.94M fewer parameters than the prior SOTA while using a shallower architecture.

Fig. 2 : Model performance using focused prompt Camera Focus from layer 0 to layer 6. shaded bands show cross-metric spread (min–max and IQR). The mean curve peaks at $L{=}3$ ; $L{=}5/6$ offer mild W-dis polish.