[论文解读] HyFI: Hyperbolic Feature Interpolation for Brain-Vision Alignment

Sangmin Jo, Wooju Jeong+3 more|arXiv (Cornell University)|Mar 24, 2026

Face Recognition and Perception被引用 0

一句话总结

HyFI 引入双曲特征插值以融合语义与感知图像特征，解决模态差距与纠缠，在 THINGS-EEG 与 THINGS-MEG 上实现零样本脑-图像检索的最前沿水平

ABSTRACT

Recent progress in artificial intelligence has encouraged numerous attempts to understand and decode human visual system from brain signals. These prior works typically align neural activity independently with semantic and perceptual features extracted from images using pre-trained vision models. However, they fail to account for two key challenges: (1) the modality gap arising from the natural difference in the information level of representation between brain signals and images, and (2) the fact that semantic and perceptual features are highly entangled within neural activity. To address these issues, we utilize hyperbolic space, which is well-suited for considering differences in the amount of information and has the geometric property that geodesics between two points naturally bend toward the origin, where the representational capacity is lower. Leveraging these properties, we propose a novel framework, Hyperbolic Feature Interpolation (HyFI), which interpolates between semantic and perceptual visual features along hyperbolic geodesics. This enables both the fusion and compression of perceptual and semantic information, effectively reflecting the limited expressiveness of brain signals and the entangled nature of these features. As a result, it facilitates better alignment between brain and visual features. We demonstrate that HyFI achieves state-of-the-art performance in zero-shot brain-to-image retrieval, outperforming prior methods with Top-1 accuracy improvements of up to +17.3% on THINGS-EEG and +9.1% on THINGS-MEG.

研究动机与目标

• Motivate and address modality gap between brain signals and image representations in visual brain decoding.
• Show that semantic and perceptual features are entangled in neural activity and should be fused rather than processed separately.
• Leverage hyperbolic geometry to interpolate between semantic and perceptual features for better brain alignment.
• Demonstrate that hyperbolic feature interpolation compresses representations, reflecting limited brain information capacity.
• Establish the method's general applicability across various visual and brain encoders.

提出的方法

• Embed semantic and perceptual image features in hyperbolic space using the Lorentz (hyperboloid) model and the exponential map.
• Interpolate between semantic and perceptual features along hyperbolic geodesics with a dynamically learned interpolation coefficient.
• Project brain signals into the same hyperbolic space and apply hyperbolic contrastive learning to align with interpolated image embeddings.
• Show that hyperbolic interpolation concentrates representations toward the origin, effectively compressing information.
• Train with a hyperbolic contrastive loss that enforces alignment between interpolated visual representations and brain embeddings.

实验结果

研究问题

• RQ1 Can hyperbolic interpolation of semantic and perceptual features improve alignment between brain signals (EEG/MEG) and visual representations compared to Euclidean approaches?
• RQ2 Does interpolating along hyperbolic geodesics better capture the entangled nature of semantic and perceptual information in neural activity?
• RQ3 How does HyFI perform in zero-shot brain-to-image retrieval across THINGS-EEG and THINGS-MEG benchmarks?
• RQ4 What is the impact of different vision and brain encoders on HyFI’s effectiveness?

主要发现

• HyFI achieves state-of-the-art zero-shot brain-to-image retrieval on THINGS-EEG with Top-1 68.2% and Top-5 91.9%.
• HyFI achieves state-of-the-art zero-shot brain-to-image retrieval on THINGS-MEG with Top-1 35.8% and Top-5 64.6%.
• Ablation shows hyperbolic space plus hyperbolic interpolation outperforms Euclidean (CLIP) space and interpolation in Euclidean space.
• Hyperbolic interpolation concentrates interpolated embeddings toward the origin, reflecting compression and reduced representational capacity.
• HyFI consistently improves performance across combinations of vision and brain encoders, demonstrating broad applicability.