[Paper Review] Learning Brain Representation with Hierarchical Visual Embeddings

Jiawen Zheng, Haonan Jia+5 more|arXiv (Cornell University)|Feb 7, 2026

Face Recognition and Perception0 citations

TL;DR

This paper presents a fusion-based brain–vision interface that aligns EEG/MEG signals with hierarchical visual embeddings from multiple pre-trained encoders and uses a pretrained fusion prior to enable robust brain-to-image retrieval and reconstruction without text prompts.

ABSTRACT

Decoding visual representations from brain signals has attracted significant attention in both neuroscience and artificial intelligence. However, the degree to which brain signals truly encode visual information remains unclear. Current visual decoding approaches explore various brain-image alignment strategies, yet most emphasize high-level semantic features while neglecting pixel-level details, thereby limiting our understanding of the human visual system. In this paper, we propose a brain-image alignment strategy that leverages multiple pre-trained visual encoders with distinct inductive biases to capture hierarchical and multi-scale visual representations, while employing a contrastive learning objective to achieve effective alignment between brain signals and visual embeddings. Furthermore, we introduce a Fusion Prior, which learns a stable mapping on large-scale visual data and subsequently matches brain features to this pre-trained prior, thereby enhancing distributional consistency across modalities. Extensive quantitative and qualitative experiments demonstrate that our method achieves a favorable balance between retrieval accuracy and reconstruction fidelity.

Motivation & Objective

• Motivate decoding visual content from brain signals by capturing multiscale visual information from pixel-level to semantic representations.
• Propose a Hierarchical Visual Fusion (HVF) framework that integrates multiple pre-trained encoders to form a fused visual embedding aligned with brain signals.
• Introduce a Fusion Prior that maps fused visual features to diffusion conditioning for stable, text-free image reconstruction.
• Demonstrate improved 200-way zero-shot retrieval and reconstruction quality on THINGS-EEG and THINGS-MEG datasets compared to prior methods.

Proposed method

• Extract high-level semantics using multiple CLIP-based encoders and low-level pixel features using a VAE encoder for each image.
• Fuse encoder outputs with a post-norm residual HVF to produce a shared visual embedding z_f.
• Project brain signals through an MBP network to a brain embedding z_b and align z_b with z_f using a CLIP-style InfoNCE loss.
• Pretrain a Fusion Prior that maps z_f to diffusion conditions z_c using a frozen diffusion backbone (IP-Adapter) for text-free reconstruction.
• During reconstruction, freeze visual encoders and fusion prior; train only the brain side to align with the fusion space; use diffusion generation conditioned on z_c.

(a) Brain-to-image retrieval.

Experimental results

Research questions

• RQ1 Can brain signals (EEG/MEG) be aligned with a fused, hierarchical visual representation that spans both semantic and pixel-level information?
• RQ2 Does incorporating low-level visual features (beyond high-level semantics) improve brain-to-image retrieval and reconstruction fidelity?
• RQ3 Can a learned Fusion Prior enable stable, text-free image reconstruction from brain-driven embeddings using a frozen diffusion model?
• RQ4 Is the proposed HVF+Fusion Prior approach robust across subjects and brain-encoder backbones for zero-shot retrieval?
• RQ5 What is the relative contribution of each visual encoder (semantic vs. pixel-level) to decoding performance?

Key findings

• The proposed method achieves state-of-the-art 200-way zero-shot brain–image retrieval on THINGS-EEG and THINGS-MEG, with substantial gains over prior methods in both intra- and inter-subject settings.
• A fusion of CLIP-based semantic encoders and a VAE-based low-level encoder (VAE) yields the best retrieval performance, outperforming single encoders and simple pairings.
• In reconstruction, the Fusion Prior enables stable, high-fidelity image generation when conditioned by brain-derived embeddings, with quantitative gains in PixCorr and SSIM and strong semantic alignment metrics.
• Ablations show that integrating both high-level semantics and low-level pixel information (RN50+B32+VAE) provides the largest improvements over semantics-only or pixels-only configurations.
• Plugging different EEG/MEG brain backbones into the same fusion-based interface consistently improves retrieval performance, indicating good plug-and-play generalization.

(b) Brain-to-image reconstruction.