[Paper Review] NeuroGen: activation optimized image synthesis for discovery neuroscience

Zijin Gu, Keith Jamison+7 more|ArXiv.org|May 15, 2021

Cell Image Analysis Techniques4 citations

TL;DR

NeuroGen proposes a novel framework that combines an fMRI-trained neural encoding model with a conditional deep generative network (BigGAN-deep) to synthesize high-fidelity images predicted to elicit specific patterns of macro-scale brain activation. By optimizing noise vectors and class codes to maximize or minimize activation in target visual regions, NeuroGen enables precise, data-efficient discovery of stimulus-response relationships in human vision, outperforming natural images in detecting individual and regional brain response differences.

ABSTRACT

Functional MRI (fMRI) is a powerful technique that has allowed us to characterize visual cortex responses to stimuli, yet such experiments are by nature constructed based on a priori hypotheses, limited to the set of images presented to the individual while they are in the scanner, are subject to noise in the observed brain responses, and may vary widely across individuals. In this work, we propose a novel computational strategy, which we call NeuroGen, to overcome these limitations and develop a powerful tool for human vision neuroscience discovery. NeuroGen combines an fMRI-trained neural encoding model of human vision with a deep generative network to synthesize images predicted to achieve a target pattern of macro-scale brain activation. We demonstrate that the reduction of noise that the encoding model provides, coupled with the generative network's ability to produce images of high fidelity, results in a robust discovery architecture for visual neuroscience. By using only a small number of synthetic images created by NeuroGen, we demonstrate that we can detect and amplify differences in regional and individual human brain response patterns to visual stimuli. We then verify that these discoveries are reflected in the several thousand observed image responses measured with fMRI. We further demonstrate that NeuroGen can create synthetic images predicted to achieve regional response patterns not achievable by the best-matching natural images. The NeuroGen framework extends the utility of brain encoding models and opens up a new avenue for exploring, and possibly precisely controlling, the human visual system.

Motivation & Objective

• To overcome limitations of traditional fMRI experiments, which rely on a priori hypotheses, are constrained by limited stimulus sets, and are noisy.
• To develop a computational framework that generates synthetic images tailored to elicit specific regional or individual brain activation patterns.
• To enable discovery of visual response differences across individuals and brain regions using only a small number of optimized synthetic images.
• To demonstrate that NeuroGen can generate images achieving activation patterns unattainable with the best-matching natural images.
• To validate that NeuroGen’s synthetic image discoveries are reflected in thousands of observed fMRI responses.

Proposed method

• Trained a ridge regression-based neural encoding model on fMRI data from 8 individuals to predict regional brain activation from image features.
• Utilized a pretrained BigGAN-deep generator with conditional class vectors and noise vectors to generate diverse, high-fidelity synthetic images.
• Performed two-stage optimization: first identifying 10 optimal image classes via random noise sampling and activation prediction, then optimizing noise vectors per class to maximize target region activation.
• Formulated the optimization as a regularized maximization problem: $\hat{z}_{ij}(c_i) = \arg\max_{z_{ij}} (\hat{r}_t(G(c_i,z_{ij})) - \lambda \|z_{ij}\|)$ with $\lambda = 0.001$.
• For multi-region goals, the loss function was extended to jointly maximize or minimize activation in paired regions, with equal weighting.
• Selected the top 10 synthetic images by choosing the best-performing image from each of the 10 optimal classes for downstream analysis.

Experimental results

Research questions

• RQ1 Can synthetic images generated via NeuroGen detect and amplify differences in regional and individual human brain response patterns more effectively than natural images?
• RQ2 Can NeuroGen generate images that elicit brain activation patterns not achievable by the best-matching natural images in the ImageNet dataset?
• RQ3 To what extent do NeuroGen’s synthetic image discoveries generalize across individuals and brain regions, as validated by observed fMRI responses?
• RQ4 How does the integration of an fMRI-trained encoding model with a deep generative network improve the robustness and precision of visual neuroscience discovery?
• RQ5 Can NeuroGen be used to precisely control or probe the functional organization of the human visual cortex through targeted image synthesis?

Key findings

• NeuroGen successfully synthesized 100 images across 10 optimal classes that maximized predicted activation in target visual regions, with the top 10 images selected for analysis.
• The use of only 10 synthetic images enabled detection and amplification of differences in regional and individual brain response patterns, outperforming natural image sets.
• NeuroGen generated images predicted to elicit activation patterns in visual regions that were not achievable by any of the best-matching natural images from ImageNet.
• The predicted activation patterns from NeuroGen’s synthetic images were validated against thousands of observed fMRI responses, confirming the reliability and generalizability of the discoveries.
• The framework demonstrated robustness to noise through the encoding model’s ability to smooth measurement variability, enabling precise discovery with minimal data.
• The truncation parameter in BigGAN-deep was set to 0.4 to balance image fidelity and diversity, resulting in high-quality synthetic stimuli suitable for neuroscience experiments.