[Paper Review] Optimizing deep video representation to match brain activity

Hugo Richard, Ana Lúısa Pinho+2 more|arXiv (Cornell University)|Sep 7, 2018

Face Recognition and Perception8 references4 citations

TL;DR

This study optimizes deep video representations using a Temporal Segment Network (TSN) pretrained on Kinetics to predict fMRI brain activity during natural movie viewing. It reveals a foveal/peripheral dichotomy in visual processing, where foveal regions are best predicted by high-level RGB features, and peripheral regions by low-level optical flow, outperforming PCA through a novel spatial compression scheme.

ABSTRACT

The comparison of observed brain activity with the statistics generated by artificial intelligence systems is useful to probe brain functional organization under ecological conditions. Here we study fMRI activity in ten subjects watching color natural movies and compute deep representations of these movies with an architecture that relies on optical flow and image content. The association of activity in visual areas with the different layers of the deep architecture displays complexity-related contrasts across visual areas and reveals a striking foveal/peripheral dichotomy.

Motivation & Objective

• To develop a method for predicting whole-brain fMRI activity from deep video representations during naturalistic stimulation.
• To investigate how different levels of abstraction in a deep neural network correspond to functional organization in the human visual cortex.
• To introduce an efficient spatial compression scheme for deep video features that improves computational efficiency and predictive performance over PCA.
• To uncover neural correlates of retinotopic organization (foveal vs. peripheral vision) through contrast analysis of deep network layer activations.

Proposed method

• Utilizes a Temporal Segment Network (TSN) based on Inception v3, pretrained on the Kinetics dataset for action recognition, to extract deep video representations from color natural movies.
• Extracts activation statistics from four convolutional layers (L1–L4) in both RGB and optical flow streams to capture hierarchical feature abstraction.
• Applies spatial smoothing and temporal averaging (one representation per 2 seconds) to compress high-dimensional features while preserving spatial structure.
• Employs a linear regression model to predict fMRI voxel activity from deep video features, using subject-specific fMRI data acquired at 1.5mm isotropic resolution.
• Introduces a novel compression method that preserves channel structure and outperforms PCA in predictive performance.
• Computes prediction score contrasts (e.g., L4_rgb - L2_rgb, L1_flow - L4_rgb) to identify functional specializations across visual areas and retinotopic maps.

Experimental results

Research questions

• RQ1 How do different levels of abstraction in a deep neural network correspond to functional specialization across visual cortical areas during naturalistic video viewing?
• RQ2 Can deep video representations derived from an action recognition network predict fMRI activity across the entire human brain with high accuracy?
• RQ3 What is the relationship between retinotopic organization (foveal vs. peripheral vision) and the predictive power of specific deep network layers?
• RQ4 Does a novel spatial compression scheme for deep video features outperform standard methods like PCA in predicting brain activity?
• RQ5 How do the raw image and optical flow streams contribute differently to predicting activity in foveal versus peripheral visual regions?

Key findings

• The study successfully predicts fMRI activity across the whole brain using deep video representations, with prediction scores significantly above chance.
• Occipital and lateral visual areas show distinct patterns: lower-level layers (L1–L2) predict activity in early visual areas, while higher-level layers (L3–L4) predict activity in lateral and dorsal stream regions.
• A striking foveal/peripheral dichotomy is revealed: foveal regions are best predicted by high-level RGB features (L4_rgb), while peripheral regions are best predicted by low-level optical flow features (L1_flow).
• The contrast L1_flow - L4_rgb closely matches the retinotopic map of preferred eccentricity, confirming that this difference specifically encodes foveal vs. peripheral visual processing.
• The proposed compression scheme for deep video features outperforms PCA in predictive performance, enabling faster training without significant loss of representational fidelity.
• Parcellation based on three key contrasts (L2_flow - L4_flow, L2_rgb - L4_rgb, L1_flow - L4_rgb) identifies three major clusters: foveal (deep blue), peripheral (green), and lateral/abstract object encoding (yellow) regions.