[Paper Review] MorphMLP: An Efficient MLP-Like Backbone for Spatial-Temporal Representation Learning

David Junhao Zhang, Kunchang Li+6 more|arXiv (Cornell University)|Nov 24, 2021

Human Pose and Action Recognition5 citations

TL;DR

MorphMLP proposes a self-attention-free, MLP-like backbone for video representation learning that uses two specialized fully connected layers—MorphFC_s for progressive spatial modeling and MorphFC_t for temporal dependency learning—achieving state-of-the-art accuracy with significantly reduced computation, outperforming models like VideoSwin and MViT under comparable FLOP budgets.

ABSTRACT

Recently, MLP-Like networks have been revived for image recognition. However, whether it is possible to build a generic MLP-Like architecture on video domain has not been explored, due to complex spatial-temporal modeling with large computation burden. To fill this gap, we present an efficient self-attention free backbone, namely MorphMLP, which flexibly leverages the concise Fully-Connected (FC) layer for video representation learning. Specifically, a MorphMLP block consists of two key layers in sequence, i.e., MorphFC_s and MorphFC_t, for spatial and temporal modeling respectively. MorphFC_s can effectively capture core semantics in each frame, by progressive token interaction along both height and width dimensions. Alternatively, MorphFC_t can adaptively learn long-term dependency over frames, by temporal token aggregation on each spatial location. With such multi-dimension and multi-scale factorization, our MorphMLP block can achieve a great accuracy-computation balance. Finally, we evaluate our MorphMLP on a number of popular video benchmarks. Compared with the recent state-of-the-art models, MorphMLP significantly reduces computation but with better accuracy, e.g., MorphMLP-S only uses 50% GFLOPs of VideoSwin-T but achieves 0.9% top-1 improvement on Kinetics400, under ImageNet1K pretraining. MorphMLP-B only uses 43% GFLOPs of MViT-B but achieves 2.4% top-1 improvement on SSV2, even though MorphMLP-B is pretrained on ImageNet1K while MViT-B is pretrained on Kinetics400. Moreover, our method adapted to the image domain outperforms previous SOTA MLP-Like architectures. Code is available at https://github.com/MTLab/MorphMLP.

Motivation & Objective

• To explore whether MLP-like architectures can be effectively extended to video representation learning, given the lack of prior work in this domain.
• To address the challenge of capturing hierarchical spatial semantics in video frames without relying on self-attention mechanisms.
• To design an efficient temporal modeling mechanism that captures long-range dependencies across frames with minimal computational cost.
• To achieve a superior accuracy-computation trade-off in video classification compared to existing vision transformers and CNNs.
• To validate the generalization of the proposed architecture to image recognition and semantic segmentation tasks.

Proposed method

• MorphMLP employs a novel block architecture composed of two sequential components: MorphFC_s for spatial modeling and MorphFC_t for temporal modeling.
• MorphFC_s progressively expands receptive fields along height and width dimensions through hierarchical token interaction, enabling multi-scale spatial feature learning.
• MorphFC_t aggregates features across all frames at each spatial location into a temporal chunk, which is then processed via fully connected layers to model long-range temporal dependencies.
• The spatial and temporal modules are stacked in sequence to build a deep MorphMLP backbone, with a skip residual connection applied after the spatial module to stabilize training.
• The method avoids self-attention by replacing it with factorized, progressive, and adaptive fully connected operations that reduce FLOPs while preserving representational capacity.
• The architecture is adapted to image tasks by removing the temporal dimension, demonstrating strong performance on ImageNet-1K and ADE20K.

Experimental results

Research questions

• RQ1 Can a self-attention-free, MLP-like architecture effectively learn spatial-temporal representations in video data?
• RQ2 How can hierarchical and progressive token interaction in spatial modeling improve feature learning compared to global MLP or convolutional operations?
• RQ3 Can fully connected layers effectively replace self-attention for modeling long-range temporal dependencies in videos with lower computational cost?
• RQ4 What is the optimal architectural configuration (e.g., order of operations, residual connections) for balancing accuracy and efficiency in video MorphMLP?
• RQ5 Does the proposed MorphMLP architecture generalize well to image recognition and segmentation tasks when the temporal dimension is removed?

Key findings

• MorphMLP-S achieves 0.9% higher top-1 accuracy than VideoSwin-T on Kinetics400 while using only 50% of its GFLOPs under ImageNet1K pretraining.
• MorphMLP-B achieves 2.4% higher top-1 accuracy than MViT-B on SSV2, despite using only 43% of the GFLOPs and being pretrained on ImageNet1K instead of Kinetics400.
• MorphFC_s outperforms standard 3×3 and 7×7 convolutions by 2.3–2.4% top-1 accuracy on ImageNet-1K, demonstrating its superiority in capturing long-range spatial context.
• MorphFC_t significantly outperforms 3×1×1 and 5×1×1 temporal convolutions on SSV1, achieving 50.6% top-1 accuracy compared to 47.9% and 48.6% respectively.
• The sequential application of MorphFC_s followed by MorphFC_t with a skip residual connection yields the best performance, outperforming parallel or alternative order configurations.
• The full MorphMLP model achieves faster training speed than SOTA models like SlowFast and Timesformer, with comparable or better accuracy and lower training cost in terms of GPU hours.