[Paper Review] TORE: Token Reduction for Efficient Human Mesh Recovery with Transformer

Zhiyang Dou, Qingxuan Wu+6 more|arXiv (Cornell University)|Nov 19, 2022

Advanced Neural Network Applications4 citations

TL;DR

TORE proposes a token reduction framework for efficient Transformer-based human mesh recovery by leveraging skeletal geometry priors and learnable clustering to prune redundant 2D image features and 3D mesh vertices. By reducing tokens through hierarchical geometry-aware queries and unsupervised feature clustering, TORE achieves SOTA accuracy with up to 78% lower FLOPs and 2.5x higher throughput, demonstrating strong efficiency-accuracy trade-offs on Human3.6M and 3DPW benchmarks.

ABSTRACT

In this paper, we introduce a set of simple yet effective TOken REduction (TORE) strategies for Transformer-based Human Mesh Recovery from monocular images. Current SOTA performance is achieved by Transformer-based structures. However, they suffer from high model complexity and computation cost caused by redundant tokens. We propose token reduction strategies based on two important aspects, i.e., the 3D geometry structure and 2D image feature, where we hierarchically recover the mesh geometry with priors from body structure and conduct token clustering to pass fewer but more discriminative image feature tokens to the Transformer. Our method massively reduces the number of tokens involved in high-complexity interactions in the Transformer. This leads to a significantly reduced computational cost while still achieving competitive or even higher accuracy in shape recovery. Extensive experiments across a wide range of benchmarks validate the superior effectiveness of the proposed method. We further demonstrate the generalizability of our method on hand mesh recovery. Visit our project page at https://frank-zy-dou.github.io/projects/Tore/index.html.

Motivation & Objective

• To address the high computational cost of Transformer-based human mesh recovery (HMR) caused by redundant tokens in input features and mesh vertices.
• To improve efficiency without sacrificing accuracy by exploiting structural priors in human body geometry and discriminative image features.
• To develop a lightweight, end-to-end trainable token reduction mechanism that prunes non-informative tokens while preserving geometric fidelity.
• To demonstrate generalization to hand mesh recovery and robustness under partial occlusion and challenging views.
• To achieve a Pareto-optimal balance between accuracy and inference speed in monocular HMR.

Proposed method

• Introduces a Neural Shape Regressor (NSR) that uses a small set of skeleton-level body tokens to hierarchically regress full mesh vertices, reducing vertex-to-vertex attention complexity.
• Proposes a learnable, unsupervised token pruner that clusters CNN feature patches using clustering-based selection to retain only discriminative image tokens.
• Employs attention matrix decomposition to interpret the hierarchical mesh recovery as a structured attention mechanism, leveraging skeletal priors to guide feature interaction.
• Uses a single-pass, end-to-end training scheme with no additional annotations, enabling joint optimization of pruning and mesh regression.
• Applies a pruning rate of 20% empirically selected to balance accuracy and efficiency, validated across multiple benchmarks.
• Integrates the token reduction module into FastMETRO and METRO-style architectures, enabling plug-and-play efficiency gains.

Experimental results

Research questions

• RQ1 Can reducing redundant tokens in both 2D image features and 3D mesh vertices significantly lower computational cost in Transformer-based HMR without degrading accuracy?
• RQ2 How effective is using skeletal joint priors to replace full mesh vertex queries in attention mechanisms for geometry-aware mesh recovery?
• RQ3 Can an unsupervised, clustering-based token pruner identify semantically meaningful image features that improve generalization and reduce FLOPs?
• RQ4 Does the proposed token reduction strategy maintain or improve performance under partial occlusion and challenging monocular views?
• RQ5 To what extent can the method generalize to other mesh recovery tasks, such as hand mesh reconstruction?

Key findings

• TORE reduces FLOPs by up to 78% compared to baseline methods while maintaining or improving accuracy on Human3.6M and 3DPW benchmarks.
• On Human3.6M, TORE achieves a PAMPJPE of 36.4 mm with 249.2 FPS throughput, outperforming TCFormer and PPT in both speed and accuracy.
• The method achieves a 1.6 GFLOP model size with 870.4 FPS throughput on FastMETRO-Eb0, surpassing PPT in both throughput and PAMPJPE.
• Self-attention visualization confirms that joints and vertices learn non-local, robust interactions, improving performance under occlusion and partial observation.
• The pruning rate of 20% yields optimal trade-off, with PAMPJPE of 43.9 mm, while higher rates (e.g., 75%) degrade performance despite lower FLOPs.
• Qualitative results on 3DPW and Human3.6M show that TORE produces more accurate and stable meshes than METRO, MeshGraphormer, and FastMETRO under diverse poses and views.