[Paper Review] Hulk: A Universal Knowledge Translator for Human-Centric Tasks

Yizhou Wang, Yixuan Wu+9 more|arXiv (Cornell University)|Dec 4, 2023

Human Pose and Action Recognition5 citations

TL;DR

Hulk is the first multimodal human-centric generalist model that unifies 2D vision, 3D vision, skeleton-based, and vision-language tasks into a single framework without task-specific fine-tuning. It achieves this by condensing task-specific heads into two general heads—discrete (e.g., language) and continuous (e.g., coordinates)—enabling modality-agnostic knowledge transfer and state-of-the-art performance across 12 benchmarks.

ABSTRACT

Human-centric perception tasks, e.g., pedestrian detection, skeleton-based action recognition, and pose estimation, have wide industrial applications, such as metaverse and sports analysis. There is a recent surge to develop human-centric foundation models that can benefit a broad range of human-centric perception tasks. While many human-centric foundation models have achieved success, they did not explore 3D and vision-language tasks for human-centric and required task-specific finetuning. These limitations restrict their application to more downstream tasks and situations. To tackle these problems, we present Hulk, the first multimodal human-centric generalist model, capable of addressing 2D vision, 3D vision, skeleton-based, and vision-language tasks without task-specific finetuning. The key to achieving this is condensing various task-specific heads into two general heads, one for discrete representations, \emph{e.g.,} languages, and the other for continuous representations, \emph{e.g.,} location coordinates. The outputs of two heads can be further stacked into four distinct input and output modalities. This uniform representation enables Hulk to treat diverse human-centric tasks as modality translation, integrating knowledge across a wide range of tasks. Comprehensive evaluations of Hulk on 12 benchmarks covering 8 human-centric tasks demonstrate the superiority of our proposed method, achieving state-of-the-art performance in 11 benchmarks. The code will be available on https://github.com/OpenGVLab/Hulk.

Motivation & Objective

• Address the lack of unified models for diverse human-centric perception tasks, including 2D/3D vision, skeleton-based, and vision-language tasks.
• Overcome the limitation of existing foundation models that require task-specific fine-tuning for each downstream task.
• Develop a generalist architecture capable of handling heterogeneous inputs and outputs (e.g., images, text, keypoint coordinates, segmentation maps) using shared parameters.
• Enable zero-shot or few-shot transfer across multiple human-centric tasks without retraining or prompt tuning.
• Establish a unified framework that treats all human-centric tasks as modality translation, integrating knowledge across modalities and tasks.

Proposed method

• Design a dual-head architecture: one for discrete representations (e.g., tokens, class labels) and one for continuous representations (e.g., 2D/3D coordinates, embeddings).
• Map all input and output modalities into four distinct combinations of the two general heads, enabling unified processing across tasks.
• Train the model end-to-end on a large-scale, multi-task dataset spanning 1.7M to 30M samples across 8 human-centric tasks.
• Use a unified loss function that supports joint optimization across diverse task types, including segmentation, detection, captioning, and pose estimation.
• Leverage pretraining on diverse human-centric data to improve generalization and reduce domain shift effects.
• Apply parameter-efficient initialization using pre-trained human-centric models (e.g., HAP, PATH) to boost performance without compromising fairness in comparison.

Experimental results

Research questions

• RQ1 Can a single unified model handle diverse human-centric tasks—including 2D/3D vision, skeleton-based, and vision-language tasks—without task-specific fine-tuning?
• RQ2 How effective is a dual-head architecture (discrete and continuous) in unifying heterogeneous inputs and outputs across multimodal human-centric tasks?
• RQ3 To what extent does scaling training data improve performance across diverse human-centric tasks in a generalist setting?
• RQ4 How does initialization from specialized human-centric pretraining models affect the performance of a generalist model like Hulk?
• RQ5 Can a generalist model achieve state-of-the-art performance across multiple benchmarks without task-specific adaptation?

Key findings

• Hulk achieves state-of-the-art performance on 11 out of 12 evaluated benchmarks across 8 human-centric tasks, demonstrating strong generalization and zero-shot transfer capability.
• Scaling training data from 1.7M to 30M samples improves average performance by 7.1 points (from 48.9 to 56.0 mIoU/AP), with notable gains in 2D pose estimation (+2.3% AP) and pedestrian detection (+1.6% AP).
• Incorporating pretraining weights from HAP and PATH improves Hulk’s accuracy, with a +5.0 absolute gain in average performance compared to MAE initialization (52.2 vs. 47.2).
• The model achieves a 26.0 MPVPE reduction in 3D mesh recovery and a 24.5 MPJPE reduction in 3D pose estimation when trained on larger datasets, indicating strong 3D generalization.
• The dual-head design enables effective zero-shot transfer across modalities, with consistent performance gains across tasks, including image captioning (+2.1 B@4) and skeleton-based action recognition (93.8% accuracy).
• Despite domain gaps in extended datasets (e.g., Kinetics-400 for skeleton tasks), performance remains stable, suggesting robustness to distribution shifts when trained on diverse data.