[Paper Review] Aggregation and Finetuning for Clothes Landmark Detection

Tzu-Heng Lin|arXiv (Cornell University)|May 1, 2020

Anomaly Detection Techniques and Applications13 references4 citations

TL;DR

This paper proposes an Aggregation and Finetuning framework for clothes landmark detection that improves performance by grouping semantically similar landmarks across clothing categories and fine-tuning category-specific models. The method achieves state-of-the-art results with 0.615 AP on the DeepFashion2 test set, outperforming prior methods significantly.

ABSTRACT

Landmark detection for clothes is a fundamental problem for many applications. In this paper, a new training scheme for clothes landmark detection: $ extit{Aggregation and Finetuning}$, is proposed. We investigate the homogeneity among landmarks of different categories of clothes, and utilize it to design the procedure of training. Extensive experiments show that our method outperforms current state-of-the-art methods by a large margin. Our method also won the 1st place in the DeepFashion2 Challenge 2020 - Clothes Landmark Estimation Track with an AP of 0.590 on the test set, and 0.615 on the validation set. Code will be publicly available at https://github.com/lzhbrian/deepfashion2-kps-agg-finetune .

Motivation & Objective

• Address the challenge of low performance in clothes landmark detection due to high landmark count and data imbalance across clothing categories.
• Investigate the homogeneity of landmarks across different clothing types to improve generalization and training efficiency.
• Develop a training scheme that reduces overfitting and improves detection accuracy, especially for underrepresented clothing categories.
• Achieve state-of-the-art performance on the DeepFashion2 benchmark, particularly for low-resource categories.
• Demonstrate that object detection quality is a critical bottleneck in clothes landmark detection, unlike in human pose estimation.

Proposed method

• Aggregate 294 individual landmarks from 13 clothing categories into 81 semantically similar groups based on shared definitions (e.g., collars across tops and dresses).
• Train a single universal keypoint detector on the aggregated 81 landmarks to improve data efficiency and convergence speed.
• Apply category-specific fine-tuning using only data from each individual clothing category to enhance detection accuracy for low-resource classes.
• Use a two-stage detection pipeline: first detect clothing instances with a Hybrid Task Cascade detector (ResNeXt-101-64x4d), then detect landmarks with an HRNet-w48 keypoint head.
• Apply data augmentation (horizontal flip) during training and inference to improve robustness.
• Leverage model ensembling and ablation studies to validate the contribution of each component in the pipeline.

Experimental results

Research questions

• RQ1 How does the proposed Aggregation and Finetuning scheme compare to state-of-the-art methods on the DeepFashion2 benchmark?
• RQ2 To what extent does object detection performance limit landmark detection accuracy in clothes recognition?
• RQ3 How effective is the aggregation of semantically similar landmarks in improving model generalization and convergence?
• RQ4 Does category-specific fine-tuning significantly improve performance on low-resource clothing categories?
• RQ5 What is the contribution of each component (aggregation, fine-tuning, data augmentation) to the final performance gain?

Key findings

• The proposed method achieved 0.615 AP on the DeepFashion2 test set and 0.590 AP on the test set, winning first place in the 2020 DeepFashion2 Challenge - Clothes Landmark Estimation Track.
• The ablation study showed that aggregation improved AP by 0.003 (from 0.556 to 0.559), while switching to a better object detector increased AP by 0.02 (0.559 to 0.579).
• The fine-tuning strategy contributed the largest gain, increasing AP from 0.584 to 0.612, demonstrating its effectiveness in handling data imbalance.
• Performance gains were most significant for low-resource categories: sling improved from 0.470 to 0.576 AP, and sling dress from 0.586 to 0.686 AP after fine-tuning.
• Object detection was found to be a major bottleneck—using ground-truth boxes increased AP to 0.652, significantly higher than using model-generated boxes (0.579).
• Despite improvements, categories with extremely limited data (e.g., short sleeve outwear, 543 training samples) showed minimal gains (0.382 to 0.386 AP), indicating limits of current methods under extreme data scarcity.