[Paper Review] Data Augmentation for Text-based Person Retrieval Using Large Language Models

Zheng Li, Lijia Si+3 more|arXiv (Cornell University)|May 20, 2024

Data Quality and Management4 citations

TL;DR

This paper proposes LLM-DA, a plug-and-play data augmentation method for text-based person retrieval that uses large language models to rewrite training texts, enhancing diversity while preserving semantics. A Text Faithfulness Filter removes hallucinated content, and a Balanced Sampling Strategy optimizes training data mix, significantly improving retrieval performance across multiple benchmarks.

ABSTRACT

Text-based Person Retrieval (TPR) aims to retrieve person images that match the description given a text query. The performance improvement of the TPR model relies on high-quality data for supervised training. However, it is difficult to construct a large-scale, high-quality TPR dataset due to expensive annotation and privacy protection. Recently, Large Language Models (LLMs) have approached or even surpassed human performance on many NLP tasks, creating the possibility to expand high-quality TPR datasets. This paper proposes an LLM-based Data Augmentation (LLM-DA) method for TPR. LLM-DA uses LLMs to rewrite the text in the current TPR dataset, achieving high-quality expansion of the dataset concisely and efficiently. These rewritten texts are able to increase the diversity of vocabulary and sentence structure while retaining the original key concepts and semantic information. In order to alleviate the hallucinations of LLMs, LLM-DA introduces a Text Faithfulness Filter (TFF) to filter out unfaithful rewritten text. To balance the contributions of original text and augmented text, a Balanced Sampling Strategy (BSS) is proposed to control the proportion of original text and augmented text used for training. LLM-DA is a plug-and-play method that can be easily integrated into various TPR models. Comprehensive experiments on three TPR benchmarks show that LLM-DA can improve the retrieval performance of current TPR models.

Motivation & Objective

• Address the scarcity of large-scale, high-quality text-based person retrieval (TPR) datasets due to privacy constraints and costly annotation.
• Overcome limitations of traditional text augmentation methods, which often degrade performance by distorting semantics or reducing diversity.
• Leverage the semantic generation power of large language models (LLMs) to create diverse, faithful, and high-quality text augmentations for TPR.
• Mitigate LLM hallucinations through a dedicated Text Faithfulness Filter (TFF) to ensure augmented texts remain semantically aligned with original descriptions.
• Balance the contribution of original and augmented texts during training via a Balanced Sampling Strategy (BSS) to improve model generalization and performance.

Proposed method

• Use a pre-trained LLM (e.g., Vicuna) to rewrite original text descriptions in the TPR dataset, generating semantically equivalent but structurally diverse variants.
• Apply a Text Faithfulness Filter (TFF) that computes semantic similarity between original and rewritten texts using a sentence encoder; retain only rewritten texts with similarity above a threshold α.
• Introduce a Balanced Sampling Strategy (BSS) that controls the proportion of original and augmented texts during training via a sampling threshold β.
• Integrate LLM-DA as a plug-and-play module into existing TPR models without modifying architecture or loss functions.
• Use CLIP (ViT-B/16) as the backbone model for evaluation, training with mixed original and filtered/augmented texts.
• Optimize hyperparameters α (TFF threshold) and β (BSS sampling ratio) via ablation on ICFG-PEDES to balance noise reduction and data diversity.

Experimental results

Research questions

• RQ1 Can LLM-based text rewriting significantly improve retrieval performance in text-based person retrieval compared to traditional augmentation methods?
• RQ2 To what extent can a Text Faithfulness Filter reduce the negative impact of LLM hallucinations in data augmentation for TPR?
• RQ3 How does balancing the ratio of original to augmented texts affect model generalization and performance in TPR?
• RQ4 What optimal hyperparameter settings (α and β) exist for TFF and BSS to maximize retrieval accuracy across benchmarks?
• RQ5 Can LLM-DA be effectively and universally applied as a plug-and-play module across different TPR models without architectural changes?

Key findings

• LLM-DA significantly improves retrieval performance on three benchmarks: Rank-1 accuracy increases by up to 4.2% on ICFG-PEDES and 3.8% on CUHK-PEDES.
• The Text Faithfulness Filter (TFF) reduces noisy data by filtering out semantically inconsistent augmented texts, leading to a 2.1% performance gain on ICFG-PEDES when α = 0.6.
• The Balanced Sampling Strategy (BSS) improves generalization, with optimal performance at β = 0.2, where augmented texts contribute meaningfully without overwhelming the training signal.
• Hyperparameter analysis shows that α > 0.8 reduces performance due to insufficient text diversity, while β > 0.3 degrades performance due to noise and distribution shift.
• Qualitative results confirm that LLM-DA generates more fluent, semantically rich, and structurally diverse texts compared to traditional methods like back-translation or random deletion.
• Combining TFF, BSS, and LLM rewriting yields the best performance, demonstrating complementary benefits across all three components.