[Paper Review] Learning-to-Rank Meets Language: Boosting Language-Driven Ordering Alignment for Ordinal Classification

Rui Wang, Peipei Li+4 more|arXiv (Cornell University)|Jun 24, 2023

Face recognition and analysis4 citations

TL;DR

This paper proposes L2RCLIP, a vision-language model that enhances ordinal classification by leveraging language priors through a novel prompt tuning method, RankFormer, and a cross-modal ordinal pairwise loss. By improving both semantic and ordering alignment in CLIP's feature space, L2RCLIP achieves state-of-the-art performance on facial age estimation, aesthetic assessment, and historical image dating, with gains of up to 3.02% accuracy and 0.35 MAE reduction over prior methods.

ABSTRACT

We present a novel language-driven ordering alignment method for ordinal classification. The labels in ordinal classification contain additional ordering relations, making them prone to overfitting when relying solely on training data. Recent developments in pre-trained vision-language models inspire us to leverage the rich ordinal priors in human language by converting the original task into a visionlanguage alignment task. Consequently, we propose L2RCLIP, which fully utilizes the language priors from two perspectives. First, we introduce a complementary prompt tuning technique called RankFormer, designed to enhance the ordering relation of original rank prompts. It employs token-level attention with residual-style prompt blending in the word embedding space. Second, to further incorporate language priors, we revisit the approximate bound optimization of vanilla cross-entropy loss and restructure it within the cross-modal embedding space. Consequently, we propose a cross-modal ordinal pairwise loss to refine the CLIP feature space, where texts and images maintain both semantic alignment and ordering alignment. Extensive experiments on three ordinal classification tasks, including facial age estimation, historical color image (HCI) classification, and aesthetic assessment demonstrate its promising performance. The code is available at https://github.com/raywang335/L2RCLIP.

Motivation & Objective

• To address the limitation of existing ordinal classification methods that fail to effectively model label ordering, especially under data scarcity or distribution shift.
• To leverage rich ordinal priors in human language to improve generalization and reduce overfitting in vision-language models.
• To simultaneously ensure semantic alignment and ordering alignment in CLIP’s cross-modal embedding space for better zero-shot and few-shot performance.
• To develop a method that outperforms existing vision-language models in ordinal classification tasks by explicitly modeling relative rank relationships using language.

Proposed method

• Introduces RankFormer, a complementary prompt tuning technique that applies token-level attention and residual-style blending to enhance the ordering relation of original rank prompts in the word embedding space.
• Reformulates the approximate bound of cross-entropy loss in the cross-modal embedding space by incorporating language priors to improve generalization.
• Proposes a cross-modal ordinal pairwise loss that enforces both semantic and ordering alignment between image and text features in CLIP’s embedding space.
• Uses interpolated rank templates to maintain relative order while preserving semantic coherence, avoiding the trade-off seen in prior methods.
• Employs a dual optimization strategy: one for semantic alignment via contrastive loss, and another for ordering alignment via pairwise ranking constraints.
• Fine-tunes CLIP using both the enhanced rank prompts and the novel pairwise loss to refine the feature space for ordinal classification.

Experimental results

Research questions

• RQ1 Can language priors be effectively leveraged to improve ordering alignment in vision-language models for ordinal classification?
• RQ2 How can prompt tuning be enhanced to explicitly model relative rank relationships without sacrificing semantic alignment?
• RQ3 Does incorporating ordinal priors from language into the CLIP feature space lead to better performance on zero-shot and few-shot ordinal classification?
• RQ4 Can a unified loss function simultaneously optimize for semantic and ordering alignment in cross-modal embedding space?
• RQ5 How does the proposed method compare to existing approaches like CoOp and OrdinalCLIP in terms of robustness and generalization under data shift?

Key findings

• L2RCLIP achieves state-of-the-art performance on facial age estimation, reducing MAE by 0.35 compared to the best baseline without language priors.
• On the CrowdBeauty dataset, L2RCLIP improves accuracy by 3.02% and reduces MAE by 0.35 compared to OrdinalCLIP, demonstrating superior performance in aesthetic assessment.
• In historical image dating, L2RCLIP achieves an MAE of 0.43 and accuracy of 67.22%, outperforming previous SOTA methods by a large margin with lower standard deviation.
• The ablation study confirms that both RankFormer and the cross-modal ordinal pairwise loss are essential components, with each contributing significantly to performance gains.
• L2RCLIP shows strong robustness in few-shot and data distribution shift scenarios, indicating effective generalization through language priors.
• The method successfully maintains both semantic and ordering alignment, avoiding the trade-off observed in prior approaches like OrdinalCLIP.