[Paper Review] OrdinalCLIP: Learning Rank Prompts for Language-Guided Ordinal Regression

Wanhua Li, Xiaoke Huang+5 more|arXiv (Cornell University)|Jun 6, 2022

Domain Adaptation and Few-Shot Learning10 citations

TL;DR

OrdinalCLIP introduces a language-powered paradigm for ordinal regression by learning rank-specific language prototypes from CLIP's pre-trained text encoder, using differentiable prompts with learnable context tokens and continuous rank embeddings to preserve ordinal structure. It achieves state-of-the-art performance on age estimation, historical image dating, and image aesthetics, with improvements in few-shot and distribution shift settings, while enabling zero inference cost via distilled language prototypes.

ABSTRACT

This paper presents a language-powered paradigm for ordinal regression. Existing methods usually treat each rank as a category and employ a set of weights to learn these concepts. These methods are easy to overfit and usually attain unsatisfactory performance as the learned concepts are mainly derived from the training set. Recent large pre-trained vision-language models like CLIP have shown impressive performance on various visual tasks. In this paper, we propose to learn the rank concepts from the rich semantic CLIP latent space. Specifically, we reformulate this task as an image-language matching problem with a contrastive objective, which regards labels as text and obtains a language prototype from a text encoder for each rank. While prompt engineering for CLIP is extremely time-consuming, we propose OrdinalCLIP, a differentiable prompting method for adapting CLIP for ordinal regression. OrdinalCLIP consists of learnable context tokens and learnable rank embeddings; The learnable rank embeddings are constructed by explicitly modeling numerical continuity, resulting in well-ordered, compact language prototypes in the CLIP space. Once learned, we can only save the language prototypes and discard the huge language model, resulting in zero additional computational overhead compared with the linear head counterpart. Experimental results show that our paradigm achieves competitive performance in general ordinal regression tasks, and gains improvements in few-shot and distribution shift settings for age estimation. The code is available at https://github.com/xk-huang/OrdinalCLIP.

Motivation & Objective

• To address overfitting and poor generalization in existing ordinal regression methods that learn rank concepts solely from training data.
• To leverage rich semantic priors from pre-trained vision-language models like CLIP to improve model generalization.
• To explicitly model the continuous ordinal nature of rank labels in the language latent space.
• To develop a differentiable prompting mechanism that learns rank-specific prompts without relying on manual prompt engineering.
• To enable zero computational overhead at inference by distilling language prototypes from CLIP, discarding the full language model.

Proposed method

• Reformulate ordinal regression as an image-text matching task using a contrastive objective, where image features are matched against learned language prototypes.
• Use CLIP's pre-trained text encoder to extract fixed language prototypes for each rank, derived from natural language descriptions such as 'this person is 23 years old'.
• Introduce learnable context tokens shared across all ranks to form dynamic prompts, enabling end-to-end training.
• Learn rank embeddings via interpolation between base features to explicitly model numerical continuity and preserve order in the latent space.
• Train the image encoder and prompt parameters end-to-end using contrastive loss to align image features with corresponding language prototypes.
• Discard the full CLIP model at inference, retaining only the distilled language prototypes for zero additional computational cost.

Experimental results

Research questions

• RQ1 Can language priors from pre-trained vision-language models improve generalization in ordinal regression tasks?
• RQ2 How can the continuous ordinal structure of rank labels be explicitly modeled in the latent space of a pre-trained language model?
• RQ3 Can differentiable prompting reduce the need for manual prompt engineering while improving performance on ordinal regression?
• RQ4 Does learning rank prototypes from language space mitigate overfitting compared to purely data-driven methods?
• RQ5 Can the method generalize to few-shot and distribution shift settings, particularly in age estimation?

Key findings

• OrdinalCLIP achieves a new state-of-the-art mean absolute error (MAE) of 0.280 and accuracy of 72.58% on the age estimation benchmark, outperforming previous SOTA methods.
• On the historical image dating dataset, OrdinalCLIP achieves 56.44% accuracy with a MAE of 0.67, significantly outperforming CoOp (51.90%) and zero-shot CLIP (26.08%).
• The method improves performance by 4.45% in accuracy and 0.11 in MAE over the previous best method on the historical image dating dataset.
• OrdinalCLIP shows consistent improvements across all nominal categories in age estimation, with lower standard deviation than previous methods.
• In few-shot and distribution shift settings for age estimation, OrdinalCLIP demonstrates stronger robustness and generalization compared to baseline and CoOp.
• The ablation study confirms that learnable rank embeddings with continuous interpolation significantly improve performance over non-continuous alternatives.