[Paper Review] Hardness Sampling for Self-Training Based Transductive Zero-Shot Learning

Bo Liu, Qiulei Dong+1 more|arXiv (Cornell University)|Jun 1, 2021

Domain Adaptation and Few-Shot Learning50 references5 citations

TL;DR

This paper proposes STHS, a self-training framework with hardness sampling for transductive zero-shot learning (T-ZSL), which improves performance by iteratively selecting diverse, hard-to-classify unseen-class samples based on prediction frequency and prior-normalized frequency. The method significantly outperforms state-of-the-art approaches on three benchmarks and establishes a competitive baseline under a stricter T-GZSL evaluation setting.

ABSTRACT

Transductive zero-shot learning (T-ZSL) which could alleviate the domain shift problem in existing ZSL works, has received much attention recently. However, an open problem in T-ZSL: how to effectively make use of unseen-class samples for training, still remains. Addressing this problem, we first empirically analyze the roles of unseen-class samples with different degrees of hardness in the training process based on the uneven prediction phenomenon found in many ZSL methods, resulting in three observations. Then, we propose two hardness sampling approaches for selecting a subset of diverse and hard samples from a given unseen-class dataset according to these observations. The first one identifies the samples based on the class-level frequency of the model predictions while the second enhances the former by normalizing the class frequency via an approximate class prior estimated by an explored prior estimation algorithm. Finally, we design a new Self-Training framework with Hardness Sampling for T-ZSL, called STHS, where an arbitrary inductive ZSL method could be seamlessly embedded and it is iteratively trained with unseen-class samples selected by the hardness sampling approach. We introduce two typical ZSL methods into the STHS framework and extensive experiments demonstrate that the derived T-ZSL methods outperform many state-of-the-art methods on three public benchmarks. Besides, we note that the unseen-class dataset is separately used for training in some existing transductive generalized ZSL (T-GZSL) methods, which is not strict for a GZSL task. Hence, we suggest a more strict T-GZSL data setting and establish a competitive baseline on this setting by introducing the proposed STHS framework to T-GZSL.

Motivation & Objective

• To address the open problem in transductive zero-shot learning (T-ZSL): how to effectively utilize unlabeled unseen-class samples for training.
• To investigate the roles of unseen-class samples with varying degrees of classification hardness in improving ZSL model performance.
• To propose a more rigorous data setting for transductive generalized ZSL (T-GZSL), where seen and unseen classes are indistinguishable during training and testing.
• To design a flexible self-training framework, STHS, that seamlessly integrates any inductive ZSL method with hardness sampling for iterative refinement.
• To establish a competitive baseline for T-GZSL under the new strict evaluation protocol using the proposed STHS framework.

Proposed method

• The method identifies hard samples by analyzing class-level prediction frequency from an initial ZSL model on unseen-class data.
• It introduces a prior estimation algorithm to approximate the true class prior and normalizes prediction frequencies using this estimate to improve sample selection.
• Two hardness sampling strategies are proposed: CFBS (Class Frequency-Based Sampling) and PN-CFBS (Prior-Normalized CFBS), which prioritize samples from under-predicted, hard classes.
• The STHS framework integrates any inductive ZSL method and performs iterative self-training using the selected hard samples from the unseen-class set.
• The framework is generalized to T-GZSL by training on a combined seen-unseen test set, where the model must predict both classes simultaneously without prior knowledge of class identity.
• A WGAN-based model is used as the base model in the generalized setting, and the resulting method is named STHS-GZSL.

Experimental results

Research questions

• RQ1 How do samples from different unseen classes (in terms of classification difficulty) contribute to improving ZSL model performance during transductive training?
• RQ2 Can prediction frequency and prior-normalized frequency of model predictions be used to effectively identify hard and diverse unseen-class samples for self-training?
• RQ3 Does the performance of self-training in T-ZSL depend on the quality of prior estimation, especially in long-tailed or high-class-count datasets?
• RQ4 How does the performance of T-ZSL methods degrade when the test set is not pre-separated into seen and unseen classes, and can a robust baseline be established under this stricter setting?
• RQ5 Can the proposed STHS framework be generalized to T-GZSL and achieve superior performance compared to existing methods under the new evaluation protocol?

Key findings

• The STHS framework with PN-CFBS sampling achieved 91.4% accuracy on AWA2, significantly outperforming the baseline WGAN (87.5%) and ST-WGAN-RS (75.1%) under the strict T-GZSL setting.
• On the AWA2 dataset, PN-CFBS improved performance over CFBS by 2.5 percentage points, demonstrating the benefit of prior normalization in long-tailed settings.
• The STHS framework with CFBS achieved 88.9% accuracy on AWA2 and 89.8% on WGAN, surpassing multiple state-of-the-art methods across all three benchmarks.
• In the strict T-GZSL setting, STHS-GZSL achieved 85.4% accuracy on AWA2, outperforming ST-WGAN-RS (75.1%) and showing robustness despite the increased difficulty of the evaluation protocol.
• The performance of PN-CFBS was slightly worse than CFBS on CUB and SUN, indicating that prior estimation is challenging in datasets with many classes and few samples per class.
• The results confirm that hard samples from under-predicted classes are more effective for self-training, and that diversity in hard classes further improves model generalization.