[Paper Review] Task-Aware Feature Generation for Zero-Shot Compositional Learning

Xin Wang, Fisher Yu+2 more|arXiv (Cornell University)|Jun 11, 2019

Domain Adaptation and Few-Shot Learning58 references9 citations

TL;DR

This paper proposes a task-aware feature generation (TFG) framework for zero-shot compositional learning that synthesizes image features for novel attribute-object compositions using a generator conditioned on task descriptions. By injecting task-conditioned noise layer-by-layer, TFG improves sample efficiency and generalization, achieving state-of-the-art performance—over 2× improvement on generalized ZSCL benchmarks—by generating realistic, discriminative feature distributions that train high-accuracy classifiers without real images.

ABSTRACT

Visual concepts (e.g., red apple, big elephant) are often semantically compositional and each element of the compositions can be reused to construct novel concepts (e.g., red elephant). Compositional feature synthesis, which generates image feature distributions exploiting the semantic compositionality, is a promising approach to sample-efficient model generalization. In this work, we propose a task-aware feature generation (TFG) framework for compositional learning, which generates features of novel visual concepts by transferring knowledge from previously seen concepts. These synthetic features are then used to train a classifier to recognize novel concepts in a zero-shot manner. Our novel TFG design injects task-conditioned noise layer-by-layer, producing task-relevant variation at each level. We find the proposed generator design improves classification accuracy and sample efficiency. Our model establishes a new state of the art on three zero-shot compositional learning (ZSCL) benchmarks, outperforming the previous discriminative models by a large margin. Our model improves the performance of the prior arts by over 2x in the generalized ZSCL setting.

Motivation & Objective

• To address the challenge of recognizing novel visual concepts with no training images, especially under data-scarce conditions.
• To improve model generalization in zero-shot compositional learning by synthesizing realistic image features for unseen attribute-object compositions.
• To enhance sample efficiency and classifier performance through a generative approach that transfers knowledge from seen compositions.
• To overcome limitations of existing discriminative models that rely on compatibility between image and text embeddings without modeling mode-specific variations.
• To develop a feature generator that captures the true distribution of image features for novel compositions using task-aware, deep sampling.

Proposed method

• The TFG framework consists of a task-aware feature generator, a discriminator, and a classifier trained end-to-end.
• The generator produces synthetic features conditioned on word embeddings of attribute-object compositions, using task-aware deep sampling with noise injected at each layer.
• Task-conditioned noise is introduced incrementally across network layers to model task-specific variations and improve feature distribution fidelity.
• The discriminator distinguishes real features from synthetic features of seen compositions, improving the realism of generated features.
• The classifier is trained exclusively on synthetic features to recognize both seen and novel compositions, enabling zero-shot inference.
• The model uses a GAN-like training objective where the generator minimizes the discriminator’s ability to distinguish real and synthetic features.

Experimental results

Research questions

• RQ1 Can a generative model effectively synthesize image features for novel visual compositions without real training images?
• RQ2 Does injecting task-conditioned noise at multiple layers improve feature quality and downstream classification accuracy compared to single-layer noise injection?
• RQ3 Can the proposed TFG framework generalize better than discriminative models in zero-shot compositional learning, especially in the generalized ZSCL setting?
• RQ4 How well do the generated features match the distribution of real image features, and do they form separable clusters for different compositions?
• RQ5 What is the impact of task-aware deep sampling on sample efficiency and convergence speed in feature generation?

Key findings

• The TFG model achieves a new state of the art on three ZSCL benchmarks: MIT-States, UT-Zap50K, and StanfordVRD.
• On the generalized ZSCL benchmark introduced by Purushwalkam et al., TFG improves prior state-of-the-art performance by over 2× in accuracy.
• Feature visualization shows that generated features closely match the real feature distribution and form cleaner, more separable clusters than real features.
• The task-aware noise injection strategy results in better clustering of noise based on task identity, as shown in t-SNE visualizations, compared to unconditioned noise.
• Ablation studies confirm that TFG improves both prediction accuracy and sample efficiency, with faster convergence during training.
• Qualitative results show that the classifier retrieves images matching novel compositions with high confidence, though some failure cases exist, particularly on complex datasets like StanfordVRD.