[Paper Review] Transductive Information Maximization For Few-Shot Learning

Mohamed Amine Boudiaf, Imtiaz Masud Ziko+4 more|arXiv (Cornell University)|Aug 25, 2020

Domain Adaptation and Few-Shot LearningComputer Science51 references49 citations

TL;DR

TIM maximizes mutual information between query features and label predictions within a few-shot task, paired with support-set supervision, using a fast ADMM-like solver for transductive inference, and yields state-of-the-art results without complex meta-learning.

ABSTRACT

We introduce Transductive Infomation Maximization (TIM) for few-shot learning. Our method maximizes the mutual information between the query features and their label predictions for a given few-shot task, in conjunction with a supervision loss based on the support set. Furthermore, we propose a new alternating-direction solver for our mutual-information loss, which substantially speeds up transductive-inference convergence over gradient-based optimization, while yielding similar accuracy. TIM inference is modular: it can be used on top of any base-training feature extractor. Following standard transductive few-shot settings, our comprehensive experiments demonstrate that TIM outperforms state-of-the-art methods significantly across various datasets and networks, while used on top of a fixed feature extractor trained with simple cross-entropy on the base classes, without resorting to complex meta-learning schemes. It consistently brings between 2% and 5% improvement in accuracy over the best performing method, not only on all the well-established few-shot benchmarks but also on more challenging scenarios,with domain shifts and larger numbers of classes.

Motivation & Objective

• Motivate few-shot learning needs with limited labeled data for new classes.
• Propose a transductive inference objective based on mutual information between query features and labels.
• Incorporate supervision from the support set via cross-entropy loss.
• Develop a fast solver (alternating-direction/ADMM) for TIM to speed up inference.
• Demonstrate strong empirical gains across standard FSL benchmarks and domain-shift scenarios.

Proposed method

• Define TIM loss as CE on the support set minus an empirical mutual-information term on the query set, with a marginal entropy regularizer to prevent degenerate solutions.
• Express the MI term as - H_hat(Y_Q) + alpha * (1/|Q|) sum_{i in Q} sum_{k} p_{ik} log p_{ik}, where p_{ik} ~ exp(-tau/2 * ||w_k - z_i||^2) and z_i is the L2-normalized embedding.
• Introduce an auxiliary assignment matrix q for query labels and formulate a constrained optimization approximated by ADMM-like updates.
• Provide two optimization strategies: TIM-GD (gradient-based updating only classifier weights W) and TIM-ADM (alternating updates of W and q with closed-form steps).
• Show that the label-marginal entropy term improves optimization stability and runtime.

Experimental results

Research questions

• RQ1 Can a transductive mutual-information objective improve few-shot learning without meta-learning?
• RQ2 Does incorporating a label-marginal entropy regularizer prevent degenerate solutions and speed up convergence?
• RQ3 How do gradient-based and ADMM-like solvers compare in accuracy and speed for TIM?
• RQ4 Is TIM robust to domain shifts and higher number of classes (ways) in few-shot tasks?
• RQ5 Can TIM be plugged on top of a fixed base-trained feature extractor across different backbones?

Key findings

• TIM achieves state-of-the-art results across mini-ImageNet, tiered-ImageNet, and CUB in 1-shot and 5-shot settings with a ResNet-18 backbone and WRN-28-10 in transductive tasks.
• TIM-GD and TIM-ADM consistently outperform previous transductive and inductive methods, with TIM-ADM offering faster inference.
• The label-marginal entropy term substantially boosts accuracy (and optimization) and can reduce transductive runtimes by orders of magnitude.
• Fine-tuning the whole backbone during transduction (as in some prior work) degrades TIM performance and is considerably slower.
• TIM demonstrates strong performance under domain shift (mini-ImageNet to CUB) and in higher-way scenarios (10-way, 20-way).
• The ADM-based solver (TIM-ADM) reduces per-task inference time by about one order of magnitude relative to TIM-GD and by over two orders of magnitude compared to prior transductive methods.