[Paper Review] Dynamic Few-Shot Visual Learning without Forgetting

Spyros Gidaris, Nikos Komodakis|arXiv (Cornell University)|Apr 25, 2018

Domain Adaptation and Few-Shot LearningComputer Science23 references72 citations

TL;DR

The paper presents a dynamic few-shot learning system that quickly learns novel categories from few examples at test time while preserving accuracy on base categories, using a cosine-similarity classifier and an attention-based few-shot weight generator. It achieves state-of-the-art results on Mini-ImageNet and the Bharath & Girshick few-shot benchmark.

ABSTRACT

The human visual system has the remarkably ability to be able to effortlessly learn novel concepts from only a few examples. Mimicking the same behavior on machine learning vision systems is an interesting and very challenging research problem with many practical advantages on real world vision applications. In this context, the goal of our work is to devise a few-shot visual learning system that during test time it will be able to efficiently learn novel categories from only a few training data while at the same time it will not forget the initial categories on which it was trained (here called base categories). To achieve that goal we propose (a) to extend an object recognition system with an attention based few-shot classification weight generator, and (b) to redesign the classifier of a ConvNet model as the cosine similarity function between feature representations and classification weight vectors. The latter, apart from unifying the recognition of both novel and base categories, it also leads to feature representations that generalize better on "unseen" categories. We extensively evaluate our approach on Mini-ImageNet where we manage to improve the prior state-of-the-art on few-shot recognition (i.e., we achieve 56.20% and 73.00% on the 1-shot and 5-shot settings respectively) while at the same time we do not sacrifice any accuracy on the base categories, which is a characteristic that most prior approaches lack. Finally, we apply our approach on the recently introduced few-shot benchmark of Bharath and Girshick [4] where we also achieve state-of-the-art results. The code and models of our paper will be published on: https://github.com/gidariss/FewShotWithoutForgetting

Motivation & Objective

• Motivate and formalize the problem of dynamic few-shot learning without forgetting base categories.
• Develop a few-shot classification weight generator that leverages past visual knowledge via attention.
• Reformulate the classifier as a cosine similarity to unify base and novel category recognition.
• Evaluate the approach on Mini-ImageNet and a contemporary few-shot benchmark, achieving state-of-the-art results.

Proposed method

• Introduce a Cosine-similarity based ConvNet classifier to unify base and novel categories.
• Implement a few-shot classification weight generator that uses feature averaging and attention over base-class weights.
• Incorporate an attention mechanism to compose novel weights from base-category vectors.
• Train in two stages: (i) learn feature extractor and base weights; (ii) train the weight generator while freezing the feature extractor.
• Remove the ReLU after the last feature layer to allow z to take negative values and improve cosine-based classification.
• Use an attention kernel with cosine similarity and a learnable scale to query a memory of base-category weights.

Experimental results

Research questions

• RQ1 Can a few-shot model dynamically incorporate novel categories at test time without degrading base-category performance?
• RQ2 Does a cosine-similarity classifier provide better generalization to unseen categories than a dot-product classifier in a few-shot setting?
• RQ3 Can an attention-based weight generator effectively compose novel classification weights from base-category vectors?
• RQ4 Does unifying base and novel category recognition improve overall performance on standard few-shot benchmarks?

Key findings

• On Mini-ImageNet, the Cosine Classifier with Attention Weight Gen achieves 74.92% (5-shot novel) and 70.88% (base) with 60.50% (both) in 1-shot/5-shot evaluation settings on the validation set, outperforming prior methods.
• The Cosine Classifier with Avg. Weight Gen also attains strong results (74.66% novel, 70.92% base, 60.26% both in 5-shot; 55.33% novel, 70.45% base, 48.56% both in 1-shot).
• A cosine-similarity based classifier yields more compact, discriminative feature clusters for unseen categories compared to dot-product baselines.
• When evaluated on the Mini-ImageNet test set, the proposed models with various backbones (e.g., ResNet, Conv4 variants) achieve competitive to superior performance across 5-shot and 1-shot tasks, maintaining strong base-category accuracy.
• The proposed approach achieves state-of-the-art results on the Bharath and Girshick few-shot benchmark.