[Paper Review] Generative OpenMax for Multi-Class Open Set Classification

Zongyuan Ge, Sergey Demyanov+2 more|arXiv (Cornell University)|Jul 24, 2017

Domain Adaptation and Few-Shot Learning18 references54 citations

TL;DR

The paper introduces Generative OpenMax (G-OpenMax), which uses GAN-generated samples to explicitly model and detect unknown classes in multi-class open set classification, improving over OpenMax on MNIST and HASYv2 while enabling visualization of unknown samples.

ABSTRACT

We present a conceptually new and flexible method for multi-class open set classification. Unlike previous methods where unknown classes are inferred with respect to the feature or decision distance to the known classes, our approach is able to provide explicit modelling and decision score for unknown classes. The proposed method, called Gener- ative OpenMax (G-OpenMax), extends OpenMax by employing generative adversarial networks (GANs) for novel category image synthesis. We validate the proposed method on two datasets of handwritten digits and characters, resulting in superior results over previous deep learning based method OpenMax Moreover, G-OpenMax provides a way to visualize samples representing the unknown classes from open space. Our simple and effective approach could serve as a new direction to tackle the challenging multi-class open set classification problem.

Motivation & Objective

• Motivate open set recognition beyond closed-set assumptions by explicitly modeling unknown classes.
• Extend OpenMax to provide explicit probability estimates for unknown categories.
• Use GAN-generated samples to represent plausible unknowns and calibrate scores accordingly.
• Evaluate on MNIST and HASYv2 to assess gains in open set recognition across varying openness.

Proposed method

• Extend OpenMax by training a classifier with an extra unknown class using GAN-generated samples.
• Use a conditional GAN to synthesize plausible unknown-class images from mixtures of known-class latent subspaces.
• Select generated samples that are misclassified by a pre-trained classifier to label as unknown for training Net^G.
• Train Net^G to produce activation vectors including an extra dimension for the unknown class, enabling explicit unknown probability estimates.
• Fit Weibull models to activation scores and calibrate using the OpenMax-style reweighting, yielding Prob(y|x) for known classes and Prob(unknown|x).
• During inference, recalibrate activations and threshold to decide known vs unknown.

Experimental results

Research questions

• RQ1 Can explicit modeling of unknown classes via synthetic samples improve multi-class open set recognition?
• RQ2 How does GAN-based unknown sample synthesis affect calibration and decision margins in open set classification?
• RQ3 What is the impact of mixing known-class latent representations to generate plausible unknown samples on open space risk?
• RQ4 Is G-OpenMax robust to openness and tail-size choices in EVT-based calibration?
• RQ5 Can G-OpenMax visualize representative unknown-class samples effectively to accompany decisions?

Key findings

• G-OpenMax consistently improves open set recognition over OpenMax on MNIST and HASYv2 across openness levels.
• Using GAN-synthesized unknown samples enables non-pseudo probability estimation and explicit unknown-class scores.
• G-OpenMax achieves about 10 percentage points higher accuracy than OpenMax at optimal thresholds on evaluated settings.
• Ablation studies show GAN-based samples improve unknown-class accuracy and calibration across tail-size variations.
• Qualitative analysis visualizes mixture-generated samples that resemble plausible unknowns in the training subspace.
• G-OpenMax is robust to large Weibull tail sizes for known-class detection while maintaining overall performance.