[Paper Review] Universal representations:The missing link between faces, text, planktons, and cat breeds

Hakan Bilen, Andrea Vedaldi|arXiv (Cornell University)|Jan 25, 2017

Advanced Image and Video Retrieval TechniquesComputer Science52 references127 citations

TL;DR

The paper investigates whether a single neural network can learn universal representations across highly diverse visual domains, showing that extensive sharing is possible with proper normalization, including domain-specific scaling or instance normalization.

ABSTRACT

With the advent of large labelled datasets and high-capacity models, the performance of machine vision systems has been improving rapidly. However, the technology has still major limitations, starting from the fact that different vision problems are still solved by different models, trained from scratch or fine-tuned on the target data. The human visual system, in stark contrast, learns a universal representation for vision in the early life of an individual. This representation works well for an enormous variety of vision problems, with little or no change, with the major advantage of requiring little training data to solve any of them. In this paper we investigate whether neural networks may work as universal representations by studying their capacity in relation to the “size” of a large combination of vision problems. We do so by showing that a single neural network can learn simultaneously several very different visual domains (from sketches to planktons and MNIST digits) as well as, or better than, a number of specialized networks. However, we also show that this requires to carefully normalize the information in the network, by using domain-specific scaling factors or, more generically, by using an instance normalization layer.

Motivation & Objective

• Motivate the idea that a universal visual representation can work across many distinct domains (faces, text, planktons, sketches, etc.).
• Assess the capacity required for a single network to learn multiple diverse vision problems simultaneously.
• Evaluate how different sharing strategies and normalization techniques affect cross-domain performance.
• Identify normalization schemes that enable effective domain adaptation within a shared network.

Proposed method

• Formulate learning from multiple domains as minimizing average risk with a shared blueprint phi_0 and domain-specific adapters phi_d'.
• Propose adapted feature sharing using domain-dependent scaling s_d and bias b_d applied after layers (phi_scale) with a domain mux to select parameters.
• Incorporate batch normalization (BN) and instance normalization (IN) with domain-specific or universal scaling/moments, and study BN+, IN variants.
• Train networks with pure-domain mini-batches in a round-robin fashion to balance domains and enable domain-specific BN moments.
• Experiment with sharing depth (deep sharing, partial sharing) and scale network capacity to assess cross-domain performance.
• Test on small (10 diverse datasets) and large-scale (ImageNet, VGG-Face, Synth90k) setups to evaluate scalability.

Experimental results

Research questions

• RQ1 Can a single CNN share parameters across very diverse visual domains without performance loss?
• RQ2 How does the amount of shared structure (full, deep, partial sharing) impact cross-domain performance?
• RQ3 Which normalization strategy best supports cross-domain universal representations (BN with domain-specific vs universal parameters, IN, BN+)?
• RQ4 How does increasing model capacity affect joint performance when sharing is enforced across domains?
• RQ5 What is the practicality of domain-agnostic normalization in enabling universal representations for new domains?

Key findings

• A single CNN can share all layers across datasets as diverse as CIFAR-10, MNIST, and SVHN with no loss in performance.
• Deep sharing (sharing all layers except the last classifier) often outperforms training domain-specific networks while reducing parameters.
• Full sharing with domain-specific scaling parameters achieves near-parity with separate models across multiple domains, and benefits from increased capacity.
• Domain-specific BN moments and scaling improve cross-domain performance; instance normalization can yield competitive universal representations with fewer domain-specific parameters.
• Instance normalization with universal scaling can operate with a single parameter set across domains, offering practical universality at a modest performance cost.
• On large-scale tasks, sharing convolutional weights across ImageNet and VGG-Face preserves performance closely, and joint training with Synth90k shows substantial parameter efficiency and scalable sharing.