[Paper Review] Quantifying Learnability and Describability of Visual Concepts Emerging in Representation Learning

Iro Laina, Ruth Fong+1 more|arXiv (Cornell University)|Oct 27, 2020

Multimodal Machine Learning Applications69 references4 citations

TL;DR

This paper introduces two objective, human-based metrics—learnability and describability—to quantify the interpretability of visual concepts discovered by self-supervised representation learning. By framing human annotators as classifiers to measure learnability and testing their ability to recognize classes from textual descriptions to assess describability, the authors provide a principled, scalable evaluation framework that reduces subjectivity and enables comparison between human and automated captioning systems for class-level descriptions.

ABSTRACT

The increasing impact of black box models, and particularly of unsupervised ones, comes with an increasing interest in tools to understand and interpret them. In this paper, we consider in particular how to characterise visual groupings discovered automatically by deep neural networks, starting with state-of-the-art clustering methods. In some cases, clusters readily correspond to an existing labelled dataset. However, often they do not, yet they still maintain an "intuitive interpretability". We introduce two concepts, visual learnability and describability, that can be used to quantify the interpretability of arbitrary image groupings, including unsupervised ones. The idea is to measure (1) how well humans can learn to reproduce a grouping by measuring their ability to generalise from a small set of visual examples (learnability) and (2) whether the set of visual examples can be replaced by a succinct, textual description (describability). By assessing human annotators as classifiers, we remove the subjective quality of existing evaluation metrics. For better scalability, we finally propose a class-level captioning system to generate descriptions for visual groupings automatically and compare it to human annotators using the describability metric.

Motivation & Objective

• To develop a principled, objective evaluation framework for assessing the interpretability of visual concepts discovered via self-supervised learning.
• To reduce subjectivity in interpretability evaluation by using human annotators as classifiers rather than relying on subjective naming or description tasks.
• To quantify how well human-learnable and describable self-supervised visual clusters are, especially those not aligned with existing labeled datasets like ImageNet.
• To investigate whether automated class-level captioning systems can generate descriptions that match human-level describability.
• To enable direct, quantitative comparison between human- and machine-generated descriptions of visual clusters.

Proposed method

• Measure learnability by training human annotators to classify images as belonging to a given cluster or not, using only a few examples; classification accuracy serves as an objective metric of semantic coherence.
• Assess describability by providing annotators with a natural language description of a cluster and measuring their ability to correctly classify new examples based on the description.
• Use a class-level captioning system—adapted from single-image captioning models—to automatically generate descriptions for visual clusters.
• Compare the performance of automated captions against human-annotated descriptions using the describability metric to validate the quality of machine-generated descriptions.
• Design a forced-choice experimental setup where humans are tested on both learnability and describability, ensuring objectivity and scalability.
• Introduce a metric to validate descriptions independently of human evaluation, enabling automated benchmarking of captioning systems for visual clusters.

Experimental results

Research questions

• RQ1 To what extent can humans learn to recognize a visual concept from a small set of examples, even when the concept is not labeled in standard datasets?
• RQ2 Can a visual cluster be faithfully summarized by a concise, natural language description that enables accurate recognition by humans?
• RQ3 How well can automated captioning systems generate descriptions for visual clusters that match the quality of human-annotated descriptions?
• RQ4 Can the proposed metrics—learnability and describability—serve as objective, scalable alternatives to subjective human evaluation in interpretability research?
• RQ5 Do self-supervised models discover semantically coherent visual concepts that are not captured by standard image classification labels?

Key findings

• Self-supervised models can discover semantically coherent visual concepts even when they do not align with standard labels like those in ImageNet, as evidenced by high human learnability scores.
• Human annotators achieve high accuracy in classifying images based on a small set of examples, indicating strong semantic coherence of discovered clusters.
• The describability metric successfully quantifies how effectively a natural language description captures the essence of a visual cluster, with human-annotated descriptions outperforming baseline automated captions in most cases.
• The proposed class-level captioning system can generate descriptions that are competitive with human-annotated ones, especially when fine-tuned on cluster-level data.
• The framework enables direct, quantitative comparison between human and automated descriptions, demonstrating that automated systems can achieve high describability when properly adapted.
• The learnability and describability metrics provide a scalable, objective alternative to subjective evaluation methods in interpretability research.