[Paper Review] Open Vocabulary Scene Parsing

Hang Zhao, Xavier Puig+3 more|arXiv (Cornell University)|Mar 26, 2017

Multimodal Machine Learning Applications4 citations

TL;DR

This paper introduces open vocabulary scene parsing, a novel task that enables scene segmentation using a large, open-ended vocabulary of object categories beyond fixed training sets. The authors propose a joint image-pixel and word-concept embedding framework that preserves hierarchical semantic relations (e.g., from WordNet), enabling zero-shot generalization, concept retrieval, and arithmetic composition of concepts—achieving strong performance on the ADE20K dataset with interpretable, structured predictions.

ABSTRACT

Recognizing arbitrary objects in the wild has been a challenging problem due to the limitations of existing classification models and datasets. In this paper, we propose a new task that aims at parsing scenes with a large and open vocabulary, and several evaluation metrics are explored for this problem. Our proposed approach to this problem is a joint image pixel and word concept embeddings framework, where word concepts are connected by semantic relations. We validate the open vocabulary prediction ability of our framework on ADE20K dataset which covers a wide variety of scenes and objects. We further explore the trained joint embedding space to show its interpretability.

Motivation & Objective

• To address the limitation of fixed-vocabulary scene parsing models that cannot generalize to unseen object categories.
• To enable scene parsing with a large, open-ended vocabulary of object concepts, including those not seen during training.
• To develop a unified framework that preserves semantic hierarchies (e.g., hypernymy/hyponymy) in a joint embedding space.
• To explore interpretability of the learned embedding space through concept retrieval, boundary loosening, and arithmetic operations.
• To establish evaluation metrics suitable for open-ended recognition tasks with inherent class ambiguity.

Proposed method

• Train a deep neural network to jointly embed image pixel features and word concepts from a knowledge graph (e.g., WordNet) into a shared high-dimensional vector space.
• Enforce semantic hierarchy preservation by optimizing for hypernym-hyponym relations in the embedding space using a margin-based loss.
• Use a weighted cross-entropy loss that accounts for class frequency via information content to reduce bias toward frequent categories.
• Enable zero-shot prediction by allowing predictions at intermediate nodes in the WordNet hierarchy, not just leaf nodes.
• Perform concept retrieval, boundary loosening, and concept synthesis via arithmetic operations (e.g., min/max) in the embedding space.
• Leverage the ADE20K dataset for training and evaluation, using its rich scene annotations and diverse object categories.

Experimental results

Research questions

• RQ1 Can a scene parsing model generalize to object categories not seen during training, using only semantic knowledge from a knowledge graph?
• RQ2 How well can a joint image-concept embedding space preserve hierarchical semantic relations such as hypernymy and hyponymy?
• RQ3 To what extent can the learned embedding space support interpretable operations like concept retrieval and arithmetic composition of concepts?
• RQ4 What is the minimum number of training classes required to achieve robust zero-shot generalization in open vocabulary parsing?
• RQ5 How do different evaluation metrics reflect the performance of open-ended recognition systems with ambiguous or hierarchical labels?

Key findings

• The proposed Joint-Hyper model achieves strong zero-shot generalization, with performance saturating after training on more than 500 classes, suggesting diminishing returns beyond that scale.
• The model successfully performs pixel-level concept retrieval, with increasingly abstract concepts (e.g., 'furniture' or 'object') yielding higher recall than Word2Vec-based baselines.
• The embedding space implicitly encodes abstract attributes such as 'sittability', allowing detection of regions that support sitting (e.g., benches, ottomans) even when not explicitly labeled as 'chair'.
• Arithmetic operations in the embedding space enable meaningful concept synthesis: max operation retrieves common hyponyms (e.g., 'pool table' as intersection of 'table' and 'game equipment'), and min operation retrieves union-like concepts (e.g., 'cart' as blend of 'bicycle' and 'canopy').
• The model produces plausible, structured predictions even for unseen categories—e.g., predicting 'vehicle' for a tricycle when 'tricycle' was not in training—demonstrating human-like generalization.
• The framework is robust to inconsistent human annotations, as the hierarchical structure reduces sensitivity to labeling noise from workers with varying domain knowledge.