[Paper Review] Learning Concept Taxonomies from Multi-modal Data

Hao Zhang, Zhiting Hu+4 more|arXiv (Cornell University)|Jun 29, 2016

Video Analysis and Summarization30 references3 citations

TL;DR

This paper proposes a discriminatively trained probabilistic Bayesian model that jointly leverages distributed representations of text and images to automatically induce hypernym taxonomies from scratch. By integrating visual and linguistic features through layer-wise weighted fusion, the method achieves a 2x improvement in F1 score over prior approaches on ImageNet hierarchies, demonstrating the effectiveness of multi-modal learning for taxonomy construction.

ABSTRACT

We study the problem of automatically building hypernym taxonomies from textual and visual data. Previous works in taxonomy induction generally ignore the increasingly prominent visual data, which encode important perceptual semantics. Instead, we propose a probabilistic model for taxonomy induction by jointly leveraging text and images. To avoid hand-crafted feature engineering, we design end-to-end features based on distributed representations of images and words. The model is discriminatively trained given a small set of existing ontologies and is capable of building full taxonomies from scratch for a collection of unseen conceptual label items with associated images. We evaluate our model and features on the WordNet hierarchies, where our system outperforms previous approaches by a large gap.

Motivation & Objective

• To address the limitation of existing taxonomy induction methods that rely solely on textual data and ignore rich visual semantics.
• To develop a joint probabilistic model that leverages both textual and visual features for automatic taxonomy construction.
• To eliminate reliance on hand-crafted features by using end-to-end distributed representations of words and images.
• To enable full taxonomy construction from unseen label items with associated images, using only a small set of existing ontologies for discriminative training.
• To evaluate the model's performance on real-world hierarchies, such as ImageNet, and demonstrate the complementary value of visual data.

Proposed method

• The model uses distributed representations—GloVe for words and pre-trained CNNs (e.g., VGG-16, CNN-128) for images—as compact, general-purpose features.
• It formulates a probabilistic Bayesian framework that models the likelihood of is-a relations between concepts based on multi-modal features.
• The model employs layer-wise weight vectors (w_l) to adaptively assign importance to features at different levels of the taxonomy hierarchy.
• Key features include word-word relations (S-T1, PC-T1), image-image similarity (S-V1, PC-V1), and parent-child semantic consistency.
• The model is discriminatively trained using a small set of existing ontologies, enabling zero-shot taxonomy induction for new label collections.
• Feature importance is analyzed via normalized weights across layers, revealing that visual features become more influential at deeper, more specific levels of the hierarchy.

Experimental results

Research questions

• RQ1 Can visual data significantly improve the performance of automatic taxonomy induction compared to text-only methods?
• RQ2 How can distributed representations of images and text be effectively combined in a unified probabilistic framework for taxonomy learning?
• RQ3 Does the importance of visual versus textual features vary across different levels of a semantic hierarchy?
• RQ4 Can a model trained on a small set of existing ontologies generalize to construct full taxonomies from scratch for unseen label collections?
• RQ5 What is the relative contribution of visual similarity and linguistic patterns to identifying hypernym-hyponym relations?

Key findings

• The proposed model achieves a 2x improvement in F1 score on the taxonomy construction task compared to previous approaches, with an F1 of 0.66 versus 0.33 in the best prior method.
• The model outperforms existing baselines on both the taxonomy construction and ancestor-F1 tasks, demonstrating robust generalization across evaluation metrics.
• Enabling visual features (S-V1, PC-V1) leads to a significant performance gain, with the ancestor-F1 score increasing from 0.42 to 0.52 when all visual features are used.
• Layer-wise weight adaptation improves performance, as visual features become more indicative at deeper, more specific levels of the hierarchy, while textual features are more relevant at higher, more abstract levels.
• The VGG-16 CNN provides only a 0.01 improvement in ancestor-F1 over the CNN-128 model, indicating that higher-dimensional features offer diminishing returns in this setting.
• Qualitative analysis confirms that the model correctly predicts most ground-truth links and identifies false positives and missing links with plausible reasoning, especially when visual and textual cues align.