[Paper Review] Learning Concept Hierarchies through Probabilistic Topic Modeling

V. S. Anoop, S. Asharaf+1 more|arXiv (Cornell University)|Nov 29, 2016

Advanced Text Analysis Techniques10 references3 citations

TL;DR

This paper proposes an unsupervised framework that leverages Latent Dirichlet Allocation (LDA) for topic modeling and combines it with lightweight linguistic processing to extract human-interpretable concepts from unstructured text. It then constructs a subsumption hierarchy using probabilistic "is-a" relations between concepts, achieving superior performance in concept extraction and hierarchy learning compared to existing methods on BBC and Reuters datasets.

ABSTRACT

With the advent of semantic web, various tools and techniques have been introduced for presenting and organizing knowledge. Concept hierarchies are one such technique which gained significant attention due to its usefulness in creating domain ontologies that are considered as an integral part of semantic web. Automated concept hierarchy learning algorithms focus on extracting relevant concepts from unstructured text corpus and connect them together by identifying some potential relations exist between them. In this paper, we propose a novel approach for identifying relevant concepts from plain text and then learns hierarchy of concepts by exploiting subsumption relation between them. To start with, we model topics using a probabilistic topic model and then make use of some lightweight linguistic process to extract semantically rich concepts. Then we connect concepts by identifying an "is-a" relationship between pair of concepts. The proposed method is completely unsupervised and there is no need for a domain specific training corpus for concept extraction and learning. Experiments on large and real-world text corpora such as BBC News dataset and Reuters News corpus shows that the proposed method outperforms some of the existing methods for concept extraction and efficient concept hierarchy learning is possible if the overall task is guided by a probabilistic topic modeling algorithm.

Motivation & Objective

• To automatically extract human-interpretable concepts from large unstructured text corpora without domain-specific training data.
• To learn a subsumption hierarchy ("is-a" relationships) between extracted concepts for ontology generation.
• To evaluate whether topic modeling-guided concept extraction improves performance over existing unsupervised methods.
• To demonstrate scalability and effectiveness on real-world news corpora like BBC and Reuters.

Proposed method

• Apply Latent Dirichlet Allocation (LDA) to model topics in a large unstructured text corpus.
• Use a lightweight linguistic process to identify multi-word terms as candidate concepts based on term frequency and inverse topic frequency.
• Compute normalized term frequency (Ntf) and inverse topic frequency (itf) to score and filter semantically rich multi-word terms.
• Establish subsumption relations by calculating conditional probabilities P(C₁|C₂) and P(C₂|C₁), where C₁ subsumes C₂ if P(C₁|C₂) = 1 and P(C₂|C₁) < 1.
• Construct a hierarchical structure by recursively applying the subsumption condition across all concept pairs.
• Use human-annotated concept repositories as ground truth for evaluation using precision, recall, and F1-score.

Experimental results

Research questions

• RQ1 Can human-interpretable concepts be effectively extracted from statistically generated topics using a lightweight linguistic process?
• RQ2 Can the proposed method learn a meaningful subsumption hierarchy using "is-a" relationships between extracted concepts?
• RQ3 Does topic modeling-guided concept extraction outperform existing unsupervised algorithms in concept identification and hierarchy learning?
• RQ4 How does the performance of the proposed method scale with increasing numbers of topics in large text corpora?

Key findings

• The proposed method achieved a precision of 0.8165, recall of 0.8901, and F1-score of 0.8516 on concept extraction, outperforming ACE (F1: 0.2517) and ICE (F1: 0.7595).
• The method extracted more human-interpretable concepts as the number of topics increased, demonstrating scalability and robustness.
• Performance on the BBC and Reuters news corpora showed consistent superiority over baseline methods like ACE and ICE.
• The subsumption hierarchy learning module successfully identified hierarchical relationships such as "dial-up internet" being subsumed by "network connection" based on document co-occurrence patterns.
• The framework operates entirely in an unsupervised manner, requiring no domain-specific training data or human-labeled corpora.
• The results indicate that topic modeling guidance significantly enhances the quality and relevance of extracted concepts and their hierarchical organization.