[Paper Review] Probabilistic Latent Semantic Analysis

Thomas Hofmann|arXiv (Cornell University)|Jan 23, 2013

Bayesian Modeling and Causal Inference13 references2,092 citations

TL;DR

Introduces Probabilistic Latent Semantic Analysis (PLSA), a probabilistic model for analyzing two-mode and co-occurrence data, based on a latent class mixture and tempered EM to prevent overfitting, with improvements over standard LSA observed in experiments.

ABSTRACT

Probabilistic Latent Semantic Analysis is a novel statistical technique for the analysis of two-mode and co-occurrence data, which has applications in information retrieval and filtering, natural language processing, machine learning from text, and in related areas. Compared to standard Latent Semantic Analysis which stems from linear algebra and performs a Singular Value Decomposition of co-occurrence tables, the proposed method is based on a mixture decomposition derived from a latent class model. This results in a more principled approach which has a solid foundation in statistics. In order to avoid overfitting, we propose a widely applicable generalization of maximum likelihood model fitting by tempered EM. Our approach yields substantial and consistent improvements over Latent Semantic Analysis in a number of experiments.

Motivation & Objective

• Motivate the need for a probabilistic approach to co-occurrence data analysis in information retrieval, NLP, and related fields.
• Develop a probabilistic latent semantic analysis framework based on a latent class model.
• Address overfitting through a tempered EM variant of maximum likelihood fitting.
• Demonstrate empirical improvements over standard Latent Semantic Analysis across tasks.

Proposed method

• Formulate PLSA as a mixture decomposition derived from a latent class model for two-mode and co-occurrence data.
• Apply tempered EM to estimate model parameters and mitigate overfitting in maximum likelihood fitting.
• Compare PLSA with standard LSA on relevant information retrieval and NLP tasks to show improvements.
• Provide a principled probabilistic foundation for latent semantic analysis, contrasting with linear algebra-based SVD.
• Discuss general applicability of the approach across related domains.

Experimental results

Research questions

• RQ1 Can a probabilistic latent class formulation improve the analysis of co-occurrence data over standard LSA?
• RQ2 Does tempered EM reduce overfitting and yield more robust parameter estimates in PLSA?
• RQ3 In which tasks and settings does PLSA outperform traditional LSA?
• RQ4 What is the empirical impact of the probabilistic approach on information retrieval and NLP applications?

Key findings

• PLSA provides a principled probabilistic framework for latent semantic analysis based on a latent class model.
• Tempered EM is proposed to generalize maximum likelihood fitting and prevent overfitting.
• Empirical experiments show substantial and consistent improvements of PLSA over standard LSA in several settings.