[Paper Review] Anchored Discrete Factor Analysis

Yoni Halpern, Steven Horng+1 more|arXiv (Cornell University)|Nov 10, 2015

Machine Learning and Data Classification32 references5 citations

TL;DR

This paper introduces Anchored Discrete Factor Analysis (ADFA), a semi-supervised method for learning structured discrete factor models with arbitrary latent variable dependencies. By leveraging user-specified anchor variables—observed variables with only one latent parent—it enables consistent, polynomial-time recovery of latent moments and model parameters via method-of-moments, outperforming discriminative baselines on real-world tag prediction and medical diagnosis tasks.

ABSTRACT

We present a semi-supervised learning algorithm for learning discrete factor analysis models with arbitrary structure on the latent variables. Our algorithm assumes that every latent variable has an "anchor", an observed variable with only that latent variable as its parent. Given such anchors, we show that it is possible to consistently recover moments of the latent variables and use these moments to learn complete models. We also introduce a new technique for improving the robustness of method-of-moment algorithms by optimizing over the marginal polytope or its relaxations. We evaluate our algorithm using two real-world tasks, tag prediction on questions from the Stack Overflow website and medical diagnosis in an emergency department.

Motivation & Objective

• To address the challenge of learning interpretable and identifiable latent variable models in the presence of unobserved confounders.
• To overcome the non-identifiability and computational intractability of traditional latent variable models in discrete factor analysis.
• To enable structure learning of latent variables without prior knowledge of the network structure, a key limitation in real-world applications.
• To improve robustness of method-of-moments estimation under model misspecification through constrained optimization over the marginal polytope.
• To demonstrate that structured latent representations outperform independent latent variable models in predictive performance on real-world datasets.

Proposed method

• Introduces the concept of 'anchor' variables—observed variables with only one latent parent—to enable consistent moment recovery of latent variables.
• Uses a two-stage method-of-moments approach: first recover low-order moments of latent variables from anchor observations, then reconstruct the joint latent structure.
• Applies constrained optimization over the marginal polytope (or its relaxations) to improve robustness of moment estimation under model misspecification.
• Employs a tree-structured latent model learning algorithm that efficiently recovers latent dependencies from recovered moments.
• Combines anchor-based moment estimation with EM-based refinement for likelihood maximization, though shows that such refinement can degrade interpretability.
• Uses simplex constraints and outer approximations of the marginal polytope to tighten moment estimation and reduce sensitivity to noise.

Experimental results

Research questions

• RQ1 Can we consistently recover the structure and parameters of discrete factor analysis models with arbitrary latent variable dependencies using only observed data and anchor variables?
• RQ2 How does enforcing constraints on the marginal polytope improve the robustness of method-of-moments estimation in the presence of model misspecification?
• RQ3 Does learning a structured representation of latent variables lead to better predictive performance than independent latent variable models in real-world tasks?
• RQ4 To what extent does likelihood-based optimization (e.g., EM) improve performance over anchor-based method-of-moments initialization, and does it preserve interpretability?
• RQ5 Can anchor-based methods achieve performance competitive with maximum likelihood estimation while remaining computationally efficient and interpretable?

Key findings

• The ADFA algorithm successfully learns structured latent variable models with arbitrary dependencies between binary latent variables using only anchor observations.
• On the Stack Overflow dataset, the tree-structured ADFA model significantly outperformed both independent latent variable models and discriminative baselines in tag prediction accuracy.
• On the emergency department dataset, the model achieved strong performance, though the high informativeness of anchors limited further gains, suggesting anchors are highly effective for knowledge discovery.
• Likelihood-based optimization (EM) starting from ADFA initialization led to a degradation in classification accuracy, indicating that likelihood maximization can distort the semantic meaning of latent variables.
• The latent variable associated with the 'headache' anchor shifted its meaning during EM optimization, becoming associated with long negation scopes rather than clinical symptoms, demonstrating a loss of interpretability.
• Constraining optimization over the marginal polytope improved density estimation of the latent distribution but did not translate to better predictive accuracy, suggesting trade-offs between model fit and task performance.