[Paper Review] Revisiting Semi-Supervised Learning with Graph Embeddings

Zhilin Yang, William W. Cohen+1 more|arXiv (Cornell University)|Mar 29, 2016

Topic Modeling17 references284 citations

TL;DR

The Planetoid framework jointly learns graph embeddings and label predictors in transductive and inductive variants, improving semi-supervised learning across text classification, distant supervision, and entity classification tasks.

ABSTRACT

We present a semi-supervised learning framework based on graph embeddings. Given a graph between instances, we train an embedding for each instance to jointly predict the class label and the neighborhood context in the graph. We develop both transductive and inductive variants of our method. In the transductive variant of our method, the class labels are determined by both the learned embeddings and input feature vectors, while in the inductive variant, the embeddings are defined as a parametric function of the feature vectors, so predictions can be made on instances not seen during training. On a large and diverse set of benchmark tasks, including text classification, distantly supervised entity extraction, and entity classification, we show improved performance over many of the existing models.

Motivation & Objective

• Leverage unlabeled data via graph structure to improve predictive accuracy.
• Jointly learn node embeddings and label predictors to utilize distributional graph information.
• Develop both transductive and inductive variants to handle seen and unseen instances.
• Demonstrate effectiveness across multiple domains: text classification, distantly supervised extraction, and entity classification.

Proposed method

• Define a neural network with embeddings e_i for each instance learned alongside standard classifier features.
• Formulate loss as L_s + lambda L_u, where L_s is supervised label prediction and L_u predicts graph context.
• Sample (i, c, gamma) pairs from a context distribution p(i, c, gamma) using a mix of graph walks and label-based context, with negative sampling.
• Use a Skipgram-like objective for context prediction with sigmoid-based binary classification.
• Transductive variant concatenates x-derived and embedding-derived representations for final softmax prediction; embeddings are learned jointly.
• Inductive variant defines e as a parametric function of input features x, enabling predictions for unseen instances; training updates only parameters theta, with embeddings derived from x.
• Training proceeds via stochastic gradient descent with alternating optimization of L_s and L_u, including a pretraining phase for L_u.

Experimental results

Research questions

• RQ1 Does joint training of embeddings and label predictors improve semi-supervised learning over graph Laplacian regularization?
• RQ2 How do transductive and inductive variants compare in accuracy and generalization?
• RQ3 How does Planetoid perform on text classification, distantly supervised entity extraction, and entity classification?
• RQ4 What is the impact of sampling graph-based context (via random walks and label-based context) on performance?

Key findings

• Inductive Planetoid-I outperforms the second-best inductive method by up to 18.7 percentage points (pp) and on average by 7.8 pp.
• The best of Planetoid’s inductive and transductive variants outperforms other methods by up to 8.5 pp and on average 4.1 pp.
• On text classification (Citeseer, Cora, Pubmed), Planetoid-I often achieves the best inductive results, Planetoid-T achieves the best transductive results on Cora and Pubmed, with Planetoid-G outperforming GraphEmb.
• In distantly-supervised entity extraction (DIEL), Planetoid-I and Planetoid-T significantly outperform competitive baselines across recall@k, with Planetoid variants achieving best results in multiple runs.
• For NELL entity classification, Planetoid-I substantially outperforms the strongest inductive baseline across various labeling rates, and GraphEmb-based variants show strong gains when graph structure is highly informative.