[Paper Review] Hierarchical Optimal Transport for Document Representation

Mikhail Yurochkin, Sebastian Claici+3 more|arXiv (Cornell University)|Jun 26, 2019

Music and Audio Processing31 references35 citations

TL;DR

Introduces hierarchical optimal transport (HOTT) for measuring document similarity by modeling documents as distributions over topics (which are distributions over words) and computing a Wasserstein distance on the topic space using WMD as the ground metric. It achieves better interpretability and scalability than Word Mover’s Distance (WMD) with comparable accuracy in k-NN classification.

ABSTRACT

The ability to measure similarity between documents enables intelligent summarization and analysis of large corpora. Past distances between documents suffer from either an inability to incorporate semantic similarities between words or from scalability issues. As an alternative, we introduce hierarchical optimal transport as a meta-distance between documents, where documents are modeled as distributions over topics, which themselves are modeled as distributions over words. We then solve an optimal transport problem on the smaller topic space to compute a similarity score. We give conditions on the topics under which this construction defines a distance, and we relate it to the word mover's distance. We evaluate our technique for k-NN classification and show better interpretability and scalability with comparable performance to current methods at a fraction of the cost.

Motivation & Objective

• Motivate efficient, interpretable document similarity that leverages semantic word information and corpus-specific topics.
• Propose a hierarchical transport distance that reduces complexity by transporting between topics rather than words.
• Relate HOTT to existing metrics (notably WMD) and analyze conditions for it to be a metric.
• Demonstrate HOTT’s performance and scalability on k-NN classification and visualization tasks.
• Explore robustness to topic count, embedding quality, and truncation strategies.

Proposed method

• Represent each document as a distribution over topics obtained from LDA.
• Represent topics as distributions over words, using word embeddings to define a ground metric between topics via WMD on word distributions.
• Define HOTT as a 1-Wasserstein distance between the topic-mixed topic distributions of two documents with topic-level Dirac masses weighted by document topic proportions.
• Precompute pairwise topic distances (WMD between topics) and truncate topics to a small set of words to enhance interpretability and reduce variance.
• Show theoretical connections and bounds to WMD, including how HOTT reduces to WMD when number of topics equals vocabulary size.
• Evaluate on k-NN classification, visualizations (t-SNE), and link prediction in citation networks; compare against RWMD, WMD, and baselines.

Experimental results

Research questions

• RQ1 Can hierarchical optimal transport provide a metric-based, scalable alternative to WMD for document similarity?
• RQ2 How does incorporating corpus-specific topics and word-embedding grounded topic distances affect interpretability and performance?
• RQ3 What is the impact of the number of topics and topic truncation on accuracy and speed?
• RQ4 Is HOTT robust to embedding quality and document length across various datasets?
• RQ5 Can HOTT be effectively used for visualization and downstream tasks like link prediction?

Key findings

• HOTT is faster than WMD/RWMD due to smaller transport problems at the topic level and achieves comparable or better k-NN accuracy on several datasets.
• Topic-level interpretability is enhanced, as transport between semantically meaningful topics can be inspected (e.g., transport between authors’ heaviest topics).
• HOTT remains robust to the number of topics and word-embedding quality, and truncating topics to top words preserves performance without sacrificing speed.
• On Gutenberg and longer documents, HOTT outperforms competitors in throughput and distance-based classification; on shorter documents, performance is close to RWMD/WMD.
• HOTT scales well to large corpora since the transport problem is solved on the smaller topic space, with precomputed topic distances.
• Visualizations (t-SNE) using HOTT distances show clearer separation of labels compared to RWMD.