[Paper Review] StarSpace: Embed All The Things!

Ledell Wu, Adam Fisch+4 more|arXiv (Cornell University)|Sep 12, 2017

Topic Modeling21 references92 citations

TL;DR

StarSpace is a general-purpose neural embedding model that embeds discrete-feature entities and compares them in a common space for labeling, ranking, recommendations, and graph embedding tasks.

ABSTRACT

We present StarSpace, a general-purpose neural embedding model that can solve a wide variety of problems: labeling tasks such as text classification, ranking tasks such as information retrieval/web search, collaborative filtering-based or content-based recommendation, embedding of multi-relational graphs, and learning word, sentence or document level embeddings. In each case the model works by embedding those entities comprised of discrete features and comparing them against each other -- learning similarities dependent on the task. Empirical results on a number of tasks show that StarSpace is highly competitive with existing methods, whilst also being generally applicable to new cases where those methods are not.

Motivation & Objective

• Develop a general embedding model that can handle labeling, ranking, and embedding tasks across diverse domains.
• Show that StarSpace is competitive with specialized methods on multiple tasks.
• Demonstrate natural handling of out-of-sample entities and multi-type comparisons.
• Provide an open-source implementation for broad adoption and experimentation.

Proposed method

• Represent each entity as a bag of discrete features from a fixed dictionary.
• Embed features into a shared d-dimensional space and form entity embeddings by summing feature embeddings.
• Train with a margin-based loss using negative sampling to discriminate positive from negative pairs.
• Use a similarity function (cosine or inner product) as the basis for ranking/classification in the loss.
• Apply stochastic gradient descent with Adagrad and Hogwild over CPUs, plus a max-norm constraint on embeddings.
• Provide task-specific generators for positive/negative pairs (E+, E−) to cover classification, recommendation, and knowledge graph linked tasks.

Experimental results

Research questions

• RQ1 Can a single embedding framework perform text classification, document/document ranking, collaborative/content-based filtering, and knowledge graph embedding effectively?
• RQ2 How does StarSpace perform relative to task-specific baselines (e.g., fastText, TransE, TFIDF, SVM) across diverse tasks?
• RQ3 Can the model naturally handle out-of-sample entities by featurizing users/items/labels as bags of features?
• RQ4 What is the impact of the number of negative samples (k) on performance and training efficiency?

Key findings

• StarSpace achieves competitive performance with fastText on text classification benchmarks and excels in document recommendation tasks.
• On Freebase 15k, StarSpace matches TransE in link prediction with comparable embedding dimensions.
• In Wikipedia search and sentence matching, StarSpace outperforms several baselines by training directly for the task (sentence/document level).
• StarSpace enables out-of-sample extension for users/items via feature-based representations rather than fixed IDs.
• Increasing the negative sample parameter k shows trade-offs between training speed and predictive performance, with certain ranges yielding best results within time limits.