[Paper Review] Tailoring Word Embeddings for Bilexical Predictions: An Experimental Comparison

Pranava Madhyastha, Xavier Carreras+1 more|arXiv (Cornell University)|Dec 22, 2014

Topic Modeling11 references3 citations

TL;DR

This paper proposes a method to learn task-specific word embeddings tailored for bilexical relations by compressing pre-trained word vectors using low-rank regularization on a bilinear compatibility model. The approach achieves state-of-the-art performance with highly compressed embeddings (as low as 5 dimensions), outperforming both general-purpose embeddings and alternative regularization schemes in predictive accuracy and efficiency.

ABSTRACT

We investigate the problem of inducing word embeddings that are tailored for a particular bilexical relation. Our learning algorithm takes an existing lexical vector space and compresses it such that the resulting word embeddings are good predictors for a target bilexical relation. In experiments we show that task-specific embeddings can benefit both the quality and efficiency in lexical prediction tasks.

Motivation & Objective

• To investigate whether task-specific word embeddings can improve predictive performance and efficiency in bilexical relation modeling.
• To determine if compressing general-purpose word embeddings using low-rank constraints enhances their predictive power for specific linguistic relations.
• To compare the effectiveness of different regularization techniques—nuclear norm ($\ell_\star$), $\ell_2$, and $\ell_1$—in learning compact, high-performing embeddings.
• To evaluate whether starting from raw distributional representations (bag-of-words) or neural network-based embeddings (skip-gram) yields better results for tailored models.
• To assess whether task-specific embeddings can outperform or refine existing general-purpose embeddings like skip-gram for specific lexical prediction tasks.

Proposed method

• The method uses a log-linear bilexical model that computes compatibility scores between query and candidate words via a bilinear form $\phi(q)^\top W \phi(c)$, where $\phi$ is a word representation and $W$ is a learnable matrix.
• The matrix $W$ is regularized using the nuclear norm ($\ell_\star$) to induce low-rank structure, enabling dimensionality reduction to $k$-dimensional task-specific embeddings via SVD.
• The model is trained in a semi-supervised manner: $\phi$ is fixed (from either bag-of-words or skip-gram), and $W$ is optimized using negative log-likelihood with regularization.
• Low-rank $W = UV^\top$ allows interpretation of $U^\top \phi(q)$ and $V^\top \phi(c)$ as $k$-dimensional task-specific embeddings for query and candidate words.
• The method compares three regularization schemes: $\ell_\star$, $\ell_2$, and $\ell_1$, and includes an unsupervised baseline where $W$ is the identity matrix.
• Optimization is performed using the FOBOS algorithm, and performance is evaluated on three bilexical relations: noun-adjective, verb-object, and verb-subject.

Experimental results

Research questions

• RQ1 Can task-specific word embeddings, learned via low-rank compression of pre-trained vectors, improve predictive accuracy for bilexical relations compared to general-purpose embeddings?
• RQ2 Does the nuclear norm ($\ell_\star$) regularization lead to better performance and more efficient embeddings than $\ell_2$ or $\ell_1$ regularization in bilexical prediction tasks?
• RQ3 Is there a performance gap between using raw bag-of-words representations and neural network-based skip-gram embeddings as the initial vector space for task-specific learning?
• RQ4 Can highly compressed embeddings (e.g., $k=5$) achieve performance close to that of full-rank models, and does this vary by linguistic relation?
• RQ5 To what extent do different bilexical relations exploit distinct lexical properties, and can task-specific embeddings capture these more effectively than general-purpose representations?

Key findings

• The nuclear norm ($\ell_\star$) regularization consistently outperformed $\ell_2$ and $\ell_1$ regularization in both predictive accuracy and model efficiency across all tested bilexical relations.
• For the noun-adjective relation, the $\ell_\star$-regularized model achieved 85.99% accuracy using only 80 dimensions (best $k$), outperforming the unsupervised baseline (85.12%) and other regularizers.
• Even with $k=5$, the $\ell_\star$ model achieved 83.99% accuracy on the noun-adjective relation, demonstrating that very low-dimensional embeddings can be highly predictive.
• For the object-verb relation, the $\ell_\star$-regularized model with skip-gram embeddings achieved 73.61% accuracy, significantly outperforming the unsupervised baseline (69.23%) and other regularizers.
• The method successfully compressed both bag-of-words (2,000D) and skip-gram (300D) representations, showing that even low-dimensional embeddings can be refined for better task-specific performance.
• Example outputs in Table 2 show that the same query word (e.g., 'city') retrieves different candidate words depending on the target relation, confirming that different relations exploit distinct lexical properties.