[Paper Review] Unsupervised Domain Adaptation with Feature Embeddings

Yi Yang, Jacob Eisenstein|arXiv (Cornell University)|Dec 14, 2014

Domain Adaptation and Few-Shot Learning13 references3 citations

TL;DR

This paper proposes Fema, a novel unsupervised domain adaptation method that learns dense feature embeddings using a skip-gram model with negative sampling, exploiting template-structured features in NLP. It outperforms prior representation learning methods like SCL and mDA by 1% on average in POS tagging adaptation from news to web text, achieving state-of-the-art results without requiring task-specific pivot feature selection.

ABSTRACT

Representation learning is the dominant technique for unsupervised domain adaptation, but existing approaches often require the specification of "pivot features" that generalize across domains, which are selected by task-specific heuristics. We show that a novel but simple feature embedding approach provides better performance, by exploiting the feature template structure common in NLP problems.

Motivation & Objective

• To address the limitations of existing unsupervised domain adaptation methods that rely on task-specific heuristics for selecting pivot features.
• To improve cross-domain generalization in NLP by learning dense, low-dimensional feature embeddings that capture co-occurrence patterns across feature templates.
• To eliminate the computational trade-off between using extensive feature co-occurrence statistics and maintaining efficient representation learning.
• To provide a simple, scalable, and effective alternative to reconstruction-based representation learning methods in structured feature spaces.

Proposed method

• Fema learns distinct input and output embeddings for each feature using a skip-gram model trained with negative sampling on the union of source and target domain data.
• The model predicts active features in neighboring templates using the input embedding of a feature, optimizing the objective function with sigmoid activation and negative sampling.
• Feature embeddings are concatenated across templates to form a dense representation for each instance, which is then combined with the original feature vector via concatenation.
• A hyperbolic tangent function is applied to the concatenated embeddings to introduce nonlinearity and improve representation robustness.
• The final augmented feature representation combines raw features and embedded features, enabling downstream models like SVMs to generalize better across domains.
• The method leverages the template structure common in NLP feature spaces, where each template contributes exactly one active feature per instance.

Experimental results

Research questions

• RQ1 Can a representation learning approach based on feature embeddings outperform pivot-feature reconstruction methods in unsupervised domain adaptation?
• RQ2 Does learning embeddings directly from feature co-occurrence patterns improve cross-domain generalization in POS tagging?
• RQ3 Can a skip-gram-based method with negative sampling be effectively applied to structured feature templates in NLP?
• RQ4 How does Fema compare in performance and efficiency to SCL, mDA, and word2vec in domain adaptation tasks?

Key findings

• Fema achieves an average accuracy of 92.60% on the SANCL test set across all target domains, outperforming the baseline by 4.15 percentage points.
• On the dev set, Fema achieves 91.26% accuracy on newsgroups, 92.82% on reviews, 94.95% on weblogs, 90.69% on answers, and 89.72% on emails, surpassing all competitive methods except FLORS on reviews.
• Fema improves upon SCL and mDA by approximately 1% on average, corresponding to a 10% relative error reduction in performance.
• The method trains in about 70 minutes on a 24-core machine using gensim, making it faster than SCL and competitive with mDA.
• Fema eliminates the need for task-specific pivot feature selection, simplifying the adaptation pipeline and reducing heuristic dependency.