[Paper Review] Predicting the Effectiveness of Self-Training: Application to Sentiment Classification

Vincent Van Asch, Walter Daelemans|arXiv (Cornell University)|Jan 13, 2016

Topic Modeling21 references15 citations

TL;DR

This paper proposes using corpus similarity measures—particularly test-train similarity—to predict whether self-training will improve sentiment classification performance. By leveraging similarity scores (e.g., Jensen-Shannon divergence), the method predicts self-training gain with high precision, showing that test/train similarity alone is sufficient and often optimal for reliable prediction.

ABSTRACT

The goal of this paper is to investigate the connection between the performance gain that can be obtained by selftraining and the similarity between the corpora used in this approach. Self-training is a semi-supervised technique designed to increase the performance of machine learning algorithms by automatically classifying instances of a task and adding these as additional training material to the same classifier. In the context of language processing tasks, this training material is mostly an (annotated) corpus. Unfortunately self-training does not always lead to a performance increase and whether it will is largely unpredictable. We show that the similarity between corpora can be used to identify those setups for which self-training can be beneficial. We consider this research as a step in the process of developing a classifier that is able to adapt itself to each new test corpus that it is presented with.

Motivation & Objective

• To identify conditions under which self-training improves performance in sentiment classification, especially when training and test data distributions differ.
• To investigate whether corpus similarity can serve as a reliable predictor of self-training success or failure.
• To evaluate whether selecting a single, most similar unlabeled corpus is better than including all available unlabeled data.
• To assess the utility of similarity measures (e.g., cosine, KL divergence, JS divergence) in predicting self-training outcomes.
• To support the development of self-adapting classifiers capable of selecting optimal data for domain adaptation.

Proposed method

• Uses a two-step self-training procedure: train on labeled data, predict labels on unlabeled data, then retrain on combined labeled and pseudo-labeled data.
• Employs multiple similarity measures—cosine distance, Kullback-Leibler divergence, and Jensen-Shannon divergence—to quantify similarity between training, test, and additional (unlabeled) corpora.
• Applies supervised and unsupervised prediction models to forecast self-training performance based on similarity features.
• Conducts leave-one-out experiments using only test/train similarity as a feature to evaluate predictive power.
• Uses macro-averaged F-score and precision on 'gain' predictions (POS class) to evaluate model performance.
• Visualizes corpus relationships in vector space using similarity measures to support interpretability.

Experimental results

Research questions

• RQ1 Can corpus similarity predict whether self-training will lead to performance gain in sentiment classification?
• RQ2 Is the similarity between the test and training corpora a more informative predictor than similarities involving additional unlabeled data?
• RQ3 Does using multiple similarity measures improve the prediction of self-training effectiveness compared to using only test/train similarity?
• RQ4 Which similarity measure (e.g., cosine, KL, JS divergence) yields the most accurate prediction of self-training gain?
• RQ5 Can a classifier automatically select the most beneficial unlabeled corpus for self-training based on similarity to the test data?

Key findings

• Test/train similarity alone is highly informative and often sufficient for predicting self-training gain, outperforming models using multiple similarity features.
• The Jensen-Shannon divergence consistently outperforms other measures in terms of precision and specificity for predicting self-training success.
• Using three similarity features (test/train, test/additional, additional/train) increases the number of predicted gain cases (true positives), but may also increase false positives, especially with Kullback-Leibler divergence.
• For the cosine distance, adding extra features leads to more false positives than true positives, reducing the macro-averaged F-score.
• When no prior data exists for a given test/train combination, similarity between test and training corpora is the most reliable predictor of self-training outcome.
• The results suggest that domain similarity is a critical factor in domain adaptation, and a standardized similarity measure could greatly improve objective evaluation of adaptation techniques.