[Paper Review] Model Selection for Cross-Lingual Transfer using a Learned Scoring Function

Yang Chen, Alan Ritter|arXiv (Cornell University)|May 4, 2021

Natural Language Processing Techniques4 citations

TL;DR

This paper proposes a learned scoring function that uses internal representations from fine-tuned multilingual models to select superior cross-lingual transfer models without relying on target-language validation data. By leveraging small amounts of annotated data in a pivot language, the method outperforms selection based on English validation data across five languages and five NLP tasks, achieving performance comparable to using small target-language development sets.

ABSTRACT

Transformers that are pre-trained on multilingual text corpora, such as, mBERT and XLM-RoBERTa, have achieved impressive cross-lingual transfer learning results. In the zero-shot cross-lingual transfer setting, only English training data is assumed, and the fine-tuned model is evaluated on another target language. No target-language validation data is assumed in this setting, however substantial variance has been observed in target language performance between different fine-tuning runs. Prior work has relied on English validation/development data to select among models that are fine-tuned with different learning rates, number of steps and other hyperparameters, often resulting in suboptimal choices. In this paper, we show that it is possible to select consistently better models when small amounts of annotated data are available in an auxiliary pivot language. We propose a machine learning approach to model selection that uses the fine-tuned model's own internal representations to predict its cross-lingual capabilities. In extensive experiments we find that our approach consistently selects better models than English validation data across five languages and five well-studied NLP tasks, achieving results that are comparable to small amounts of target language development data.

Motivation & Objective

• To address the high variance in zero-shot cross-lingual transfer performance across different fine-tuning runs when no target-language data is available.
• To reduce reliance on English validation data for hyperparameter selection, which often leads to suboptimal model choices.
• To improve model selection consistency and performance in zero-shot cross-lingual transfer by utilizing small amounts of annotated data in an auxiliary pivot language.
• To develop a machine learning-based method that predicts a model's cross-lingual capabilities using its internal representations.

Proposed method

• The method trains a scoring function that maps a fine-tuned model's internal representations to a predicted cross-lingual performance score.
• The scoring function is trained on a small set of annotated examples in a pivot language, using the model's representations from the final layer as input.
• The model's representations are extracted from the fine-tuned multilingual encoder (e.g., mBERT or XLM-RoBERTa) during inference on pivot language data.
• The learned scoring function is optimized using a regression objective to predict the model's performance on downstream tasks in the target language.
• The method selects the best-performing model based on the predicted score, avoiding the need for target-language validation data.
• The approach is evaluated across multiple languages and NLP tasks, comparing selection performance against English validation and small target-language development sets.

Experimental results

Research questions

• RQ1 Can a learned scoring function based on internal model representations improve model selection in zero-shot cross-lingual transfer?
• RQ2 Does using small amounts of pivot language data lead to better model selection than relying on English validation data?
• RQ3 How does the performance of model selection using the proposed method compare to using small target-language development sets?
• RQ4 To what extent does the method generalize across different languages and NLP tasks?
• RQ5 Can internal representations alone reliably predict a model's cross-lingual transfer capability?

Key findings

• The proposed method consistently selects better models than selection based on English validation data across five target languages and five NLP tasks.
• The method achieves performance comparable to using small amounts of target-language development data, significantly improving selection reliability.
• Using a pivot language with minimal annotated data enables effective model selection even in the absence of target-language validation data.
• The learned scoring function generalizes well across diverse NLP tasks, including text classification, sequence labeling, and natural language inference.
• The approach reduces variance in model selection outcomes, leading to more consistent and reliable performance in zero-shot cross-lingual transfer.