[Paper Review] SentEval: An Evaluation Toolkit for Universal Sentence Representations

Alexis Conneau, Douwe Kiela|arXiv (Cornell University)|Mar 14, 2018

Topic ModelingComputer Science35 references343 citations

TL;DR

SentEval provides a centralized Python toolkit and dataset pipeline to evaluate universal sentence representations across a wide set of transfer tasks, including classification, NLI, and semantic similarity. It standardizes preprocessing and hyperparameters to enable fair comparisons.

ABSTRACT

We introduce SentEval, a toolkit for evaluating the quality of universal sentence representations. SentEval encompasses a variety of tasks, including binary and multi-class classification, natural language inference and sentence similarity. The set of tasks was selected based on what appears to be the community consensus regarding the appropriate evaluations for universal sentence representations. The toolkit comes with scripts to download and preprocess datasets, and an easy interface to evaluate sentence encoders. The aim is to provide a fairer, less cumbersome and more centralized way for evaluating sentence representations.

Motivation & Objective

• Motivate the need for fair, centralized evaluation of universal sentence representations.
• Provide a standardized evaluation pipeline with fixed hyperparameters to enable reproducible comparisons.
• Offer an easy-to-use toolkit with data download/preprocessing and examples to evaluate encoders.

Proposed method

• Introduce SentEval as a Python-based evaluation toolkit for universal sentence encoders.
• Define a central, community-aligned set of evaluation tasks spanning classification, NLI, and semantic similarity.
• Describe the evaluation protocol: fixed hyperparameters, downstream transfer tasks, and cosine similarity for certain STS tasks.
• Provide a user interface with prepare and batcher functions to plug in any encoder and output sentence embeddings.
• Detail how standard classifiers (Logistic Regression or MLP) are trained on top of embeddings for transfer tasks.
• Explain the dataset acquisition script and preprocessing steps (MOSES tokenizer, UTF-8 conversion).

Experimental results

Research questions

• RQ1 What constitutes a fair, centralized suite of tasks to evaluate universal sentence representations?
• RQ2 How do different encoders perform across a broad transfer- and similarity-based evaluation space?
• RQ3 What are the practical requirements and setup steps to reproduce SentEval results across models?
• RQ4 How does SentEval enable fair comparison by fixing hyperparameters and preprocessing across methods?

Key findings

• SentEval demonstrates a broad evaluation regime covering binary/multi-class classification, NLI, and semantic similarity tasks.
• The toolkit provides a unified interface (prepare and batcher) to plug in any encoder and run standardized evaluations.
• Baseline results show varying transfer performance across methods (e.g., GloVe, fastText, SkipThought, InferSent) under fixed evaluation settings.
• The evaluation framework highlights how supervised, task-specific models can outperform transfer methods on some benchmarks, while universal representations aim for broad generalization.
• The authors provide practical guidance for reproducible evaluation, including a CLI script for data preparation and detailed parameter defaults.
• SentEval emphasizes the need for fair benchmarking to encourage development of generalizable sentence representations.