[Paper Review] CompLex: A New Corpus for Lexical Complexity Prediction from Likert Scale Data

Matthew Shardlow, M. Cooper+1 more|arXiv (Cornell University)|Mar 16, 2020

Text Readability and Simplification13 references33 citations

TL;DR

CompLex provides the first English corpus for continuous lexical complexity prediction using a 5-point Likert scale across Bible, Europarl, and biomedical texts, with baseline predictions showing promising MAE results.

ABSTRACT

Predicting which words are considered hard to understand for a given target population is a vital step in many NLP applications such as text simplification. This task is commonly referred to as Complex Word Identification (CWI). With a few exceptions, previous studies have approached the task as a binary classification task in which systems predict a complexity value (complex vs. non-complex) for a set of target words in a text. This choice is motivated by the fact that all CWI datasets compiled so far have been annotated using a binary annotation scheme. Our paper addresses this limitation by presenting the first English dataset for continuous lexical complexity prediction. We use a 5-point Likert scale scheme to annotate complex words in texts from three sources/domains: the Bible, Europarl, and biomedical texts. This resulted in a corpus of 9,476 sentences each annotated by around 7 annotators.

Motivation & Objective

• Motivate lexical complexity prediction as a continuous task rather than binary classification.
• Create a diverse English corpus annotated with Likert-scale complexity scores.
• Analyze annotation quality and inter-genre differences in lexical complexity.
• Provide baseline predictive models to establish feasibility for continuous complexity prediction.

Proposed method

• Develop a 5-point Likert scale annotation scheme for lexical complexity (1=Very Easy to 5=Very Difficult).
• Collect 9,476 sentences from Bible, Europarl, and biomedical domains and annotate targets (single words and MWEs) using crowdsourcing.
• Aggregate annotations into a normalized 0–1 complexity score (1->0, 2->0.25, 3->0.5, 4->0.75, 5->1).
• Use 20 annotations per instance, with quality controls and discard annotations with insufficient participation (keep at least 4 valid annotations).
• Train a baseline linear regression using a mix of features: 300-dim GloVe word embeddings, 4,096-dim InferSent context embeddings, and hand-crafted features (frequency, length, syllables).
• Evaluate on a held-out 10% test set and report mean absolute error (MAE).

Experimental results

Research questions

• RQ1 Can lexical complexity be effectively modeled as a continuous attribute rather than a binary label?
• RQ2 How do domain genres (Bible, Europarl, Biomed) differ in lexical complexity and modelability?
• RQ3 What is the predictive value of word-level embeddings versus context embeddings and hand-crafted features for LCP?
• RQ4 Do multi-word expressions (MWEs) differ in complexity from single words across domains?
• RQ5 What baseline performance can linear models achieve on CompLex using standard lexical features and embeddings?

Key findings

• The corpus comprises 9,476 sentences with average complexity around 0.395 (SD 0.115).
• Single words average 0.385 complexity, MWEs average 0.444, with domain differences (Biomed higher complexity).
• Baseline linear regression with hand-crafted features achieved MAE 0.0853 on the best setup.
• Word embeddings alone yield MAE 0.0875, comparable to hand-crafted features; sentence embeddings were less predictive (MAE not improved).
• Europarl generally yielded lower MAE (0.0801) than Bible (0.6648) and Biomed (0.2954) in some configurations, indicating domain-specific predictability.
• Contextual embeddings did not consistently improve predictions in this baseline setup, suggesting potential gains with alternative models.