[Paper Review] Language Modeling by Clustering with Word Embeddings for Text Readability Assessment

Miriam Cha, Youngjune Gwon+1 more|arXiv (Cornell University)|Sep 5, 2017

Text Readability and Simplification15 references3 citations

TL;DR

This paper proposes a clustering-based language model using word embeddings to improve text readability prediction. By applying K-means clustering to pre-trained fastText word vectors and representing documents as histogram features, the method achieves state-of-the-art Spearman correlation of 0.83 and Pearson correlation of 0.82 on the Common Core Standards corpus, outperforming prior approaches using shallow features or standard embeddings.

ABSTRACT

We present a clustering-based language model using word embeddings for text readability prediction. Presumably, an Euclidean semantic space hypothesis holds true for word embeddings whose training is done by observing word co-occurrences. We argue that clustering with word embeddings in the metric space should yield feature representations in a higher semantic space appropriate for text regression. Also, by representing features in terms of histograms, our approach can naturally address documents of varying lengths. An empirical evaluation using the Common Core Standards corpus reveals that the features formed on our clustering-based language model significantly improve the previously known results for the same corpus in readability prediction. We also evaluate the task of sentence matching based on semantic relatedness using the Wiki-SimpleWiki corpus and find that our features lead to superior matching performance.

Motivation & Objective

• To improve text readability assessment by leveraging semantic features from word embeddings rather than relying on shallow linguistic features.
• To investigate whether clustering word vectors in a semantic space yields more discriminative features for text regression tasks.
• To evaluate the effectiveness of the proposed method on both readability prediction and sentence matching for semantic similarity.
• To demonstrate that histogram-based representations from clustered word embeddings can handle variable-length documents naturally.

Proposed method

• The method uses fastText word embeddings trained on the Common Core Standards corpus to represent words in a high-dimensional semantic space.
• K-means clustering is applied to word vectors to group semantically similar words into clusters, forming a cluster-based language model.
• Document-level features are created by counting cluster occurrences per document, forming histogram representations that accommodate variable-length texts.
• The histogram features are used as input to a regression model for predicting text readability levels.
• For sentence matching, sentence embeddings are generated by averaging word embeddings in a sentence, and semantic similarity is evaluated via nearest-neighbor search.
• The approach is evaluated on two benchmarks: the Common Core Standards corpus for readability and the Wiki-SimpleWiki corpus for semantic sentence matching.

Experimental results

Research questions

• RQ1 Can clustering word embeddings in a semantic space yield better features for text readability prediction than traditional shallow features or raw embeddings?
• RQ2 Does a histogram representation of cluster frequencies improve performance on variable-length documents in readability assessment?
• RQ3 Can the proposed clustering-based language model effectively capture semantic similarity between sentences with different vocabulary and grammar?
• RQ4 How does the performance of the proposed method compare to state-of-the-art approaches on the Common Core Standards corpus?

Key findings

• The proposed method achieved a Spearman correlation of 0.83 and a Pearson correlation of 0.82 on the Common Core Standards corpus, surpassing previous state-of-the-art results.
• K-means clustering on fastText word vectors outperformed both Brown clustering and baseline embedding models, with the best performance when word vectors were fine-tuned on the target corpus.
• The method achieved a 95.9% probability that a Wikipedia sentence and its SimpleWiki counterpart are among the 4 nearest neighbors in the semantic space, indicating strong semantic preservation.
• The use of fastText embeddings with K-means clustering significantly improved readability prediction over bag-of-words, word2vec, and doc2vec baselines.
• The histogram-based feature representation effectively handled documents of varying lengths, enabling robust performance across diverse text lengths.