[Paper Review] Augmenting Holistic Review in University Admission using Natural Language Processing for Essays and Recommendation Letters

Jinsook Lee, Bradon Thymes+3 more|arXiv (Cornell University)|Jun 30, 2023

Online Learning and Analytics4 citations

TL;DR

This study evaluates whether natural language processing (NLP) of essays and recommendation letters can compensate for the exclusion of protected attributes—such as race and gender—in machine learning models predicting university admissions. Using data from 14,915 applicants, it finds that while textual features improve model performance slightly and help restore gender diversity, they fail to recover the significant drop in underrepresented minorit (URM) representation when demographic data is omitted, indicating NLP cannot fully substitute for protected attributes in maintaining diversity.

ABSTRACT

University admission at many highly selective institutions uses a holistic review process, where all aspects of the application, including protected attributes (e.g., race, gender), grades, essays, and recommendation letters are considered, to compose an excellent and diverse class. In this study, we empirically evaluate how influential protected attributes are for predicting admission decisions using a machine learning (ML) model, and in how far textual information (e.g., personal essay, teacher recommendation) may substitute for the loss of protected attributes in the model. Using data from 14,915 applicants to an undergraduate admission office at a selective U.S. institution in the 2022-2023 cycle, we find that the exclusion of protected attributes from the ML model leads to substantially reduced admission-prediction performance. The inclusion of textual information via both a TF-IDF representation and a Latent Dirichlet allocation (LDA) model partially restores model performance, but does not appear to provide a full substitute for admitting a similarly diverse class. In particular, while the text helps with gender diversity, the proportion of URM applicants is severely impacted by the exclusion of protected attributes, and the inclusion of new attributes generated from the textual information does not recover this performance loss.

Motivation & Objective

• To assess the impact of excluding protected attributes (e.g., race, gender) on machine learning models predicting university admissions.
• To investigate whether textual features from essays and recommendation letters can compensate for the loss of demographic data in maintaining prediction accuracy and demographic diversity.
• To evaluate the extent to which NLP techniques can support holistic review processes in the absence of protected attributes.
• To understand the limitations of NLP in recovering diversity when demographic features are unavailable in admission prediction models.
• To inform policy and practice on algorithmic fairness in university admissions under potential legal constraints on using protected attributes.

Proposed method

• The study uses a machine learning model trained on 14,915 applicant records from a selective U.S. university’s 2022–2023 admission cycle.
• Feature sets include demographic attributes (gender, race/ethnicity), academic performance (grades, honors), extracurriculars, and textual data (essays, writing supplements, recommendation letters).
• Textual features are processed using TF-IDF and Latent Dirichlet Allocation (LDA) for topic modeling to extract semantic representations.
• Model performance is evaluated using AUC (Area Under the Curve) for admission prediction, with diversity analyzed in the top decile of predicted applicants.
• The study compares models with and without protected attributes, and with varying combinations of textual features, to isolate their effects on performance and diversity.
• Statistical significance of performance differences is assessed using p-values, with a threshold of p < 0.05 for significance.

Experimental results

Research questions

• RQ1 To what extent does excluding protected attributes (e.g., race, gender) degrade the performance of a machine learning model in predicting university admissions?
• RQ2 Can textual features from personal essays and recommendation letters compensate for the loss of protected attributes in maintaining model performance and demographic diversity?
• RQ3 How does the inclusion of textual features affect the representation of underrepresented minorities (URMs) in the top-ranked pool of applicants when demographic features are excluded?
• RQ4 Does the model’s ability to predict gender diversity improve when textual data is included, especially in the absence of gender as a direct feature?
• RQ5 To what extent do NLP techniques such as TF-IDF and LDA capture socio-demographic signals that could substitute for protected attributes in admission prediction?

Key findings

• Excluding protected attributes from the model leads to a substantial drop in admission-prediction performance, indicating their critical role in accurate decision-making.
• The inclusion of textual features via TF-IDF and LDA partially restores model performance, but the improvement is minimal—AUC increases from 0.884 to 0.892, which is not statistically significant (p = 0.303).
• While textual features help recover gender diversity when demographic data is excluded, they fail to restore the proportion of underrepresented minority (URM) applicants, which drops significantly.
• The proportion of URM applicants remains relatively constant across models that exclude demographic features, indicating no meaningful recovery through text alone.
• Recommendation letters, due to explicit use of gender pronouns, are particularly effective at recovering gender diversity when gender is not used as a feature.
• The study concludes that NLP-based textual analysis cannot fully substitute for protected attributes in maintaining demographic diversity, especially for URMs, in holistic admission processes.