[Paper Review] Resume Evaluation through Latent Dirichlet Allocation and Natural Language Processing for Effective Candidate Selection

Vidhita Jagwani, Smit Meghani+2 more|arXiv (Cornell University)|Jul 28, 2023

Topic Modeling4 citations

TL;DR

This paper proposes a content-driven resume evaluation system using Latent Dirichlet Allocation (LDA) and SpaCy-based Named Entity Recognition (NER) to extract and analyze education, experience, and skills. By modeling resume entities as topics, the method achieves 82% accuracy in overall resume scoring, outperforming keyword-based approaches through semantic topic modeling and entity-aware representation learning.

ABSTRACT

In this paper, we propose a method for resume rating using Latent Dirichlet Allocation (LDA) and entity detection with SpaCy. The proposed method first extracts relevant entities such as education, experience, and skills from the resume using SpaCy's Named Entity Recognition (NER). The LDA model then uses these entities to rate the resume by assigning topic probabilities to each entity. Furthermore, we conduct a detailed analysis of the entity detection using SpaCy's NER and report its evaluation metrics. Using LDA, our proposed system breaks down resumes into latent topics and extracts meaningful semantic representations. With a vision to define our resume score to be more content-driven rather than a structure and keyword match driven, our model has achieved 77% accuracy with respect to only skills in consideration and an overall 82% accuracy with all attributes in consideration. (like college name, work experience, degree and skills)

Motivation & Objective

• To develop a resume scoring system that prioritizes content semantics over keyword matching or structural formatting.
• To improve candidate selection accuracy by modeling resumes as latent topics using LDA on extracted entities.
• To evaluate the performance of SpaCy's NER for entity extraction in resumes, focusing on education, experience, and skills.
• To establish a content-driven resume score that reflects semantic relevance and completeness.
• To achieve higher accuracy than traditional keyword-based or structure-dependent resume screening methods.

Proposed method

• Named Entity Recognition (NER) using SpaCy to extract key resume entities such as education, work experience, degrees, and skills.
• Construction of a latent topic model using Latent Dirichlet Allocation (LDA) on the extracted entities to represent resumes as topic distributions.
• Topic probability assignment per entity to quantify semantic relevance and content richness in the resume.
• Integration of multiple entity types—college names, degrees, work history, and skills—into a unified scoring framework.
• Application of LDA to generate semantic representations of resumes based on entity co-occurrence and distribution.
• Scoring system designed to be content-driven, minimizing reliance on keyword frequency or formatting.

Experimental results

Research questions

• RQ1 How accurately can LDA and NER-based entity extraction model the semantic content of resumes for candidate evaluation?
• RQ2 To what extent does a content-driven approach outperform keyword- or structure-based resume scoring?
• RQ3 What is the performance of SpaCy’s NER in extracting education, experience, and skill entities from real-world resumes?
• RQ4 How does incorporating multiple resume attributes (education, experience, skills) affect overall scoring accuracy?
• RQ5 Can topic modeling of resume entities yield a more meaningful and accurate representation of candidate qualifications?

Key findings

• The proposed system achieved 82% overall accuracy in resume scoring when considering all attributes: education, experience, degree, and skills.
• When evaluating only skills, the model achieved 77% accuracy, demonstrating strong performance on a core component of resume evaluation.
• SpaCy’s NER was evaluated in detail, with reported metrics provided for entity detection, though specific values are not included in the abstract.
• The use of LDA enabled effective semantic representation of resumes by identifying latent topics from structured entity inputs.
• The model successfully reduced dependency on keyword matching and structural formatting, favoring semantic content analysis.
• The results indicate that topic modeling on extracted entities improves candidate selection accuracy over conventional methods.