[Paper Review] Exploring a New Competency Modeling Process with Large Language Models

Silin Du, Manqing Xin+1 more|arXiv (Cornell University)|Feb 13, 2026

Competency Development and Evaluation0 citations

TL;DR

This paper reformulates competency modeling by using large language models (LLMs) to extract behavioral and psychological descriptions from interviews, map them to a competency library via embeddings, learn adaptive weights, and evaluate the model offline without new data collection.

ABSTRACT

Competency modeling is widely used in human resource management to select, develop, and evaluate talent. However, traditional expert-driven approaches rely heavily on manual analysis of large volumes of interview transcripts, making them costly and prone to randomness, ambiguity, and limited reproducibility. This study proposes a new competency modeling process built on large language models (LLMs). Instead of merely automating isolated steps, we reconstruct the workflow by decomposing expert practices into structured computational components. Specifically, we leverage LLMs to extract behavioral and psychological descriptions from raw textual data and map them to predefined competency libraries through embedding-based similarity. We further introduce a learnable parameter that adaptively integrates different information sources, enabling the model to determine the relative importance of behavioral and psychological signals. To address the long-standing challenge of validation, we develop an offline evaluation procedure that allows systematic model selection without requiring additional large-scale data collection. Empirical results from a real-world implementation in a software outsourcing company demonstrate strong predictive validity, cross-library consistency, and structural robustness. Overall, our framework transforms competency modeling from a largely qualitative and expert-dependent practice into a transparent, data-driven, and evaluable analytical process.

Motivation & Objective

• Motivate the need for a rigorous, scalable competency modeling process in HRM beyond traditional expert-driven methods.
• Propose CoLLM, an LLM-based workflow to extract, embed, and integrate behavioral and psychological data from BEIs into a predefined competency library.
• Provide an offline evaluation framework to select model components and validate the approach without large-scale data collection.
• Demonstrate the method with real-world data from a software outsourcing company to show predictive validity and robustness.

Proposed method

• Decompose the traditional BEI workflow and use in-context learning (ICL) with LLMs to summarize behavioral and psychological descriptions from interview segments.
• Transform segment-level outputs into embeddings for both behavioral and psychological descriptions, and map them to embeddings of competency library descriptions.
• Introduce a learnable weight alpha to adaptively combine behavioral and psychological scores at the group level via an objective inspired by triplet loss.
• Define an offline evaluation procedure that uses train/test splits and Spearman rank correlation (and AUC) to select the number of key competencies and assess predictive validity.
• Optimize the weighting parameter alpha with stochastic gradient descent using a triplet loss that encourages within-group similarity and between-group differences.
• Apply an offline evaluation that selects top key competencies based on training data and assesses rankings against actual performance using Spearman’s rho (and AUC when binary labels are used).

Experimental results

Research questions

• RQ1 Can CoLLM produce a coherent and interpretable competency model for a target role?
• RQ2 How to determine the optimal number of key competencies for robust performance separation?
• RQ3 How robust is the modeling process to changes in the underlying LLM or when compared to human expert coding?
• RQ4 Are identified key competencies consistent across different competency libraries?

Key findings

• CoLLM yields interpretable competency scores by integrating behavioral and psychological signals.
• An offline evaluation framework enables data-driven selection of the number of key competencies (Q) and validates model performance without new data collection.
• The approach shows alignment with expert judgments and can adapt to different LLMs and competency libraries in the reported setting.
• Empirical results from 40 BEIs (split into high- and average-performance groups) demonstrate the framework’s predictive validity and robustness.
• A learnable weight alpha captures the relative importance of psychological versus behavioral data for the target context, as shown by optimized alpha values (e.g., alpha = 12.23 in the example).
• The seven most critical competencies identified in the study align with expert analysis for the tested group.