[Paper Review] Automated Analysis and Prediction of Job Interview Performance

Iftekhar Naim, Md. Iftekhar Tanveer+3 more|arXiv (Cornell University)|Apr 14, 2015

Emotion and Mood Recognition43 references16 citations

TL;DR

This paper presents a multimodal computational framework that automatically analyzes verbal, prosodic, and facial features in job interview videos to predict interview performance ratings. Trained on 138 MIT student interviews, the system predicts key traits like engagement and friendliness with correlation coefficients above 0.75 and identifies speech fluency, reduced filler words, and increased smiling as key success factors.

ABSTRACT

We present a computational framework for automatically quantifying verbal and nonverbal behaviors in the context of job interviews. The proposed framework is trained by analyzing the videos of 138 interview sessions with 69 internship-seeking undergraduates at the Massachusetts Institute of Technology (MIT). Our automated analysis includes facial expressions (e.g., smiles, head gestures, facial tracking points), language (e.g., word counts, topic modeling), and prosodic information (e.g., pitch, intonation, and pauses) of the interviewees. The ground truth labels are derived by taking a weighted average over the ratings of 9 independent judges. Our framework can automatically predict the ratings for interview traits such as excitement, friendliness, and engagement with correlation coefficients of 0.75 or higher, and can quantify the relative importance of prosody, language, and facial expressions. By analyzing the relative feature weights learned by the regression models, our framework recommends to speak more fluently, use less filler words, speak as "we" (vs. "I"), use more unique words, and smile more. We also find that the students who were rated highly while answering the first interview question were also rated highly overall (i.e., first impression matters). Finally, our MIT Interview dataset will be made available to other researchers to further validate and expand our findings.

Motivation & Objective

• To develop an automated, scalable framework for quantifying verbal and nonverbal behaviors in job interviews.
• To predict overall interview ratings and specific social traits (e.g., excitement, friendliness) using multimodal audio-visual data.
• To determine the relative importance of language, prosody, and facial expressions in influencing interview ratings.
• To generate actionable recommendations for improving interview performance based on data-driven insights.
• To release the MIT Interview dataset to support future research in automated interview analysis.

Proposed method

• Collected 138 job interview videos from 69 MIT undergraduates conducted with professional career counselors.
• Extracted multimodal features: lexical (word counts, topic modeling), prosodic (pitch, intonation, pauses), and facial (smiles, head gestures, facial tracking points).
• Used Mechanical Turk workers (9 per video) to generate ground truth ratings, aggregated via the EM algorithm to account for worker reliability.
• Trained support vector regression (SVR) models with linear, quadratic, cubic, and Gaussian kernels to predict overall and trait-specific ratings.
• Analyzed learned feature weights from regression models to infer relative importance and derive performance recommendations.
• Evaluated model performance using correlation coefficients (r) and AUC scores against baseline (AUC = 0.50).

Experimental results

Research questions

• RQ1 Can we automatically quantify verbal and nonverbal behaviors and assess their role in overall job interview ratings?
• RQ2 Can we build a computational framework to predict overall interview ratings from audio-visual recordings?
• RQ3 What is the relative importance of language, prosody, and facial expressions in predicting interview performance?
• RQ4 Can we generate automated, data-driven recommendations to improve social traits like excitement, friendliness, and engagement in interviews?

Key findings

• The framework predicted overall interview ratings with a correlation coefficient of r = 0.80, significantly outperforming the baseline AUC of 0.50.
• Key traits such as engagement, excitement, and friendliness were predicted with correlation coefficients of r ≥ 0.75 and AUC > 0.85.
• The system identified speaking more fluently, using fewer filler words, speaking as 'we' instead of 'I', using more unique words, and smiling more as top recommendations for improved performance.
• Interviewees who scored highly on the first question were consistently rated highly overall, indicating the strong impact of first impressions.
• Linear kernel SVR outperformed non-linear kernels, suggesting that the features exhibit relatively simple, linear relationships rather than complex interactions.
• The MIT Interview dataset, comprising 138 interview sessions, is publicly released to support further research in multimodal interview analysis.