[Paper Review] Beyond Accuracy Optimization: On the Value of Item Embeddings for Student Job Recommendations

Emanuel Lacić, Dominik Kowald+3 more|arXiv (Cornell University)|Nov 21, 2017

Topic Modeling19 references3 citations

TL;DR

This paper proposes using neural item embeddings combined with the Base-Level Learning (BLL) equation to improve job recommendations for university students, balancing accuracy, diversity, and novelty. By integrating frequency and recency of interactions, the BLL-based approach achieves the best novelty performance while maintaining high accuracy and diversity, outperforming standard content-based and collaborative filtering methods.

ABSTRACT

In this work, we address the problem of recommending jobs to university students. For this, we explore the utilization of neural item embeddings for the task of content-based recommendation, and we propose to integrate the factors of frequency and recency of interactions with job postings to combine these item embeddings. We evaluate our job recommendation system on a dataset of the Austrian student job portal Studo using prediction accuracy, diversity and an adapted novelty metric. This paper demonstrates that utilizing frequency and recency of interactions with job postings for combining item embeddings results in a robust model with respect to accuracy and diversity, which also provides the best adapted novelty results.

Motivation & Objective

• To improve job recommendation systems for university students beyond just accuracy, focusing on diversity and novelty.
• To investigate how frequency and recency of user interactions with job postings affect recommendation quality.
• To develop and evaluate a novel adaptation of the BLL equation from human memory theory for combining item embeddings in content-based recommendation.
• To propose and validate an adapted novelty metric that reflects real user behavior (e.g., application patterns) rather than global popularity.
• To demonstrate that combining BLL-based embeddings with collaborative filtering yields a robust recommendation model.

Proposed method

• Employing Doc2Vec to learn dense, semantic item embeddings from job descriptions in an unsupervised manner.
• Using three strategies to combine item embeddings: LAST (most recent interaction), AVG (average of past interactions), and BLL (frequency and recency-weighted combination via the BLL equation).
• Applying the BLL equation to model user memory dynamics, where recency and frequency of interactions are weighted to compute a dynamic user preference vector.
• Introducing an adapted novelty metric, Novelty*, defined as 1 - |N_A - N@k|, where N_A = 0.58 is the average novelty of actual job applications.
• Evaluating the system using a real-world dataset from the Austrian student job portal Studo, with k=3 and k=6 recommended jobs.
• Combining the best-performing BLL-based item embeddings with user-based collaborative filtering in a round-robin fashion to enhance overall performance.

Experimental results

Research questions

• RQ1 Can item embeddings improve job recommendation performance beyond accuracy, particularly in terms of diversity and novelty?
• RQ2 How do frequency and recency of user interactions with job postings affect the quality of content-based recommendations?
• RQ3 Does integrating the BLL equation from human memory theory into item embedding combination lead to better-balanced recommendations?
• RQ4 Can an adapted novelty metric that reflects real user application behavior provide more meaningful evaluation than standard popularity-based novelty measures?
• RQ5 Does combining BLL-based item embeddings with collaborative filtering yield a more robust recommendation system?

Key findings

• The BLL-based approach achieved the best performance on the adapted novelty metric (Novelty*), closely matching the target novelty value of 0.58, which represents the average novelty of actual student applications.
• Using the BLL equation resulted in slightly lower accuracy than the LAST strategy but significantly better diversity than the CBF method, while outperforming all other methods on the Novelty* metric.
• The AVG strategy achieved the highest diversity but at the cost of reduced accuracy, indicating a trade-off between these metrics.
• Increasing embedding dimensions to 200 or 300 led to higher novelty scores, which worsened the Novelty* performance, suggesting that higher dimensions may overemphasize tail items.
• Combining the best BLL-based embeddings with collaborative filtering produced a robust model that achieved high accuracy, good diversity, and nearly optimal novelty (Novelty* ≈ 0.01 deviation from target).
• The results demonstrate that incorporating human memory principles (via BLL) into embedding combination improves recommendation quality in real-world settings where user satisfaction depends on balanced diversity and relevance.