[Paper Review] Hierarchical Attention Network for Visually-aware Food Recommendation

Xiaoyan Gao, Fuli Feng+6 more|arXiv (Cornell University)|Oct 11, 2018

Recommender Systems and Techniques54 references6 citations

TL;DR

This paper proposes Hierarchical Attention-based Food Recommendation (HAFR), a neural network model that jointly models user-item interactions, recipe ingredients, and visual features from food images using a hierarchical attention mechanism. HAFR achieves a 12% average improvement over state-of-the-art methods like Factorization Machine and Visual Bayesian Personalized Ranking on a large-scale food recommendation dataset.

ABSTRACT

Food recommender systems play an important role in assisting users to identify the desired food to eat. Deciding what food to eat is a complex and multi-faceted process, which is influenced by many factors such as the ingredients, appearance of the recipe, the user's personal preference on food, and various contexts like what had been eaten in the past meals. In this work, we formulate the food recommendation problem as predicting user preference on recipes based on three key factors that determine a user's choice on food, namely, 1) the user's (and other users') history; 2) the ingredients of a recipe; and 3) the descriptive image of a recipe. To address this challenging problem, we develop a dedicated neural network based solution Hierarchical Attention based Food Recommendation (HAFR) which is capable of: 1) capturing the collaborative filtering effect like what similar users tend to eat; 2) inferring a user's preference at the ingredient level; and 3) learning user preference from the recipe's visual images. To evaluate our proposed method, we construct a large-scale dataset consisting of millions of ratings from AllRecipes.com. Extensive experiments show that our method outperforms several competing recommender solutions like Factorization Machine and Visual Bayesian Personalized Ranking with an average improvement of 12%, offering promising results in predicting user preference for food. Codes and dataset will be released upon acceptance.

Motivation & Objective

• To address the limitations of existing food recommender systems that treat recipes as atomic items and neglect multi-faceted influences on user preference.
• To jointly model user-item interactions, recipe ingredients, and visual semantics from food images to capture complex preference patterns.
• To develop a unified deep learning framework that captures collaborative filtering effects, ingredient-level preferences, and image-based visual cues.
• To construct and release a large-scale, real-world dataset for food recommendation research.
• To enable more accurate and personalized food recommendations by integrating multimodal and sequential information.

Proposed method

• HAFR employs a hierarchical attention mechanism to first attend over individual ingredients and then over user interaction histories, capturing user preference at the ingredient level.
• The model uses a CNN-based feature extractor to encode recipe images into visual embeddings, which are then fused with ingredient and interaction representations.
• A dual-attention mechanism is applied: one at the ingredient level to weigh the importance of each ingredient, and another at the user interaction level to aggregate past behaviors.
• User and item representations are learned through a multi-modal fusion of collaborative filtering signals, ingredient embeddings, and visual features using attention gates.
• The model is trained end-to-end using a ranking loss function to optimize for user preference prediction.
• The hierarchical structure allows the model to dynamically weigh the contribution of ingredients and images based on user-specific preferences.

Experimental results

Research questions

• RQ1 How can we effectively model user preference on food by integrating user interaction history, recipe ingredients, and visual image features?
• RQ2 To what extent does hierarchical attention improve the representation of recipes compared to flat or non-attentive models?
• RQ3 Can jointly modeling ingredients and visual features lead to better recommendation performance than modeling them in isolation?
• RQ4 How do collaborative filtering effects interact with content-based features (ingredients and images) in predicting user preferences?
• RQ5 Does the proposed HAFR model generalize across diverse user tastes, especially when preferences are driven by specific ingredients or visual appeal?

Key findings

• HAFR achieves a 12% average improvement in recommendation performance over baseline models such as Factorization Machine and Visual Bayesian Personalized Ranking on the constructed large-scale dataset.
• Ablation studies confirm that the hierarchical attention mechanism significantly enhances performance by effectively aggregating information across ingredients and user interactions.
• The integration of visual features from food images leads to a measurable improvement in recommendation accuracy, highlighting the importance of visual semantics in food choice.
• Model variants that exclude ingredients or visual features show substantial performance drops, demonstrating the complementary value of all three input modalities.
• The model effectively captures user-specific preferences, such as a taste for spicy food or seafood, by attending to relevant ingredients and image cues.
• The results indicate that collaborative filtering signals, when combined with content-based features, yield more accurate and personalized recommendations than either signal alone.