[Paper Review] Learning to Substitute Ingredients in Recipes

Bahare Fatemi, Quentin Duval+3 more|arXiv (Cornell University)|Feb 15, 2023

Culinary Culture and TourismAgricultural and Biological Sciences3 citations

TL;DR

This paper introduces GISMo, a graph-based model that leverages both recipe-specific context and generic ingredient relational knowledge from the FlavorGraph to predict plausible ingredient substitutions. It achieves a 14% improvement in mean reciprocal rank over the best baseline, enabling effective recipe personalization through integration into an image-to-recipe generation pipeline.

ABSTRACT

Recipe personalization through ingredient substitution has the potential to help people meet their dietary needs and preferences, avoid potential allergens, and ease culinary exploration in everyone's kitchen. To address ingredient substitution, we build a benchmark, composed of a dataset of substitution pairs with standardized splits, evaluation metrics, and baselines. We further introduce Graph-based Ingredient Substitution Module (GISMo), a novel model that leverages the context of a recipe as well as generic ingredient relational information encoded within a graph to rank plausible substitutions. We show through comprehensive experimental validation that GISMo surpasses the best performing baseline by a large margin in terms of mean reciprocal rank. Finally, we highlight the benefits of GISMo by integrating it in an improved image-to-recipe generation pipeline, enabling recipe personalization through user intervention. Quantitative and qualitative results show the efficacy of our proposed system, paving the road towards truly personalized cooking and tasting experiences.

Motivation & Objective

• To address the challenge of personalized recipe creation through context-aware ingredient substitution.
• To build a standardized benchmark, Recipe1MSubs, for ingredient substitution using real user comments and recipe data.
• To develop a model that integrates recipe-specific context with generic ingredient relationships from a large-scale food graph.
• To enable practical recipe personalization by integrating substitution into an image-to-recipe generation pipeline.
• To reduce reliance on rare, fully supervised recipe-editing pairs by decoupling substitution from instruction editing.

Proposed method

• The authors construct Recipe1MSubs, a benchmark dataset of ingredient substitution pairs derived from user comments on recipe websites, with standardized train/val/test splits.
• GISMo uses a graph neural network (GNN) to encode generic ingredient relationships from the FlavorGraph, which captures co-occurrence and chemical flavor profiles.
• The model fuses contextual embeddings from a recipe’s text with graph-encoded ingredient representations using a learnable attention mechanism.
• It predicts plausible substitutions by ranking candidate ingredients based on joint contextual and relational features.
• The system is integrated into an inverse cooking pipeline: an image is first encoded via a Vision Transformer to predict ingredients, then GISMo substitutes specified ingredients, and a language model generates the updated recipe.
• The method enables zero-shot recipe personalization by leveraging the compositionality of large language models, avoiding full supervision on edited recipes.

Experimental results

Research questions

• RQ1 Can a model effectively predict context-aware ingredient substitutions by combining recipe context and generic ingredient relational knowledge?
• RQ2 How does the performance of a graph-based model compare to strong baselines in ingredient substitution, especially under realistic, non-universal substitution assumptions?
• RQ3 Can the integration of a substitution module into an image-to-recipe generation pipeline enable effective, personalized recipe creation without requiring paired original-edited recipes?
• RQ4 To what extent does leveraging both recipe-specific and global ingredient relational information improve substitution accuracy?

Key findings

• GISMo achieves a 14% relative improvement in mean reciprocal rank (MRR) over the best-performing baseline, demonstrating the effectiveness of combining contextual and relational information.
• The model significantly outperforms baselines that rely solely on recipe context or generic embeddings, highlighting the value of graph-structured ingredient knowledge.
• Qualitative results show that GISMo proposes plausible substitutions—such as replacing beer with water in a bread recipe—while the downstream recipe generation pipeline correctly adapts instructions and titles.
• The integration of GISMo into the inverse cooking pipeline enables personalized recipe generation with minimal supervision, leveraging the compositional generalization of language models.
• Random substitution leads to higher perplexity and lower IoU, confirming that intelligent substitution is crucial for coherent recipe generation.
• The system shows promise for real-world use, though it currently supports only one-to-one substitutions and may be biased toward Western cuisines due to training data.