[Paper Review] Learning Structural Representations for Recipe Generation and Food Retrieval

Hao Wang, Guosheng Lin+2 more|arXiv (Cornell University)|Oct 4, 2021

Multimodal Machine Learning Applications4 citations

TL;DR

This paper proposes an unsupervised method to learn sentence-level tree structures for lengthy cooking recipes, enhancing recipe generation and food cross-modal retrieval. By extending ON-LSTM with Quick Thoughts for unsupervised tree structure labeling and integrating graph attention networks (GAT) to encode these structures, the model improves coherence and performance on the Recipe1M benchmark, achieving state-of-the-art results in both tasks.

ABSTRACT

Food is significant to human daily life. In this paper, we are interested in learning structural representations for lengthy recipes, that can benefit the recipe generation and food cross-modal retrieval tasks. Different from the common vision-language data, here the food images contain mixed ingredients and target recipes are lengthy paragraphs, where we do not have annotations on structure information. To address the above limitations, we propose a novel method to unsupervisedly learn the sentence-level tree structures for the cooking recipes. Our approach brings together several novel ideas in a systematic framework: (1) exploiting an unsupervised learning approach to obtain the sentence-level tree structure labels before training; (2) generating trees of target recipes from images with the supervision of tree structure labels learned from (1); and (3) integrating the learned tree structures into the recipe generation and food cross-modal retrieval procedure. Our proposed model can produce good-quality sentence-level tree structures and coherent recipes. We achieve the state-of-the-art recipe generation and food cross-modal retrieval performance on the benchmark Recipe1M dataset.

Motivation & Objective

• To address the lack of structural annotations in food image-text datasets, where recipes are long paragraphs and ingredients are mixed, making traditional vision-language methods less effective.
• To learn sentence-level tree structures for recipes in an unsupervised manner, without requiring manual structural annotations.
• To improve recipe generation quality by incorporating learned tree structures as structural guidance during text generation.
• To enhance food cross-modal retrieval by fusing tree-structured representations into recipe features for better matching with images.
• To demonstrate that unsupervised structural learning leads to improved performance in both recipe generation and cross-modal retrieval tasks.

Proposed method

• Propose a novel recipe2tree module based on an extended ON-LSTM architecture to generate sentence-level tree structures from recipe text in an unsupervised way.
• Use the Quick Thoughts training objective to pre-train the ON-LSTM variant, enabling it to learn hierarchical sentence structures without supervision.
• Design a structure-aware generation network (SGN) that uses an img2tree module to predict recipe tree structures from food image features.
• Employ graph attention networks (GAT) to encode the predicted tree structures into contextual embeddings for use in both generation and retrieval.
• Integrate tree structure embeddings into the recipe generation decoder to guide coherent, long-form recipe generation.
• Enhance cross-modal retrieval by fusing tree-structure features with original recipe embeddings to improve image and recipe matching.

Experimental results

Research questions

• RQ1 Can unsupervised learning of sentence-level tree structures improve the coherence and quality of generated recipes?
• RQ2 Can structural representations derived from recipes enhance cross-modal retrieval performance between food images and long-form cooking instructions?
• RQ3 How effective is the proposed unsupervised tree structure learning method when no ground-truth structural annotations are available?
• RQ4 To what extent do tree-structure embeddings improve generation and retrieval compared to baseline models?
• RQ5 Can the model generalize across diverse food classes and generate structurally coherent recipes from image inputs?

Key findings

• The proposed SGN model achieves state-of-the-art performance on recipe generation, outperforming existing methods on the Recipe1M benchmark in both automatic and human evaluation metrics.
• The integration of tree-structure embeddings into cross-modal retrieval significantly improves retrieval accuracy, with top-10 recall and mAP scores exceeding previous baselines.
• Qualitative analysis shows that the generated recipe tree structures are semantically coherent and structurally similar to ground-truth trees, despite no supervision on structure.
• The model successfully retrieves relevant recipes for diverse food images, including complex dishes like chicken wellingtons and potato cakes, demonstrating robust cross-modal alignment.
• The img2tree module produces diverse yet plausible tree structures from image features, which guide the generation of longer, more coherent recipes.
• The unsupervised tree structure learning approach effectively captures the sequential and hierarchical nature of cooking processes, even without explicit structural annotations.