[Paper Review] Large Scale Visual Food Recognition

Weiqing Min, Zhiling Wang+6 more|arXiv (Cornell University)|Mar 30, 2021

Advanced Chemical Sensor Technologies97 references17 citations

TL;DR

This paper introduces Food2K, the largest food recognition dataset with 2,000 categories and over 1 million images, significantly surpassing existing datasets in scale. It proposes a deep progressive region enhancement network that uses progressive training and self-attention to learn diverse local features and enhance them with multi-scale context, achieving state-of-the-art performance and strong generalization across food recognition, retrieval, detection, segmentation, and cross-modal recipe tasks.

ABSTRACT

Food recognition plays an important role in food choice and intake, which is essential to the health and well-being of humans. It is thus of importance to the computer vision community, and can further support many food-oriented vision and multimodal tasks. Unfortunately, we have witnessed remarkable advancements in generic visual recognition for released large-scale datasets, yet largely lags in the food domain. In this paper, we introduce Food2K, which is the largest food recognition dataset with 2,000 categories and over 1 million images.Compared with existing food recognition datasets, Food2K bypasses them in both categories and images by one order of magnitude, and thus establishes a new challenging benchmark to develop advanced models for food visual representation learning. Furthermore, we propose a deep progressive region enhancement network for food recognition, which mainly consists of two components, namely progressive local feature learning and region feature enhancement. The former adopts improved progressive training to learn diverse and complementary local features, while the latter utilizes self-attention to incorporate richer context with multiple scales into local features for further local feature enhancement. Extensive experiments on Food2K demonstrate the effectiveness of our proposed method. More importantly, we have verified better generalization ability of Food2K in various tasks, including food recognition, food image retrieval, cross-modal recipe retrieval, food detection and segmentation. Food2K can be further explored to benefit more food-relevant tasks including emerging and more complex ones (e.g., nutritional understanding of food), and the trained models on Food2K can be expected as backbones to improve the performance of more food-relevant tasks. We also hope Food2K can serve as a large scale fine-grained visual recognition benchmark.

Motivation & Objective

• To address the lack of large-scale, high-quality datasets in food visual recognition, which limits progress in food computing and fine-grained visual analysis.
• To establish a scalable benchmark for training and evaluating advanced models in food recognition and related vision tasks.
• To improve food visual representation learning by enabling models to capture fine-grained, diverse, and context-rich features through progressive training and attention mechanisms.
• To demonstrate the generalization capability of Food2K across multiple downstream tasks, including image recognition, retrieval, detection, segmentation, and cross-modal recipe retrieval.
• To support emerging applications such as few-shot food recognition, transfer learning across cuisines and super-classes, and food image generation.

Proposed method

• Proposes a deep progressive region enhancement network (DPREN) that learns local features through a progressive training strategy to capture diverse and complementary visual patterns from food images.
• Introduces a region feature enhancement module that uses multi-scale self-attention to integrate contextual information into local features, improving their discriminative power.
• Employs iterative data cleaning, professional annotation, and quality control to ensure high reliability and diversity in the Food2K dataset.
• Utilizes a hierarchical food ontology with 2,000 categories across super-classes such as vegetables, meat, and fried food to ensure semantic coverage and scalability.
• Trains models on Food2K and evaluates their transferability to downstream tasks using zero-shot and few-shot evaluation protocols.
• Supports transfer learning across cuisines, super-classes, and scenarios, enabling cross-domain generalization analysis.

Experimental results

Research questions

• RQ1 Can a large-scale, high-quality food dataset significantly improve the performance and generalization of food visual recognition models?
• RQ2 How effective is progressive training in learning diverse and complementary local features for fine-grained food recognition?
• RQ3 To what extent can self-attention mechanisms enhance local features with multi-scale context for improved recognition accuracy?
• RQ4 Can models pre-trained on Food2K generalize effectively to downstream tasks such as food detection, segmentation, and cross-modal recipe retrieval?
• RQ5 What is the potential of Food2K as a benchmark for emerging tasks like few-shot food recognition and cross-cuisine transfer learning?

Key findings

• Food2K contains 1,036,564 images across 2,000 categories, surpassing existing datasets in both category count and image volume by an order of magnitude.
• The proposed deep progressive region enhancement network achieves state-of-the-art performance on Food2K, demonstrating superior feature learning through progressive training and self-attention.
• Models pre-trained on Food2K show strong generalization, improving performance across multiple tasks including food image recognition, retrieval, detection, segmentation, and cross-modal recipe retrieval.
• Food2K enables effective zero-shot and few-shot transfer learning, with promising results in cross-cuisine and cross-super-class transfer learning scenarios.
• The dataset supports novel applications such as food image generation using GANs, with promising results in generating realistic, semantically consistent food images.
• The dataset and models are publicly released at http://123.57.42.89/FoodProject.html, facilitating broader research in food vision and multimodal learning.