[Paper Review] Mining Discriminative Food Regions for Accurate Food Recognition

Jianing Qiu, Frank P.-W. Lo+3 more|arXiv (Cornell University)|Jul 8, 2022

Advanced Chemical Sensor Technologies44 citations

TL;DR

The paper presents PAR-Net, a three-branch network that iteratively mines discriminative food regions with Adversarial Erasing and fuses local and global features for end-to-end food recognition, achieving state-of-the-art results on multiple datasets including a new Sushi-50 fine-grained dataset.

ABSTRACT

Automatic food recognition is the very first step towards passive dietary monitoring. In this paper, we address the problem of food recognition by mining discriminative food regions. Taking inspiration from Adversarial Erasing, a strategy that progressively discovers discriminative object regions for weakly supervised semantic segmentation, we propose a novel network architecture in which a primary network maintains the base accuracy of classifying an input image, an auxiliary network adversarially mines discriminative food regions, and a region network classifies the resulting mined regions. The global (the original input image) and the local (the mined regions) representations are then integrated for the final prediction. The proposed architecture denoted as PAR-Net is end-to-end trainable, and highlights discriminative regions in an online fashion. In addition, we introduce a new fine-grained food dataset named as Sushi-50, which consists of 50 different sushi categories. Extensive experiments have been conducted to evaluate the proposed approach. On three food datasets chosen (Food-101, Vireo-172, and Sushi-50), our approach performs consistently and achieves state-of-the-art results (top-1 testing accuracy of $90.4\%$, $90.2\%$, $92.0\%$, respectively) compared with other existing approaches. Dataset and code are available at https://github.com/Jianing-Qiu/PARNet

Motivation & Objective

• Motivate passive dietary monitoring by improving automatic food recognition accuracy.
• Develop a weakly supervised method to discover discriminative food regions within images.
• Design an end-to-end trainable architecture that integrates global and mined regional representations.

Proposed method

• Introduce PAR-Net with three sub-networks: P-Net (global image classification), A-Net (classifies with discriminative regions erased), and R-Net (classifies mined regions).
• Use Adversarial Erasing to progressively mine discriminative regions guided by CAM heatmaps.
• Compute CAM online to highlight discriminative regions and to generate region crops for R-Net.
• Concatenate representations from the full image and mined regions, then classify via an extra fully connected layer.
• Train end-to-end with a composite loss that includes P-Net, A-Net, R-Net, and the concatenated representation losses.

Experimental results

Research questions

• RQ1 Can discriminative region mining improve food recognition beyond global-image features?
• RQ2 How many regions should be mined to maximize performance without excessive computation?
• RQ3 Does integrating global and regional representations via end-to-end training outperform single-network baselines?
• RQ4 Is online CAM-based region mining effective for end-to-end optimization in food recognition?

Key findings

• PAR-Net consistently achieves state-of-the-art top-1 accuracy on Food-101 (1-crop: 89.3; 10-crop: 90.4), Vireo-172 (1-crop: 89.6; 10-crop: 90.2), and Sushi-50 (1-crop: 91.8; 10-crop: 92.0).
• Using three discriminative regions (T=3) provides strong performance with manageable overhead.
• Independent sub-networks for global, regional, and erased-region tasks outperform shared-weight or single-network variants.
• Concatenating global image features with mined regional features yields higher accuracy than using either source alone.