[Paper Review] Wide-Slice Residual Networks for Food Recognition

Niki Martinel, Gian Luca Foresti+1 more|arXiv (Cornell University)|Dec 20, 2016

Advanced Chemical Sensor Technologies19 citations

TL;DR

This paper proposes WISeR, a novel deep residual network for food recognition that integrates a slice convolution layer to capture vertical food layer structures and a residual learning branch for general feature extraction. By fusing these branches, WISeR achieves a top-1 accuracy of 90.27% on the Food-101 dataset, outperforming existing methods through architecture-specific design for food composition.

ABSTRACT

Food diary applications represent a tantalizing market. Such applications, based on image food recognition, opened to new challenges for computer vision and pattern recognition algorithms. Recent works in the field are focusing either on hand-crafted representations or on learning these by exploiting deep neural networks. Despite the success of such a last family of works, these generally exploit off-the shelf deep architectures to classify food dishes. Thus, the architectures are not cast to the specific problem. We believe that better results can be obtained if the deep architecture is defined with respect to an analysis of the food composition. Following such an intuition, this work introduces a new deep scheme that is designed to handle the food structure. Specifically, inspired by the recent success of residual deep network, we exploit such a learning scheme and introduce a slice convolution block to capture the vertical food layers. Outputs of the deep residual blocks are combined with the sliced convolution to produce the classification score for specific food categories. To evaluate our proposed architecture we have conducted experimental results on three benchmark datasets. Results demonstrate that our solution shows better performance with respect to existing approaches (e.g., a top-1 accuracy of 90.27% on the Food-101 challenging dataset).

Motivation & Objective

• Address the challenge of high intra-class variation in food images, where the same dish varies significantly in appearance due to ingredients, preparation, and presentation.
• Overcome limitations of existing approaches that apply off-the-shelf deep architectures without considering food-specific structural properties.
• Design a deep learning architecture tailored to food composition, particularly vertical layering common in many dishes.
• Improve classification performance by combining structural feature learning (via slice convolution) with generic deep representation learning (via residual blocks).

Proposed method

• Introduce a novel slice convolution layer that extracts features from vertical slices of food images, modeling the layered structure of many dishes.
• Combine the slice convolution branch with a wide residual network (WRN) to learn general visual features from complex and varied food appearances.
• Enrich feature maps across layers to mitigate feature reuse degradation in deep residual networks, enhancing representational capacity.
• Fuse features from both branches through concatenation before final classification, enabling the network to leverage both structural and general visual cues.
• Utilize pre-trained ImageNet weights for the residual branch to inject strong inductive biases, improving feature discriminability.
• Apply Guided Grad-CAM for visual attention analysis to interpret and validate the model’s focus on relevant image regions during classification.

Experimental results

Research questions

• RQ1 Can a deep learning architecture specifically designed for food structure, particularly vertical layering, improve food recognition performance?
• RQ2 How does combining structural feature learning (via slice convolution) with generic residual learning enhance classification accuracy compared to standard architectures?
• RQ3 To what extent does pre-training the residual branch on ImageNet improve performance in food recognition without fine-tuning?
• RQ4 Does the proposed architecture self-identify and focus on relevant image regions during classification, as indicated by visual attention analysis?

Key findings

• WISeR achieves a top-1 accuracy of 90.27% on the Food-101 benchmark, surpassing state-of-the-art methods.
• The residual learning branch contributes significantly more to performance than the slice convolution branch, with the latter adding incremental gains when fused.
• Training the full WISeR architecture from scratch yields only 78.12% top-1 accuracy on Food-101, confirming the importance of ImageNet pre-training.
• Visual attention analysis via Guided Grad-CAM confirms that WISeR focuses only on relevant food regions, ignoring irrelevant objects like plates, spoons, or background.
• The fusion of slice and residual features leads to superior performance, indicating that structural and general features are complementary in food recognition.
• The architecture generalizes well across datasets (Food-101, UECFood100, UECFood256), demonstrating robustness to domain shifts and diverse food compositions.