[Paper Review] Transferring Knowledge for Food Image Segmentation using Transformers and Convolutions

Grant Sinha, Krish Parmar+5 more|arXiv (Cornell University)|Jun 15, 2023

Nutritional Studies and DietMedicine3 citations

TL;DR

This paper proposes transferring knowledge from Vision Transformers (BEiT-v2) and convolutional networks (InternImage) for food image segmentation using the FoodSeg103 benchmark. BEiT-v2 achieved a new state-of-the-art mIoU of 49.4, outperforming prior methods due to its global receptive field and vector-quantized knowledge distillation, demonstrating superior feature transferability in complex food segmentation tasks with high inter-class similarity and long-tailed data distributions.

ABSTRACT

Food image segmentation is an important task that has ubiquitous applications, such as estimating the nutritional value of a plate of food. Although machine learning models have been used for segmentation in this domain, food images pose several challenges. One challenge is that food items can overlap and mix, making them difficult to distinguish. Another challenge is the degree of inter-class similarity and intra-class variability, which is caused by the varying preparation methods and dishes a food item may be served in. Additionally, class imbalance is an inevitable issue in food datasets. To address these issues, two models are trained and compared, one based on convolutional neural networks and the other on Bidirectional Encoder representation for Image Transformers (BEiT). The models are trained and valuated using the FoodSeg103 dataset, which is identified as a robust benchmark for food image segmentation. The BEiT model outperforms the previous state-of-the-art model by achieving a mean intersection over union of 49.4 on FoodSeg103. This study provides insights into transfering knowledge using convolution and Transformer-based approaches in the food image domain.

Motivation & Objective

• To evaluate the transferability of Vision Transformers versus convolutional networks in food image segmentation.
• To address challenges in food segmentation, including inter-class similarity, intra-class variability, and long-tailed data distributions.
• To benchmark performance on the FoodSeg103 dataset, identified as a robust and challenging benchmark for fine-grained food segmentation.
• To investigate whether Transformer-based models like BEiT-v2 can better capture global context and semantic relationships in complex food scenes than convolutional models.
• To compare the effectiveness of knowledge distillation and self-supervised pretraining in improving segmentation performance on low-resource food datasets.

Proposed method

• Fine-tuned BEiT-v2, a Vision Transformer pre-trained on ImageNet-1K with vector-quantized knowledge distillation, for semantic segmentation on FoodSeg103.
• Trained InternImage-B, a state-of-the-art convolutional network using deformable convolutions (DCN-V3), as a strong baseline for comparison.
• Applied standard semantic segmentation head with cross-entropy and Dice loss to both models during fine-tuning.
• Used data augmentation and class-balancing strategies to mitigate long-tailed class distribution issues in FoodSeg103.
• Employed a two-stage training process: pre-training on ImageNet-1K followed by fine-tuning on FoodSeg103 with learning rate scheduling.
• Utilized vector-quantized tokenization in BEiT-v2’s pretraining to learn a discrete visual codebook, enhancing semantic representation learning.

Experimental results

Research questions

• RQ1 Does the Vision Transformer backbone BEiT-v2 transfer knowledge more effectively than a modern convolutional network (InternImage-B) for food image segmentation?
• RQ2 How do global receptive fields and self-attention mechanisms improve segmentation performance on complex, overlapping food scenes?
• RQ3 To what extent does vector-quantized knowledge distillation in BEiT-v2 enhance feature representation for fine-grained food segmentation?
• RQ4 How do long-tailed class distributions and inter-class similarity affect model generalization in food segmentation?
• RQ5 Can BEiT-v2 overcome limitations of convolutional models in distinguishing visually similar food items like filet mignon and pork chop?

Key findings

• BEiT-v2 achieved a mean intersection over union (mIoU) of 49.4 on the FoodSeg103 dataset, setting a new state-of-the-art performance.
• The InternImage-B model achieved a mIoU of 41.1, demonstrating that while strong, it underperforms relative to the Vision Transformer-based approach.
• BEiT-v2 showed superior performance on long-tail classes such as hamburger, pudding, and kelp, though all models struggled with these due to sparse training data.
• The global receptive field of BEiT-v2 enabled better context understanding, improving segmentation of overlapping and mixed food items compared to InternImage.
• Despite BEiT-v2’s overall superiority, there were cases—such as misclassifying mashed potatoes and meat as a single segment—where InternImage performed better, highlighting model-specific failure modes.
• Vector-quantized knowledge distillation in BEiT-v2’s pretraining contributed to stronger semantic representation learning, enhancing performance on fine-grained segmentation tasks.