[Paper Review] Personalized Classifier for Food Image Recognition

Shota Horiguchi, Sosuke Amano+2 more|arXiv (Cornell University)|Apr 8, 2018

Image Retrieval and Classification Techniques51 references3 citations

TL;DR

This paper proposes a sequential personalized classifier (SPC) for food image recognition that combines deep features with a weighted nearest neighbor approach to enable incremental personalization with minimal user data. The method achieves superior accuracy over existing techniques by dynamically adapting to individual users' food image patterns without retraining, using only a few samples per novel class.

ABSTRACT

Currently, food image recognition tasks are evaluated against fixed datasets. However, in real-world conditions, there are cases in which the number of samples in each class continues to increase and samples from novel classes appear. In particular, dynamic datasets in which each individual user creates samples and continues the updating process often have content that varies considerably between different users, and the number of samples per person is very limited. A single classifier common to all users cannot handle such dynamic data. Bridging the gap between the laboratory environment and the real world has not yet been accomplished on a large scale. Personalizing a classifier incrementally for each user is a promising way to do this. In this paper, we address the personalization problem, which involves adapting to the user's domain incrementally using a very limited number of samples. We propose a simple yet effective personalization framework which is a combination of the nearest class mean classifier and the 1-nearest neighbor classifier based on deep features. To conduct realistic experiments, we made use of a new dataset of daily food images collected by a food-logging application. Experimental results show that our proposed method significantly outperforms existing methods.

Motivation & Objective

• Address the gap between lab-based fixed-dataset image classification and real-world dynamic food image data where users continuously add new, personalized food classes.
• Overcome limitations of conventional incremental learning methods that assume stable class distributions and equal sample sizes across classes.
• Enable effective personalization in real-world settings where users have limited labeled images per food category, especially for novel or rare items.
• Develop a lightweight, scalable framework that adapts to individual user preferences and visual domains without heavy retraining.
• Improve food logging applications by reducing manual labeling effort through accurate, user-specific image classification.

Proposed method

• The SPC framework combines a fixed-class deep CNN classifier with a user-specific nearest class mean (NCM) and 1-NN classifier based on deep features.
• For each user, the system maintains a personalized prototype vector for each class, updated incrementally as new images are added.
• Classification uses a weighted fusion of the fixed-class CNN and user-specific classifiers, with weights determined by confidence scores.
• The method leverages deep features extracted from a pre-trained CNN (e.g., VGG-16) to represent image embeddings in a high-dimensional space.
• The user-specific classifier uses a nearest neighbor approach with distance-based weighting to handle one-shot and few-shot learning scenarios.
• The system supports incremental learning: new samples are added without retraining the entire model, enabling real-time adaptation.

Experimental results

Research questions

• RQ1 Can a lightweight, incremental classifier framework effectively adapt to individual users' food image patterns with minimal labeled data?
• RQ2 How does the performance of a personalized classifier compare to a fixed-class baseline in real-world food image recognition with dynamic, user-generated data?
• RQ3 To what extent can a hybrid approach combining fixed-class CNN and user-specific nearest neighbor learning improve classification accuracy under one-shot and few-shot conditions?
• RQ4 Does the proposed method outperform existing incremental learning and domain adaptation techniques in personalized food image recognition?
• RQ5 How does the system handle high intra-class variation and inter-class similarity across different users?

Key findings

• The proposed SPC framework significantly outperforms existing methods in personalized food image recognition, especially under low-data regimes.
• The method achieves competitive accuracy with the fixed-class CNN at initialization and improves steadily with user-specific data updates.
• The SPC framework reduces labeling effort in food logging applications by presenting correct labels more frequently than the fixed-class baseline.
• The linear combination strategy (SPCsum) failed to improve performance, indicating that simple fusion of fixed and user-specific classifiers is suboptimal.
• The new FLD dataset revealed high inter-class similarity and intra-class diversity across users, justifying the need for personalization.
• The system demonstrated robust performance even with only one or a few samples per novel food class, confirming its effectiveness for one-shot learning.