[论文解读] Eating Smart: Advancing Health Informatics with the Grounding DINO-based Dietary Assistant App

Abdelilah Nossair, Hamza El Housni|arXiv (Cornell University)|Jun 21, 2024

Mobile Health and mHealth Applications被引用 1

一句话总结

本文介绍了Smart Dietary Assistant应用程序，这是一款移动健康解决方案，利用基于Grounding DINO模型的零样本目标检测能力，无需标注数据集即可从智能手机图像中识别食物。该应用使用自托管的PostgreSQL数据库实现安全、私密的数据存储，并提供个性化饮食建议，在食物识别方面达到92.30%的F1分数，用户满意度高（NPS 41.3）。

ABSTRACT

Abstract: The Smart Dietary Assistant project combines technology and Machine Learning (ML) to offer personalized advice for people with dietary concerns such as diabetes. This approach focuses on the user helping them make decisions about their diet using the Grounding DINO model. Grounding DINO uses a text encoder and image backbone to improve detection accuracy without relying on a labeled dataset making it practical for real world situations with various food types. This model uses a 52.5 AP score on the COCO dataset and attention mechanisms that leverage features based on user-provided labels and food images to allow precise object recognition. The feature is at the core of the user app, turning smartphones into a helpful dietary advisor that enables people to manage their health effectively. The app can use your device camera to take photos that will be analyzed by the model for detection and categorize the food items correctly. This is what differs in this system: it decides to be free and not to be connected to annoying cloud databases of information. The application uses a database managed by itself that is of PostgreSQL type, ensuring the preservation of data integrity and control. This database hosting information includes all types of food products, from profiles to health insights drawn from their consumption by human beings. This helps in effective and efficient data access speed, reliability, and enhances user privacy through localized storage within the organizational infrastructure. The app focuses on improving the experiences of the users, considering that it allows them to create profiles through which they describe themselves based on preferences and tips on nutrition. In addition to calories information, the app provides insights to nutrients such as proteins, vitamins, and minerals. This makes it possible for one to decide the kind of food to take, either for weight management, muscle building, or managing health conditions. On the other part, it also assesses food compatibility versus profiles and gives personal recommendations for alternatives and recipes. Such kind of personal help is highly convenient for persons with needs as it helps them take their healthy options confidently. Developed using React Native and TypeScript, the Smart Dietary Assistant app guarantees operation across devices and platforms. It incorporates technologies beyond modeling to ensure optimal performance in food recognition, scalability for future enhancements and seamless integration, with other dietary tools. Users have the option to enjoy features like using the camera to scan food items, for tracking habits and receiving insightful analysis. They can also interact with an assistant for recommendations. The protection of data is ensured through user authentication whereas customizable settings enhance the user experience. React Native enables smooth screen transitions. The expo camera allows scanning capabilities. Local storage efficiently manages data to create an easy/appealing to use interface. The Smart Dietary Assistant app’s interface stands out for striking a balance between aesthetics and usability. The use of buttons, and a vibrant color scheme enhances user experience by making navigation and feature selection simple. The chatbot feature, represented by an avatar encourages user engagement and personalized guidance seeking. Users find camera scanning convenient although it is noted that varying lighting conditions may affect accuracy. It is this appreciation that opened doors to improvement that can guarantee success in all situations. The choice of a self-hosted PostgreSQL database for this project re-emphasizes its importance in the realms of health informatics and nutritional science. This is data that can be stored without really depending on outside cloud services, and just with that, the same can be retained as reliable information, since there are chances that it can be changed from the outside. In the future, the Smart Dietary Assistant is planned to be empowered with collaboration with devices. With this development, the application can sync with fitness trackers and smartwatches to give time-based suggestions from physiological data such as blood sugar level and calories burnt. This will connect users to devices that give them individualized advice regarding their health needs, depending on the style of activity. The application is open to collaborations with AI-powered tools in the development of personalized recipes and meal plans that would give the user an easy time adhering to his preferences, dietary restrictions, and time-in sync physiological information. With conditions like diabetes, this holistic approach to diet management is deemed beneficial because it would make the app utilities more effective, always supports objectives for weight management or muscle building, and therefore supports the overall well-being of the user. Key words: Food Image Recognition, Machine Learning in Nutrition, Zero-Shot Object Detection.

研究动机与目标

• 开发一款移动饮食助手，利用设备端人工智能实现实时、准确的食物识别，以支持糖尿病等饮食相关疾病患者。
• 通过采用自托管的PostgreSQL数据库而非依赖云存储，确保用户数据隐私。
• 基于用户档案和实时图像输入，提供个性化营养洞察，包括卡路里、宏量营养素及食物搭配建议。
• 通过集成聊天机器人助手和基于摄像头的食物扫描功能，提升用户参与度。
• 为未来与可穿戴设备集成铺平道路，实现时间精准、基于生理状态的饮食建议。

提出的方法

• 采用Grounding DINO模型，这是一种结合CLIP文本编码器与视觉主干网络的视觉-语言模型，用于零样本目标检测。
• 利用注意力机制将视觉特征与用户提供的文本标签对齐，实现在未针对标注食物数据集微调的情况下进行检测。
• 在本地部署PostgreSQL数据库，用于存储用户档案、食物数据及营养洞察，确保数据完整性和隐私性。
• 使用React Native与TypeScript构建应用，实现跨平台兼容性，并通过Expo的摄像头集成实现流畅的用户界面与体验。
• 从COCO数据集（52.5 AP）迁移学习，以提升在真实环境中多样化食物类型的泛化能力。
• 集成用户身份验证与安全数据处理协议，符合健康数据隐私标准。

实验结果

研究问题

• RQ1 像Grounding DINO这样的零样本目标检测模型是否能在未针对标注食物数据集微调的情况下，有效识别真实世界智能手机图像中的多样化食物？
• RQ2 自托管数据库架构在移动饮食助手应用中对数据隐私、完整性与性能有何影响？
• RQ3 该应用的个性化推荐引擎在多大程度上提升了糖尿病等疾病患者用户的参与度与饮食决策质量？
• RQ4 在1,589张图像的验证集上，该模型在检测食物时的精确率、召回率与F1分数表现如何？
• RQ5 用户在真实使用中对应用的易用性、准确性及推荐意愿的评分如何？

主要发现

• 该模型在1,589张图像的验证集上达到92.30%的F1分数，精确率（90.79%）与召回率（93.84%）保持良好平衡。
• 该应用获得41.3的净 promoter 分数（NPS），表明用户满意度高，具有较强的用户推荐意愿。
• 系统正确识别出1,589张图像中的1,144张（72.0%）为目标食物，同时正确拒绝了254张非目标图像。
• 在116起案例中出现误报（占总数的7.3%），另有75种食物被遗漏（占总数的4.7%），表明在稀有或模糊食物类别上仍存在挑战。
• 用户反馈证实其对界面、摄像头扫描功能及个性化推荐高度满意。
• 该应用的自托管PostgreSQL数据库确保了安全、私密且可靠的数据管理，且无需依赖外部云服务。