[Paper Review] Learning to Taste: A Multimodal Wine Dataset

Thoranna Bender, Simon Lyck Bjært Sørensen+6 more|arXiv (Cornell University)|Aug 31, 2023

Wine Industry and TourismBusiness, Management and Accounting53 references3 citations

TL;DR

This paper introduces WineSensed, a large-scale multimodal wine dataset integrating 897k wine label images, 824k user reviews, and 5k+ pairwise flavor similarity annotations from a crowdsourced tasting study. It proposes FEAST, a shared embedding model that fuses image, text, and flavor data into a low-dimensional space, significantly improving alignment with human flavor perception and performance in coarse flavor classification tasks.

ABSTRACT

We present WineSensed, a large multimodal wine dataset for studying the relations between visual perception, language, and flavor. The dataset encompasses 897k images of wine labels and 824k reviews of wines curated from the Vivino platform. It has over 350k unique bottlings, annotated with year, region, rating, alcohol percentage, price, and grape composition. We obtained fine-grained flavor annotations on a subset by conducting a wine-tasting experiment with 256 participants who were asked to rank wines based on their similarity in flavor, resulting in more than 5k pairwise flavor distances. We propose a low-dimensional concept embedding algorithm that combines human experience with automatic machine similarity kernels. We demonstrate that this shared concept embedding space improves upon separate embedding spaces for coarse flavor classification (alcohol percentage, country, grape, price, rating) and aligns with the intricate human perception of flavor.

Motivation & Objective

• To bridge the gap between machine learning and food science by creating a rich, multimodal dataset for modeling human flavor perception in wine.
• To capture the nuances of human flavor perception through a large-scale sensory study using the Napping methodology.
• To develop a shared embedding space that integrates visual, textual, and flavor modalities for improved downstream classification and perception alignment.
• To validate that flavor annotations enhance multimodal representation learning beyond traditional attributes like grape, price, and rating.

Proposed method

• Conducted a large-scale Napping study with 256 participants to collect pairwise flavor similarity judgments across over 100 wines.
• Digitized physical annotation sheets using Harris corner detection and homographic projection to extract pixel-wise distances between wine positions.
• Mapped colored stickers on the sheets to wine identities and computed Euclidean distances as flavor similarity scores.
• Proposed FEAST (Flavor Embeddings from Annotated Similarity & Text-Image), a low-dimensional concept embedding algorithm combining human-annotated flavor distances with vision-language embeddings.
• Trained the model to align image and text embeddings with the human-annotated flavor space, using pairwise distances as supervision.
• Leveraged pre-trained multimodal models to extract semantic and structural features from wine label images and user reviews, which were fused with flavor embeddings.

Experimental results

Research questions

• RQ1 Can a shared multimodal embedding space improve the modeling of coarse flavor concepts such as alcohol percentage, grape variety, and country of origin?
• RQ2 Does incorporating human-annotated flavor similarity data enhance the alignment of machine representations with human perception of taste?
• RQ3 How do different data modalities—images, text, and flavor annotations—contribute jointly to learning a more accurate flavor representation?
• RQ4 Is the use of pairwise flavor distances more effective than ordinal rankings for training flavor-aware representations?

Key findings

• Using pairwise flavor distances (rather than ordinal rankings) significantly improved the quality of the learned flavor representation, validating the annotation methodology.
• The integration of multiple modalities—images, text, and flavor annotations—resulted in a substantial performance boost in downstream classification tasks.
• The proposed FEAST model achieved strong alignment with human perception of flavor, demonstrating that the shared embedding space captures intricate sensory relationships.
• The model successfully encoded coarse flavor concepts such as grape composition, alcohol percentage, and country of origin within the shared embedding space.
• The dataset and preprocessing pipeline are publicly available under a Creative Commons Attribution 4.0 license, enabling broad reuse and extension.
• No data retention or consent revocation mechanisms were implemented, as the dataset is fully anonymized and cannot be traced to individuals.