[Paper Review] Social Media Image Analysis for Public Health

Kiran Garimella, Abdulrahman Alfayad+1 more|arXiv (Cornell University)|Dec 14, 2015

Data-Driven Disease Surveillance24 references9 citations

TL;DR

This study investigates whether geo-tagged Instagram images—using both user-provided and machine-generated image tags—can predict county-level public health statistics. It finds that while user tags generally outperform machine-generated tags, automated tags like 'liquid' and 'glass' are superior for predicting excessive drinking, suggesting machine-generated tags can reveal stigmatized health behaviors invisible in text alone.

ABSTRACT

Several projects have shown the feasibility to use textual social media data to track public health concerns, such as temporal influenza patterns or geographical obesity patterns. In this paper, we look at whether geo-tagged images from Instagram also provide a viable data source. Especially for "lifestyle" diseases, such as obesity, drinking or smoking, images of social gatherings could provide information that is not necessarily shared in, say, tweets. In this study, we explore whether (i) tags provided by the users and (ii) annotations obtained via automatic image tagging are indeed valuable for studying public health. We find that both user-provided and machine-generated tags provide information that can be used to infer a county's health statistics. Whereas for most statistics user-provided tags are better features, for predicting excessive drinking machine-generated tags such as "liquid" and "glass" yield better models. This hints at the potential of using machine-generated tags to study substance abuse.

Motivation & Objective

• To assess whether geo-tagged Instagram images can serve as a viable data source for public health monitoring.
• To compare the predictive power of user-provided tags versus machine-generated tags (via deep learning) for county-level health statistics.
• To evaluate whether automatic image tagging can uncover health-related behaviors not explicitly labeled by users, particularly for stigmatized conditions like substance abuse.
• To explore the feasibility of using visual data to study lifestyle diseases such as obesity, drinking, and physical inactivity.

Proposed method

• The study uses geo-tagged Instagram images from U.S. counties, extracting both user-provided hashtags and machine-generated tags from the Imagga API.
• It applies regression models to predict county-level health statistics using a combination of image tags and demographic data (e.g., race, income, education).
• The performance of models is evaluated using Pearson correlation (r) and mean absolute error (MAE), comparing models with and without image tags.
• Feature importance is analyzed by computing the absolute correlation between individual tags and health outcomes to identify the most predictive visual cues.
• Models are trained and tested on a county-level dataset, with cross-validation to ensure robustness.
• The study uses deep learning-based image recognition (e.g., Convolutional Neural Networks) to generate tags such as 'liquid', 'beverage', and 'glass' from image content.

Experimental results

Research questions

• RQ1 Can user-provided hashtags on geo-tagged Instagram images predict county-level public health statistics more accurately than demographic data alone?
• RQ2 Do machine-generated image tags from deep learning models provide additional predictive power beyond user-provided tags for public health indicators?
• RQ3 For which health outcomes—especially stigmatized behaviors like excessive drinking—do machine-generated tags outperform user-provided tags?
• RQ4 What specific visual concepts (e.g., 'glass', 'liquid') are most predictive of substance abuse indicators in image data?
• RQ5 Can image-based features reveal behavioral patterns related to lifestyle diseases that are not captured in textual social media data?

Key findings

• For most health statistics, models using user-provided hashtags outperformed those using only demographic data, with Pearson correlation (r) improvements ranging from 0.46 to 0.84.
• For excessive drinking, machine-generated tags such as 'liquid', 'beverage', and 'glass' significantly outperformed user-provided tags, achieving a Pearson correlation of 0.48 compared to 0.22 with user tags alone.
• The model using machine-generated tags achieved a mean absolute error (MAE) of 5.2 for excessive drinking, compared to 6.2 with user tags, indicating better predictive accuracy.
• User-provided tags like '#fatgirlstatus' were highly correlated with physical inactivity, suggesting self-referential language reflects lifestyle behaviors.
• Machine-generated tags were particularly effective for stigmatized behaviors where users are unlikely to self-identify, such as substance use, due to implicit visual cues.
• The feature analysis revealed that tags like 'meeting' and 'party' were strongly correlated with excessive drinking, indicating social context is captured by automated tagging.