[Paper Review] Everything About You: A Multimodal Approach towards Friendship Inference in Online Social Networks

Tahleen Rahman, Mario Fritz+2 more|arXiv (Cornell University)|Mar 2, 2020

Social Media and Politics43 references4 citations

TL;DR

This paper proposes a multimodal framework for inferring friendships in online social networks by combining images, text, hashtags, geo-locations, and existing friendships. Using a real-world Instagram dataset of 22 million posts, the multimodal attack achieves an AUC above 0.9, significantly outperforming monomodal approaches, highlighting the need for comprehensive privacy defenses against multifaceted data exploitation.

ABSTRACT

Most previous works in privacy of Online Social Networks (OSN) focus on a restricted scenario of using one type of information to infer another type of information or using only static profile data such as username, profile picture or home location. However the multimedia footprints of users has become extremely diverse nowadays. In reality, an adversary would exploit all types of information obtainable over time, to achieve its goal. In this paper, we analyse OSN privacy by jointly exploiting longterm multimodal information. We focus in particular on inference of social relationships. We consider five popular components of posts shared by users, namely images, hashtags, captions, geo-locations and published friendships. Large scale evaluation on a real-world OSN dataset shows that while our monomodal attacks achieve strong predictions, our multimodal attack leads to a stronger performance with AUC (area under the ROC curve) above 0.9. Our results highlight the need for multimodal obfuscation approaches towards protecting privacy in an era where multimedia footprints of users get increasingly diverse.

Motivation & Objective

• To assess privacy risks arising from the collective, long-term multimodal content shared by users in online social networks.
• To investigate whether combining multiple modalities—images, text, hashtags, locations, and existing friendships—improves friendship inference beyond single-modality approaches.
• To develop and evaluate novel monomodal attacks based on hashtags, images, and textual captions for friendship inference.
• To design and evaluate a robust multimodal fusion framework that combines all five components to maximize inference accuracy.
• To demonstrate the robustness and scalability of the multimodal attack across varying data availability and user posting behaviors.

Proposed method

• Proposes three novel monomodal friendship inference attacks: one based on hashtag co-occurrence patterns, one using NLP-based statistical features (e.g., entropy, frequency) from captions, and one using deep learning (ResNet) to extract semantic embeddings from images.
• Employs a multimodal fusion mechanism that combines predictions from all five components—images, text, hashtags, geo-locations, and existing friendships—into a single, unified inference model.
• Uses a real-world Instagram dataset containing 22 million user posts from two cities to train and evaluate all models under realistic conditions.
• Applies state-of-the-art techniques: ResNet-50 for image feature extraction, TF-IDF and entropy-based features for text, and bipartite graph embedding for hashtag analysis.
• Employs ROC-AUC as the primary evaluation metric to compare monomodal and multimodal attack performance.
• Validates robustness by testing on subsets with varying data availability, showing consistent performance gains when combining modalities.

Experimental results

Research questions

• RQ1 Can hashtag co-occurrence patterns be effectively leveraged to infer social relationships in online social networks?
• RQ2 To what extent do textual captions and image content contribute to accurate friendship inference when used individually?
• RQ3 How much does combining multiple modalities—images, text, hashtags, locations, and existing friendships—improve the accuracy of friendship inference compared to monomodal approaches?
• RQ4 Which components of user-generated content are most complementary in a multimodal inference setting?
• RQ5 How robust are multimodal friendship inference attacks when the amount of shared data per user varies?

Key findings

• The hashtag-based monomodal attack achieves an AUC of 0.86, significantly outperforming prior work by 7 percentage points.
• The text-based attack achieves an AUC of 0.83, demonstrating the predictive power of linguistic features in friendship inference.
• The image-based attack achieves an AUC of 0.637, establishing that visual content alone is a viable, though less effective, signal for friendship inference.
• The multimodal attack, combining all five components, achieves an AUC of 0.92 in the LA dataset, exceeding the best monomodal result (0.8555 from hashtags) by a significant margin.
• The fusion of complementary modalities leads to a substantial performance boost, with each additional component improving overall inference accuracy.
• The multimodal attack remains robust even when data availability is limited, indicating its practical threat potential across diverse user behaviors.