[Paper Review] Leveraging User Diversity to Harvest Knowledge on the Social Web

Jeon-Hyung Kang, Kristina Lerman|arXiv (Cornell University)|Oct 21, 2011

Multimodal Machine Learning Applications23 references3 citations

TL;DR

This paper proposes a framework to identify expert users on the Social Web based on the structure and content of their annotations, using their high-quality, detailed tags and hierarchical organization to guide folksonomy learning. The method improves both accuracy and completeness of folksonomies by weighting expert annotations more heavily, while also showing that novice contributions are essential for comprehensive coverage.

ABSTRACT

Social web users are a very diverse group with varying interests, levels of expertise, enthusiasm, and expressiveness. As a result, the quality of content and annotations they create to organize content is also highly variable. While several approaches have been proposed to mine social annotations, for example, to learn folksonomies that reflect how people relate narrower concepts to broader ones, these methods treat all users and the annotations they create uniformly. We propose a framework to automatically identify experts, i.e., knowledgeable users who create high quality annotations, and use their knowledge to guide folksonomy learning. We evaluate the approach on a large body of social annotations extracted from the photosharing site Flickr. We show that using expert knowledge leads to more detailed and accurate folksonomies. Moreover, we show that including annotations from non-expert, or novice, users leads to more comprehensive folksonomies than experts' knowledge alone.

Motivation & Objective

• To address the challenge of variable quality in user-generated annotations on social web platforms by identifying expert users who produce high-quality, detailed annotations.
• To improve folksonomy learning by leveraging expert knowledge, which leads to more accurate and detailed taxonomies.
• To investigate the complementary role of novice users in enhancing the completeness of folksonomies beyond what experts alone can provide.
• To develop a robust method for expert identification that remains effective even with moderate errors in expert labeling.
• To generalize the approach beyond Flickr to other structured data sources such as Delicious, Bibsonomy, and file systems.

Proposed method

• Uses structural and linguistic features of user-created personal directories (e.g., Flickr collections and sets) to identify experts, focusing on depth, granularity, and use of technical terms.
• Applies a supervised classification model trained on features such as directory depth, number of intermediate nodes, and semantic consistency of tags.
• Extends the RAP (Relational Affinity Propagation) algorithm to incorporate expert annotations with higher weight during folksonomy inference.
• Assigns preference values to expert and novice nodes in the inference process, with expert nodes receiving higher weights to guide taxonomy construction.
• Employs a robustness evaluation framework that simulates expert misidentification (up to 50% error) to test the stability of the learned folksonomy.
• Validates the method on a large dataset of Flickr annotations, comparing folksonomies learned with and without expert weighting.

Experimental results

Research questions

• RQ1 Can user annotation structure be used to automatically identify experts who produce high-quality, detailed annotations on social web platforms?
• RQ2 Does weighting expert annotations during folksonomy learning lead to more accurate and detailed taxonomies compared to treating all users equally?
• RQ3 To what extent do novice users contribute to the completeness of folksonomies, and is their inclusion necessary even when expert knowledge is used?
• RQ4 How robust is the folksonomy learning process to errors in expert identification, particularly when up to 50% of experts are misclassified?
• RQ5 Can the proposed framework be generalized to other structured data sources beyond Flickr, such as Delicious bundles or Bibsonomy relations?

Key findings

• Using expert annotations with higher weight in the RAP inference process leads to significantly more accurate and detailed folksonomies compared to uniform weighting.
• Even with up to 50% error in expert identification, the quality of the learned folksonomy remains stable, demonstrating strong robustness to misclassification.
• While expert knowledge is essential for accuracy and detail, novice users contribute unique, non-redundant tags that enhance the completeness of the folksonomy.
• Folksonomies learned by combining expert and novice annotations are more comprehensive than those based on experts alone, indicating complementary roles.
• The method successfully identifies experts based on structural features of annotations, such as directory depth and use of intermediate concepts, outperforming uniform user weighting.
• The framework generalizes well to other data sources, including eBay categories, Delicious bundles, and Bibsonomy, suggesting broad applicability.