[Paper Review] Sequential Voting Promotes Collective Discovery in Social Recommendation Systems

L. Elisa Celis, P. M. Krafft+1 more|arXiv (Cornell University)|Mar 14, 2016

Mobile Crowdsensing and Crowdsourcing21 references12 citations

TL;DR

This study investigates whether sequential voting in social recommendation systems improves collective discovery of high-quality educational content. Using a simulated MOOC on Amazon Mechanical Turk, it finds that sequential voting significantly increases self-reported learning compared to independent voting and expert curation, suggesting social influence can enhance content ranking when quality is measured objectively via learning outcomes.

ABSTRACT

One goal of online social recommendation systems is to harness the wisdom of crowds in order to identify high quality content. Yet the sequential voting mechanisms that are commonly used by these systems are at odds with existing theoretical and empirical literature on optimal aggregation. This literature suggests that sequential voting will promote herding---the tendency for individuals to copy the decisions of others around them---and hence lead to suboptimal content recommendation. Is there a problem with our practice, or a problem with our theory? Previous attempts at answering this question have been limited by a lack of objective measurements of content quality. Quality is typically defined endogenously as the popularity of content in absence of social influence. The flaw of this metric is its presupposition that the preferences of the crowd are aligned with underlying quality. Domains in which content quality can be defined exogenously and measured objectively are thus needed in order to better assess the design choices of social recommendation systems. In this work, we look to the domain of education, where content quality can be measured via how well students are able to learn from the material presented to them. Through a behavioral experiment involving a simulated massive open online course (MOOC) run on Amazon Mechanical Turk, we show that sequential voting systems can surface better content than systems that elicit independent votes.

Motivation & Objective

• To evaluate whether sequential voting in social recommendation systems leads to better content discovery than independent voting or expert curation.
• To assess the role of social influence in collective intelligence using an objective measure of content quality based on learning outcomes.
• To investigate whether sequential voting mechanisms can outperform expert-curated content in identifying high-quality educational materials.
• To examine whether herding behavior under sequential voting contributes to improved learning outcomes.

Proposed method

• Conducted a behavioral experiment using Amazon Mechanical Turk to simulate a MOOC forum with peer-generated explanations.
• Collected data on content quality using two exogenous metrics: self-reported learning and post-consumption test scores.
• Compared three voting conditions: sequential voting (with public vote visibility), independent voting (no visibility of prior votes), and expert-curated content.
• Used the Gini index to measure inequality in vote distribution as a proxy for herding behavior.
• Selected explanations based on expert grading for the expert condition, ensuring high-quality content was available as a benchmark.
• Analyzed differences in learning outcomes across conditions to assess the impact of social influence on content discovery.

Experimental results

Research questions

• RQ1 Does sequential voting lead to better content discovery than independent voting in social recommendation systems?
• RQ2 Can sequential voting outperform expert-curated content in identifying high-quality educational materials?
• RQ3 To what extent does herding behavior—evidenced by vote inequality—contribute to improved learning outcomes in sequential voting?
• RQ4 How do self-reported learning and test scores correlate with content quality under different voting mechanisms?
• RQ5 Is there a mechanism by which social influence enhances learning beyond what is achievable through independent voting or expert selection?

Key findings

• Self-reported learning was significantly higher in the sequential voting condition compared to both independent voting and expert-curated content.
• Average test scores were consistently higher in the sequential condition, although the difference did not reach statistical significance.
• The Gini index indicated higher vote inequality in the sequential condition, suggesting the presence of herding behavior.
• The sequential voting condition outperformed the expert-curated condition in terms of self-reported learning, indicating potential superiority of crowd-based discovery.
• The study found no conclusive evidence that herding was the primary driver of improved outcomes, particularly in courses with fewer high-quality explanations.
• The results suggest that sequential voting may enable the crowd to discover better content than experts in some contexts, challenging assumptions about expert superiority in content curation.