[Paper Review] Social Bayesian Learning in the Wisdom of the Crowd

Dhaval Adjodah, Yan Leng+3 more|arXiv (Cornell University)|Dec 28, 2017

Advanced Text Analysis Techniques8 references3 citations

TL;DR

This paper proposes a Social Bayesian model that accurately captures how individuals update their beliefs in response to social information in crowd-sourced prediction tasks. By leveraging pre- and post-social predictions and peer distribution histograms, the model outperforms established models like DeGroot by up to 147.1% in mean absolute error reduction, offering a more accurate representation of belief updating and enabling improved aggregation through inter-individual belief correlation estimation.

ABSTRACT

Being able to correctly aggregate the beliefs of many people into a single belief is a problem fundamental to many important social, economic and political processes such as policy making, market pricing and voting. Although there exist many models and mechanisms for aggregation, there is a lack of methods and literature regarding the aggregation of opinions when influence and learning between individuals exist. This is in part because there are not many models of how people update their belief when exposed to the beliefs of others, and so it is hard to quantify the dependencies between people's mental models which is essential to minimizing redundancies in the aggregation. In this paper, we explore many models of how users influence and learn from each other, and we benchmark our models against the well-known DeGroot model. Our main contributions are: 1) we collect a new dataset of unprecedented size and detail to be posted online; 2) we develop a new Social Bayesian model of how people update their mental models, 3) we compare of our model to other well-known social learning models. Specifically, we show that our new Social Bayesian model is superior to the other models tested.

Motivation & Objective

• To address the lack of models that account for social influence and learning in belief aggregation within the wisdom of the crowd.
• To develop a more accurate model of how individuals update their beliefs after exposure to peers' predictions.
• To estimate inter-individual belief correlations using belief distributions, reducing redundancy in aggregation.
• To benchmark the new model against established models like DeGroot and naive Bayesian approaches.
• To provide a dataset of unprecedented size and detail for future research on social learning and belief aggregation.

Proposed method

• The model uses Bayesian updating to estimate post-social beliefs based on pre-social beliefs and observed social information (mean of peer predictions).
• It assumes conditional independence between social information and prior belief given the post-social belief, and tests this assumption via numerical joint distribution estimation.
• Belief distributions are discretized into bins to compute joint probabilities and validate the model's fit.
• The model is evaluated using mean absolute error (MAE) between predicted and actual post-social predictions across seven rounds of data.
• Alternative models are tested using combinations of normal/uniform priors, empirical/social likelihoods, and posterior modes/means.
• The model's superiority is quantified by comparing error reduction relative to the DeGroot baseline.

Experimental results

Research questions

• RQ1 How do individuals update their beliefs when exposed to the predictions of others in a crowd-sourced setting?
• RQ2 To what extent does the Social Bayesian model better capture human belief updating than existing models like DeGroot or naive Bayesian approaches?
• RQ3 Can belief distributions estimated via the Social Bayesian model reveal meaningful inter-individual correlations in beliefs?
• RQ4 How does the model's performance vary across different market conditions, such as Brexit or the Trump election?
• RQ5 Can belief correlation measures derived from the model improve the accuracy of wisdom of the crowd aggregations?

Key findings

• The Social Bayesian model reduces mean absolute error by 54.2% compared to the best baseline model in Round 1, with improvements ranging from 2.0% to 147.1% across all rounds.
• In Round 4, the model achieved a 58.9% improvement over the baseline, and in Round 7, it achieved a 147.1% improvement, indicating strong performance under diverse conditions.
• The model's error was consistently lower than all other models tested, including DeGroot, naive Bayesian variants, and probabilistic learning models.
• The model's ability to represent belief distributions enables estimation of inter-individual belief correlations using metrics like Kullback-Leibler divergence.
• The model's superior fit suggests that social influence and learning dynamics are better captured by Bayesian updating than by linear averaging or simple independence assumptions.
• The dataset collected—featuring 17,420 predictions from 2,037 students over seven rounds, including during major market events—provides a rich resource for future research on social learning and belief aggregation.