[Paper Review] The Size Conundrum: Why Online Knowledge Markets Can Fail at Scale

Himel Dev, Chase Geigle+3 more|arXiv (Cornell University)|Dec 1, 2017

Expert finding and Q&A systems42 references4 citations

TL;DR

This paper models StackExchange community question answering sites as knowledge markets using the Cobb-Douglas production function to analyze content generation dynamics. It finds that larger markets often suffer diseconomies of scale—measures of health like answer rates decline with size—due to diminishing returns and non-proportional scaling of high-engagement users, implying site operators must design size-dependent incentives.

ABSTRACT

In this paper, we interpret the community question answering websites on the StackExchange platform as knowledge markets, and analyze how and why these markets can fail at scale. A knowledge market framing allows site operators to reason about market failures, and to design policies to prevent them. Our goal is to provide insights on large-scale knowledge market failures through an interpretable model. We explore a set of interpretable economic production models on a large empirical dataset to analyze the dynamics of content generation in knowledge markets. Amongst these, the Cobb-Douglas model best explains empirical data and provides an intuitive explanation for content generation through concepts of elasticity and diminishing returns. Content generation depends on user participation and also on how specific types of content (e.g. answers) depends on other types (e.g. questions). We show that these factors of content generation have constant elasticity---a percentage increase in any of the inputs leads to a constant percentage increase in the output. Furthermore, markets exhibit diminishing returns---the marginal output decreases as the input is incrementally increased. Knowledge markets also vary on their returns to scale---the increase in output resulting from a proportionate increase in all inputs. Importantly, many knowledge markets exhibit diseconomies of scale---measures of market health (e.g., the percentage of questions with an accepted answer) decrease as a function of number of participants. The implications of our work are two-fold: site operators ought to design incentives as a function of system size (number of participants); the market lens should shed insight into complex dependencies amongst different content types and participant actions in general social networks.

Motivation & Objective

• To understand why large-scale online knowledge markets, such as StackExchange, can become unhealthy despite growth in user base.
• To model content generation dynamics in community question answering (CQA) platforms using economic production functions.
• To identify whether and how market health metrics deteriorate as system size increases.
• To provide site operators with insights for designing incentives that scale with community size.

Proposed method

• Models content generation on StackExchange as a production process using economic production functions.
• Tests multiple functional forms (e.g., Cobb-Douglas, linear, power) to find the best fit for predicting content output (questions, answers, comments).
• Employs a prediction task on empirical data from 125 StackExchange sites to identify the optimal model structure.
• Uses the Cobb-Douglas model to analyze elasticity, diminishing returns, and returns to scale in content production.
• Analyzes user participation patterns, including power-law distributions of engagement, to explain scaling failures.
• Measures market health using metrics like the percentage of questions with accepted answers and answers per question.

Experimental results

Research questions

• RQ1 How does the size of a knowledge market affect its health and content production efficiency?
• RQ2 What economic production model best explains content generation dynamics in large-scale CQA platforms?
• RQ3 Do factors like user participation and content dependencies exhibit constant elasticity or diminishing returns in knowledge markets?
• RQ4 Do knowledge markets exhibit diseconomies of scale, where larger size leads to declining health metrics?
• RQ5 How does the distribution of user engagement (especially high-engagement users) scale with system size?

Key findings

• The Cobb-Douglas model provides the best fit for predicting content generation across 125 StackExchange sites, indicating constant elasticity in user participation and content dependencies.
• Many StackExchange markets exhibit diseconomies of scale, with key health metrics like the percentage of questions with accepted answers declining as user count increases.
• Diminishing returns are observed: incremental increases in active answerers lead to progressively smaller gains in answer production.
• A power-law distribution of user participation persists, but the exponent decreases with system size, indicating later users contribute less intensely than early adopters.
• A stable core of high-engagement users exists, but their proportion does not scale with total user count, creating a supply-demand gap in market health.
• The model reveals that market health is not inherently improved by growth—large markets can become less efficient and less responsive over time.