[Paper Review] The Cost of Collaboration for Code and Art: Evidence from a Remixing Community

Benjamin Mako Hill, Andrés Monroy‐Hernández|arXiv (Cornell University)|Jul 5, 2015

Digital Games and Media41 references20 citations

TL;DR

This study analyzes collaboration in Scratch, a youth-oriented online community for creating and remixing digital projects, using peer ratings to assess quality. It finds that collaborative projects are rated lower than individual ones, but code-intensive collaborations outperform media-heavy ones, challenging assumptions about collaboration improving quality in creative domains.

ABSTRACT

In this paper, we use evidence from a remixing community to evaluate two pieces of common wisdom about collaboration. First, we test the theory that jointly produced works tend to be of higher quality than individually authored products. Second, we test the theory that collaboration improves the quality of functional works like code, but that it works less well for artistic works like images and sounds. We use data from Scratch, a large online community where hundreds of thousands of young users share and remix millions of animations and interactive games. Using peer-ratings as a measure of quality, we estimate a series of fitted regression models and find that collaborative Scratch projects tend to receive ratings that are lower than individually authored works. We also find that code-intensive collaborations are rated higher than media-intensive efforts. We conclude by discussing the limitations and implications of these findings.

Motivation & Objective

• To test whether jointly produced works are higher quality than individually authored works in creative domains.
• To investigate whether collaboration enhances functional works (e.g., code) more than artistic works (e.g., images, sounds).
• To analyze the quality trade-offs in collaborative digital creation within a large-scale online community.
• To evaluate the role of content type—code vs. media—in shaping collaborative outcomes.
• To provide empirical evidence on the 'cost' of collaboration in creative and technical domains.

Proposed method

• Data was collected from Scratch, a global online community where users create and remix interactive projects.
• Peer ratings from the Scratch community were used as a proxy for project quality.
• Regression models were fitted to analyze the relationship between collaboration and quality, controlling for project type and other factors.
• Projects were classified as code-intensive or media-intensive based on content composition.
• The analysis compared individual-authored projects to collaborative ones, with separate models for code and media-heavy works.
• Statistical modeling assessed the impact of collaboration on peer ratings, using robust standard errors and model diagnostics.

Experimental results

Research questions

• RQ1 Do collaborative projects on Scratch receive higher peer ratings than individual projects?
• RQ2 Does collaboration improve the quality of code-intensive projects more than media-intensive projects?
• RQ3 Is there a measurable 'cost' to collaboration in artistic and creative domains?
• RQ4 How does the composition of a project (code vs. media) moderate the effect of collaboration on perceived quality?
• RQ5 To what extent do collaborative efforts in creative communities result in lower-quality outcomes compared to individual work?

Key findings

• Collaborative Scratch projects received significantly lower peer ratings compared to individual projects, indicating a measurable 'cost' of collaboration.
• Code-intensive collaborations were rated higher than media-intensive collaborations, suggesting that collaboration is more effective in functional domains.
• Media-heavy projects showed the most negative impact from collaboration, with ratings substantially lower than individual efforts.
• The overall effect of collaboration on quality was negative, contradicting the common belief that collaboration universally improves output.
• The quality gap between individual and collaborative projects was most pronounced in artistic and multimedia content.
• Regression models confirmed that collaboration had a statistically significant negative effect on perceived quality, even after controlling for project complexity and popularity.