[Paper Review] Online Evaluation for Effective Web Service Development

Roman Budylin, Alexey Drutsa+3 more|arXiv (Cornell University)|Sep 3, 2018

Service-Oriented Architecture and Web Services51 references4 citations

TL;DR

This paper presents a comprehensive tutorial on online evaluation for web service development, focusing on A/B testing and online controlled experiments in industrial settings. It covers statistical foundations, metric design, experimentation pipelines, and machine learning-enhanced techniques—offering practical guidance and state-of-the-art methods to improve the efficiency, accuracy, and scalability of online evaluation in large-scale web platforms.

ABSTRACT

Development of the majority of the leading web services and software products today is generally guided by data-driven decisions based on evaluation that ensures a steady stream of updates, both in terms of quality and quantity. Large internet companies use online evaluation on a day-to-day basis and at a large scale. The number of smaller companies using A/B testing in their development cycle is also growing. Web development across the board strongly depends on quality of experimentation platforms. In this tutorial, we overview state-of-the-art methods underlying everyday evaluation pipelines at some of the leading Internet companies. Software engineers, designers, analysts, service or product managers --- beginners, advanced specialists, and researchers --- can learn how to make web service development data-driven and do it effectively.

Motivation & Objective

• To provide a practical, data-driven framework for web service development using online evaluation and A/B testing.
• To address the challenges of designing effective evaluation metrics that are both directionally accurate and sensitive to real user impact.
• To share industrial best practices and pitfalls in large-scale experimentation pipelines at leading internet companies.
• To introduce advanced techniques—especially machine learning-based methods—for improving the efficiency and sensitivity of online experiments.
• To highlight open research challenges in online experimentation, including heterogeneous treatment effects and sequential testing.

Proposed method

• Uses statistical foundations including p-values, type I/II errors, and hypothesis testing (t-test, Mann-Whitney U, bootstrap) to validate experimental results.
• Introduces key components of A/B testing: control vs. treatment groups, key metrics, OEC (Overall Evaluation Criterion), and OAC (Overall Acceptance Criterion).
• Proposes metric design principles emphasizing directionality and sensitivity to detect meaningful changes in user behavior.
• Applies variance reduction techniques such as regression adjustment, stratification, and gradient boosted trees to improve metric precision.
• Employs Optimal Distribution Decomposition (ODD) to detect differences between treatment and control distributions beyond mean shifts.
• Introduces methods for early stopping and optimal scheduling of experiments to reduce duration and resource use while controlling type I error.

Experimental results

Research questions

• RQ1 How can evaluation metrics be designed to reliably reflect user experience changes and avoid manipulation or gaming?
• RQ2 What are the key statistical and methodological pitfalls in large-scale online experimentation, and how can they be mitigated?
• RQ3 How can machine learning techniques improve the sensitivity and efficiency of online A/B testing beyond traditional methods?
• RQ4 In what ways can distribution-level differences between treatment and control groups be detected and leveraged for better evaluation?
• RQ5 How can online experimentation pipelines be optimized for scalability, speed, and robustness in industrial settings?

Key findings

• The use of directionally sound and sensitive metrics significantly improves the reliability of online evaluation, reducing false conclusions about service changes.
• Regression adjustment and gradient boosted trees reduce metric variance, enabling earlier detection of treatment effects with fewer users.
• The ODD method detects distributional differences beyond mean shifts, increasing sensitivity to complex user behavior changes.
• Learning metric combinations improves sensitivity in large-scale experiments, as demonstrated in empirical evaluations.
• Optimal scheduling and early stopping procedures reduce experiment duration while maintaining statistical validity and controlling type I error.
• Industrial-scale experimentation pipelines at companies like Yandex, Bing, and Google run hundreds to over a thousand A/B tests daily, demonstrating the scalability of the framework.