[Paper Review] The Bach Doodle: Approachable music composition with machine learning at scale

Cheng-Zhi Anna Huang, Curtis Hawthorne+5 more|arXiv (Cornell University)|Jul 14, 2019

Music and Audio Processing32 references41 citations

TL;DR

The paper presents the Bach Doodle, an in-browser harmonization tool powered by Coconet, ported to TensorFlow.js with speedups, a small 400 KB model, and a large-scale public dataset of 21.6 million user-generated harmonizations and 55 million requests. It also analyzes parallel fifths/octaves and deployment choices between local and TPU-backed inference.

ABSTRACT

To make music composition more approachable, we designed the first AI-powered Google Doodle, the Bach Doodle, where users can create their own melody and have it harmonized by a machine learning model Coconet (Huang et al., 2017) in the style of Bach. For users to input melodies, we designed a simplified sheet-music based interface. To support an interactive experience at scale, we re-implemented Coconet in TensorFlow.js (Smilkov et al., 2019) to run in the browser and reduced its runtime from 40s to 2s by adopting dilated depth-wise separable convolutions and fusing operations. We also reduced the model download size to approximately 400KB through post-training weight quantization. We calibrated a speed test based on partial model evaluation time to determine if the harmonization request should be performed locally or sent to remote TPU servers. In three days, people spent 350 years worth of time playing with the Bach Doodle, and Coconet received more than 55 million queries. Users could choose to rate their compositions and contribute them to a public dataset, which we are releasing with this paper. We hope that the community finds this dataset useful for applications ranging from ethnomusicological studies, to music education, to improving machine learning models.

Motivation & Objective

• Demonstrate an accessible, scalable AI-powered music harmonization experience for a broad audience.
• Design a browser-based interface that lowers the barrier to entry for composing with machine learning.
• Deploy and optimize a neural counterpoint model (Coconet) for web and cloud runtimes at scale.
• Release a large, public dataset of user interactions to foster research in ethnomusicology, education, and ML improvement.

Proposed method

• Re-implement Coconet in TensorFlow.js to run in the browser and port to TPU servers for fallback computation.
• Apply dilated depth-wise separable convolutions and operation fusion to reduce inference latency.
• Compress model weights via post-training weight quantization to ~400 KB download size.
• Calibrate a browser-based speed test to choose local TF.js vs. remote TPU for harmonization requests.
• Collect and analyze a dataset of user interactions, including melodies, harmonizations, ratings, and metadata.

Experimental results

Research questions

• RQ1 How can a Bach-style harmonization model be made approachable and fast enough for large-scale browser-based interaction?
• RQ2 What interface and interaction design enable users with little musical training to input melodies and receive harmonizations effectively?
• RQ3 What are the trade-offs in running Coconet locally in the browser versus on TPU servers, in terms of latency and user experience?
• RQ4 How does the model perform in terms of stylistic fidelity (e.g., avoidance of parallel fifths/octaves) and user-rated quality in a real-world, diverse user base?

Key findings

• Coconet ported to TF.js achieves ~2s harmonization latency in-browser with dilated depthwise separable convolutions and operation fusion.
• Model download size reduced to ~400 KB via post-training weight quantization.
• Hybrid execution strategy: ~47.4% of harmonizations run locally; remaining requests served by TPU backends.
• Over 21.8 million analyzed harmonizations show P5s and P8s occur at 0.365 and 0.391 per measure on average, correlated with input distribution and user feedback.
• Across 55 million requests, users produced 21.6 million unique sequences across 8.5 million sessions; 53.8% of harmonizations rated as Good.
• Dataset release (CC-license) enables ethnomusicology, education, and ML research applications.