[Paper Review] Getting Closer to the Essence of Music: The Con Espressione Manifesto

Gerhard Widmer|arXiv (Cornell University)|Nov 29, 2016

Music and Audio Processing67 references19 citations

TL;DR

This manifesto calls for a paradigm shift in Music Information Research (MIR) toward deeper musical understanding by centering human perception, expressivity, and performance. It proposes advancing computational models of musical structure, expressive performance, and listener perception through large-scale machine learning and interdisciplinary collaboration, with the goal of enabling musically intelligent systems that can interact sympathetically with human performers.

ABSTRACT

This text offers a personal and very subjective view on the current situation of Music Information Research (MIR). Motivated by the desire to build systems with a somewhat deeper understanding of music than the ones we currently have, I try to sketch a number of challenges for the next decade of MIR research, grouped around six simple truths about music that are probably generally agreed on, but often ignored in everyday research.

Motivation & Objective

• To address the lack of deep musical understanding in current MIR systems, which often fail to perceive or convey expressive qualities like emotion, surprise, or flow.
• To reorient MIR research toward the core principles of music as a perceptual, temporal, and expressive process shaped by human listeners and performers.
• To advance computational models of musical structure, performance expression, and listener perception through interdisciplinary research and large-scale data.
• To develop systems capable of musically intelligent interaction—such as accompanying live performers with expressive sensitivity—by integrating perception, cognition, and performance science.
• To inspire the broader MIR community to prioritize expressive and perceptual dimensions in future research agendas, beyond technical signal analysis.

Proposed method

• Developing description frameworks to categorize and characterize expressive dimensions in music performance, including timing, dynamics, articulation, and intonation.
• Advancing audio-based extraction of expressive performance parameters beyond basic onsets and dynamics, using sophisticated signal processing and machine learning.
• Combining MIR advances with information-theoretic models and unsupervised learning to model structure perception at both score and audio levels.
• Creating discriminative models that recognize intended expressive messages in performances using large curated corpora.
• Building predictive models that generate or modify performances to express specific emotional or stylistic qualities.
• Designing and deploying interactive demonstrators, such as the Compassionate Music Companion, to test real-time, expressive musical interaction.

Experimental results

Research questions

• RQ1 How can computational models better capture the non-Markovian, temporally extended nature of musical perception and structure?
• RQ2 To what extent can machine learning systems recognize and classify expressive character in music performances, such as 'flowing' or 'dramatic', across different genres and performers?
• RQ3 How can MIR systems learn to anticipate and adapt to a human performer’s expressive style in real time during live musical interaction?
• RQ4 What expressive features—beyond timing and dynamics—most strongly influence listeners’ perception of musical character and emotion?
• RQ5 How can large-scale, curated corpora of expressive performances be used to train models that understand the relationship between musical structure and expressive expression?

Key findings

• Current MIR systems lack the ability to distinguish between musically interesting and monotonous pieces based on perceptual principles like redundancy and unpredictability.
• Computers cannot yet identify or classify expressive qualities such as 'relaxed' or 'flowing' in performances, even when these are perceptually salient to human listeners.
• There is a significant gap in modeling how musical structure and expressive performance jointly shape the perceived character of a piece, especially in classical music.
• The Con Espressione project has initiated a crowdsourced game to assess consensus in human perception of expressive qualities across different performances of the same piece, revealing initial patterns in perceptual agreement.
• The Compassionate Music Companion, a key demonstrator of the project, aims to enable real-time, expressive musical interaction by recognizing and adapting to a performer’s expressive style.
• The manifesto identifies six core 'truths' about music—its temporal nature, non-Markovian structure, listener-centered perception, learned appreciation, performance dependence, and expressivity—as foundational for future MIR research.