[Paper Review] Structured Interactive Music Scores

Mauricio Toro|arXiv (Cornell University)|Aug 23, 2015

Music and Audio ProcessingComputer Science15 references21 citations

TL;DR

This paper proposes a formal framework for structured interactive music scores using Non-deterministic Timed Concurrent Constraint (ntcc) calculus to model temporal relations and global constraints in multimedia scenarios. It integrates ntcc for control logic with Faust for real-time audio/video processing, aiming to enable formal verification of behavioral properties while maintaining real-time performance through hybrid system semantics and verification techniques.

ABSTRACT

Interactive Scores is a formalism for the design and performance of interactive scenarios that provides temporal relations (TRs) among the objects of the scenario. We can model TRs among objects in Time Stream Petri nets, but it is difficult to represent global constraints. This can be done explicitly in the Non-deterministic Timed Concurrent Constraint (ntcc) calculus. We want to formalize a heterogeneous system that controls in one subsystem the concurrent execution of the objects using ntcc, and audio and video processing in the other. We also plan to develop an automatic verifier for ntcc.

Motivation & Objective

• To formalize a heterogeneous system combining ntcc for control logic, Faust for audio/video processing, and a media loader for file handling.
• To address the challenge of verifying temporal properties in real-time interactive music systems where traditional ntcc verification is too slow.
• To develop a formal semantics for a system that synchronizes concurrent multimedia objects with global constraints.
• To enable automatic verification of properties such as reachability and liveness in interactive music scenarios.
• To bridge the gap between formal verification (via ntcc) and efficient real-time execution (via Faust plugins)

Proposed method

• Model interactive music scenarios using the ntcc calculus to express temporal relations and global constraints.
• Integrate Faust as a real-time signal processing component to replace computationally heavy ntcc processes.
• Define a heterogeneous system semantics combining ntcc (control), Faust (signal processing), and a media loader (file I/O).
• Leverage the linear temporal logic interpretation of ntcc to formally specify and verify temporal properties.
• Use model checking techniques to enable automatic verification of system properties in practical subsets of ntcc.
• Design a hybrid architecture where verified ntcc models guide Faust plugin behavior, ensuring property preservation.

Experimental results

Research questions

• RQ1 How can a heterogeneous system be formally specified to combine ntcc-based control, Faust-based signal processing, and media file handling?
• RQ2 Can Faust plugins correctly preserve the temporal properties formally verified in ntcc models?
• RQ3 Is it possible to achieve real-time performance in ntcc-based interactive music systems without sacrificing formal verification?
• RQ4 What formal semantics can unify concurrent control, global constraints, and real-time audio/video streams?
• RQ5 How can automatic verification tools be developed for useful subsets of ntcc to support practical music application development?

Key findings

• The integration of Faust with ntcc enables real-time audio processing, overcoming the performance limitations of pure ntccrt execution.
• The proposed framework allows formal verification of temporal properties such as '4 time units after event e_A, stream B is the gain-filtered output of stream A'.
• A prototype score with 500 temporal objects achieved an average time-unit duration of 30 ms, demonstrating compatibility with real-time interaction.
• The use of model checking on practical subsets of ntcc offers a feasible path toward automatic verification of interactive music systems.
• The formal semantics of the heterogeneous system enable precise specification of global constraints and synchronization across multimedia objects.
• The framework supports complex interactive features like conditional branching and loops, extending beyond basic temporal constraints.