[Paper Review] Symbolic Music Genre Transfer with CycleGAN

Gino Brunner, Yuyi Wang+2 more|arXiv (Cornell University)|Sep 20, 2018

Music and Audio Processing4 citations

TL;DR

This paper proposes a CycleGAN-based approach for symbolic music genre transfer, adapting image-style domain transfer to music by preserving melodic structure while transforming genre characteristics. The key contribution is the introduction of additional discriminators that enhance structural fidelity, resulting in perceptible, harmonious genre transfers that are both audible to humans and quantitatively measurable via genre classifiers.

ABSTRACT

Deep generative models such as Variational Autoencoders (VAEs) and Generative Adversarial Networks (GANs) have recently been applied to style and domain transfer for images, and in the case of VAEs, music. GAN-based models employing several generators and some form of cycle consistency loss have been among the most successful for image domain transfer. In this paper we apply such a model to symbolic music and show the feasibility of our approach for music genre transfer. Evaluations using separate genre classifiers show that the style transfer works well. In order to improve the fidelity of the transformed music, we add additional discriminators that cause the generators to keep the structure of the original music mostly intact, while still achieving strong genre transfer. Visual and audible results further show the potential of our approach. To the best of our knowledge, this paper represents the first application of GANs to symbolic music domain transfer.

Motivation & Objective

• To enable deep neural network-based domain transfer in symbolic music, specifically from one genre to another, using generative models.
• To preserve the original melody and structural integrity of music during genre transfer, ensuring recognizability of the source piece.
• To improve generation quality by introducing auxiliary discriminators that regularize the generator to stay on the 'music manifold'.
• To demonstrate that GAN-based music style transfer is feasible and perceptible to human listeners, not just neural classifiers.
• To provide a reproducible framework with code and data to support future research in music generation and style transfer.

Proposed method

• Adapt the CycleGAN architecture to symbolic music by using MIDI representations as input, with pitch as the primary transformation variable.
• Introduce additional discriminators that assess the realism of generated music not only in the target domain but also in terms of structural preservation of the input.
• Apply cycle consistency loss to ensure that the transformation from source to target and back to source remains consistent, preserving content.
• Use separate genre classifiers to quantitatively evaluate the success of genre transfer, measuring the model's ability to map inputs to the target genre.
• Train the model end-to-end using adversarial loss, cycle consistency loss, and perceptual losses from additional discriminators to balance style transfer and structural fidelity.
• Fine-tune the trade-off between genre transfer strength and content preservation using hyperparameters controlling the influence of the additional discriminators.

Experimental results

Research questions

• RQ1 Can CycleGAN be successfully adapted to perform symbolic music genre transfer, maintaining structural integrity while transforming genre characteristics?
• RQ2 How do additional discriminators affect the balance between genre transfer strength and preservation of the original music's structure?
• RQ3 To what extent can the genre transfer be perceived by human listeners, not just by neural network classifiers?
• RQ4 How does the performance of the model vary across different genre pairs, such as Jazz vs. Classic or Jazz vs. Pop?
• RQ5 What is the impact of architectural modifications, such as the addition of discriminators, on the perceptual quality and harmonic coherence of generated music?

Key findings

• The model with full additional discriminators (M_full) achieved the highest genre transfer performance, with a total domain transfer score of 58.1% on the Classic-Pop pair, outperforming M_base and M_partial.
• On the Jazz-Pop pair, M_full achieved a 49.24% transfer rate from Pop to Jazz, indicating effective genre transfer despite lower overall scores.
• Human listeners could perceive genre changes in audio samples, confirming that the transfer is not only detectable by classifiers but also audible to untrained ears.
• The addition of discriminators significantly improved structural preservation: M_full consistently produced more coherent and harmonious music than M_base, especially in complex transfers like Classic to Jazz.
• Cycle consistency and perceptual regularization led to fewer dissonant notes and rhythmic inconsistencies, resulting in more musically plausible outputs.
• The model's performance was quantitatively validated using genre classifiers, with M_full showing the strongest transfer capability (e.g., 87.18% accuracy in A→B→A cycle on Classic-Pop pair).