[Paper Review] Deep Layered Learning in MIR

Anders Elowsson|arXiv (Cornell University)|Apr 18, 2018

Music and Audio Processing51 references3 citations

TL;DR

This paper proposes Deep Layered Learning (DLL), a modular, hierarchical deep learning framework for music information retrieval (MIR) that uses intermediate representations—such as pitch, chords, and rhythm—as priors to improve generalization and structural invariance. By organizing learning modules in a directed acyclic graph and guiding end-to-end training with intermediate supervisory signals, DLL enhances pitch tracking, particularly in offset detection, when guided by framewise fundamental frequency estimates.

ABSTRACT

Deep learning has boosted the performance of many music information retrieval (MIR) systems in recent years. Yet, the complex hierarchical arrangement of music makes end-to-end learning hard for some MIR tasks - a very deep and flexible processing chain is necessary to model some aspect of music audio. Representations involving tones, chords, and rhythm are fundamental building blocks of music. This paper discusses how these can be used as intermediate targets and priors in MIR to deal with structurally complex learning problems, with learning modules connected in a directed acyclic graph. It is suggested that this strategy for inference, referred to as deep layered learning (DLL), can help generalization by (1) - enforcing the validity and invariance of intermediate representations during processing, and by (2) - letting the inferred representations establish the musical organization to support higher-level invariant processing. A background to modular music processing is provided together with an overview of previous publications. Relevant concepts from information processing, such as pruning, skip connections, and performance supervision are reviewed within the context of DLL. A test is finally performed, showing how layered learning affects pitch tracking. It is indicated that especially offsets are easier to detect if guided by extracted framewise fundamental frequencies.

Motivation & Objective

• Address the challenge of modeling complex, hierarchical music structures in end-to-end MIR systems.
• Improve generalization and structural invariance in music representation learning by enforcing valid intermediate representations.
• Integrate modular processing chains with musical priors (e.g., pitch, chords, rhythm) to guide higher-level inference.
• Explore the role of pruning, skip connections, and performance supervision in layered deep learning architectures.
• Demonstrate the effectiveness of DLL in a pitch tracking task, particularly for offset detection.

Proposed method

• Design a directed acyclic graph (DAG) of learning modules, each responsible for extracting a specific musical representation (e.g., pitch, chords, rhythm).
• Use intermediate representations as supervisory targets during training to enforce validity and invariance.
• Integrate skip connections and pruning mechanisms to stabilize training and improve gradient flow.
• Apply performance supervision to align model outputs with perceptually relevant musical structures.
• Train the system end-to-end while maintaining modular structure, allowing for incremental refinement of representations.
• Use framewise fundamental frequency estimates as a guiding signal for offset detection in the final stage.

Experimental results

Research questions

• RQ1 Can hierarchical, modular deep learning improve generalization in music information retrieval tasks?
• RQ2 How does enforcing validity and invariance in intermediate representations affect model performance?
• RQ3 To what extent does using musical priors (e.g., pitch, chords, rhythm) enhance structural understanding in MIR?
• RQ4 How do architectural components like skip connections and pruning impact training stability and accuracy in DLL?
• RQ5 Does guiding offset detection with extracted fundamental frequencies improve pitch tracking performance?

Key findings

• Deep Layered Learning (DLL) improves pitch tracking performance by leveraging intermediate representations as priors.
• Offset detection becomes significantly more accurate when guided by extracted framewise fundamental frequencies.
• Enforcing invariance and validity in intermediate representations leads to more robust and generalizable models.
• The use of skip connections and pruning enhances training stability and gradient flow in the layered architecture.
• Modular, hierarchical design allows for better interpretability and incremental refinement of musical representations.
• The framework demonstrates measurable gains in pitch tracking, especially in challenging offset detection tasks.