[Paper Review] BeatNet: CRNN and Particle Filtering for Online Joint Beat Downbeat and Meter Tracking

Mojtaba Heydari, Frank Cwitkowitz+1 more|arXiv (Cornell University)|Aug 8, 2021

Music and Audio Processing4 citations

TL;DR

BeatNet proposes an online, causal system for joint beat, downbeat, and meter tracking using a CRNN for frame-level activation estimation and two-stage particle filtering for temporal inference. It achieves state-of-the-art online performance with a novel information gate that reduces particle filtering computation by skipping unnecessary resampling steps.

ABSTRACT

The online estimation of rhythmic information, such as beat positions, downbeat positions, and meter, is critical for many real-time music applications. Musical rhythm comprises complex hierarchical relationships across time, rendering its analysis intrinsically challenging and at times subjective. Furthermore, systems which attempt to estimate rhythmic information in real-time must be causal and must produce estimates quickly and efficiently. In this work, we introduce an online system for joint beat, downbeat, and meter tracking, which utilizes causal convolutional and recurrent layers, followed by a pair of sequential Monte Carlo particle filters applied during inference. The proposed system does not need to be primed with a time signature in order to perform downbeat tracking, and is instead able to estimate meter and adjust the predictions over time. Additionally, we propose an information gate strategy to significantly decrease the computational cost of particle filtering during the inference step, making the system much faster than previous sampling-based methods. Experiments on the GTZAN dataset, which is unseen during training, show that the system outperforms various online beat and downbeat tracking systems and achieves comparable performance to a baseline offline joint method.

Motivation & Objective

• To develop a real-time, online system for joint beat, downbeat, and meter tracking without requiring prior time signature input.
• To address the computational inefficiency of particle filtering in online rhythmic structure estimation.
• To improve online performance in beat and downbeat tracking while maintaining causal inference.
• To enable meter estimation and dynamic adjustment during inference without priming.

Proposed method

• A causal convolutional-recurrent neural network (CRNN) processes audio frames to generate beat and downbeat activation scores.
• Two sequential Monte Carlo particle filters are used: one for beat tracking and one for downbeat tracking, both estimating tempo and rhythmic structure over time.
• An information gate mechanism selectively skips resampling steps in the particle filter based on prediction confidence, reducing computational cost.
• The system jointly estimates beat positions, downbeats, and time signature (meter) in real time using only past and present audio features.
• The particle filters are initialized with diverse hypotheses and updated iteratively using audio observations and transition models.
• The model is trained end-to-end on audio features and evaluated on unseen data, with inference performed in a causal, streaming fashion.

Experimental results

Research questions

• RQ1 Can a fully online system jointly estimate beat, downbeat, and meter without requiring a known time signature at inference time?
• RQ2 How can particle filtering be optimized for real-time rhythmic tracking to reduce computational cost?
• RQ3 Does a CRNN-based activation estimator outperform prior models in online beat and downbeat tracking?
• RQ4 Can an information gate mechanism significantly reduce particle filtering inference time without degrading tracking accuracy?

Key findings

• BeatNet outperforms all other online beat and downbeat tracking systems on the GTZAN dataset, achieving 80.64% beat tracking F1 and 54.07% downbeat tracking F1.
• The system achieves comparable performance to an offline baseline (Böck [20]) despite being causal and not requiring time signature input.
• The information gate reduces particle filtering inference time by up to 75% compared to prior sampling-based methods, cutting processing time from 21.30s to 8.87s for a 30-second excerpt.
• The model dynamically estimates and adjusts meter during inference, eliminating the need for priming with time signature.
• BeatNet's performance is robust across diverse musical styles, as shown on the unseen GTZAN dataset.
• The joint estimation framework shows that cascade error propagation from beat to downbeat tracking is manageable, with only marginal performance loss compared to offline methods.