BeatNet is the state-of-the-art AI-based Python library for joint music beat, downbeat, tempo, and meter tracking. This repo includes the BeatNet neural structure along with the efficient two-stage cascade particle filtering algorithm that is proposed in the paper. It offers four distinct working modes, as follows:
- Streaming mode: This mode captures streaming audio directly from the microphone.
- Real-time mode: In this mode, audio files are read and processed in real-time, yielding immediate results.
- Online mode: Similar to Real-time mode, Online mode employs the same causal algorithm for track processing. However, rather than reading the files in real-time, it reads them faster, while still producing identical outcomes to the real-time mode.
- Offline mode: Inferes beats and downbeats in an offline fashion.
To gain a better understanding of each mode, please refer to the Usage examples provided in this document.
This repository contains the user package and the source code of the Monte Carlo particle flitering inference model of the "BeatNet" music online joint beat/downbeat/tempo/meter tracking system. The arxiv version of the original ISMIR-2021 paper:
In addition to the proposed online inference, we added madmom's DBN beat/downbeat inference model for the offline usages. Note that, the offline model still utilize BeatNet's neural network rather than that of Madmom which leads to better performance and significantly faster results.
Note: All models are trained using pytorch and are included in the models folder. In order to recieve the training script and the datasets data/feature handlers, shoot me an email at mheydari [at] ur.rochester.edu
Raw audio waveform object or directory.
- By using the audio directory as the system input, the system automatically resamples the audio file to 22050 Hz. However, in the case of using an audio object as the input, make sure that the audio sample rate is equal to 22050 Hz.
A vector including beats and downbeats columns, respectively with the following shape: numpy_array(num_beats, 2).
model
: An scalar in the range [1,3] to select which pre-trained CRNN models to utilize.
mode
: An string to determine the working mode, of whcih only offline
is supproted
inference model
: A string to choose the inference approach. i.e. 'PF' standing for Particle Filtering for causal inferences and 'DBN' standing for Dynamic Bayesian Network for non-causal usages.
plot
: A list of strings to plot. It can include 'activations', 'beat_particles' and 'downbeat_particles'
Note that to speed up plotting the figures, rather than new plots per frame, the previous plots get updated. However, to secure realtime results, it is recommended to not plot or have as less number of plots as possible at the time.
thread
: To decide whether accomplish the inference at the main thread or another thread.
device
: Type of device being used. Cuda or cpu (by default).
batch_size
: The number of files to be processed at once. Alternatively, the size of the first dimension for any torch.Tensor
input.
Approach #1: Installing binaries from the pypi website:
pip install BeatNet
Approach #2: Installing directly from the Git repository:
pip install git+https://github.com/mjhydri/BeatNet
- Note: Before installing the BeatNet make sure Librosa and Madmom packages are installed. Also, pyaudio is a python binding for Portaudio to handle audio streaming. If Pyaudio is not installed in your machine, depending on your machine type either install it thorugh pip (Mac OS and Linux) or download an appropriate version for your machine (Windows) from here. Then, navigate to the file location through commandline and use the following command to install the wheel file locally:
pip install <Pyaduio_file_name.whl>
Can be used on audio files or torch.Tensors
.
from BeatNet.BeatNet import BeatNet
files = [
"path/to/audio.mp3",
"path/to/more.mp3"
]
num_files = 2
estimator = BeatNet(1, mode='offline', inference_model='DBN', plot=[], thread=false, batch_size=num_files)
beats = estimator.process(files)
from BeatNet.BeatNet import BeatNet
from BeatNet.utils import zero_pad_cat
files = [
"path/to/audio.mp3",
"path/to/more.mp3"
]
num_files = len(files)
# load audio to memory
audio = list()
sample_rates = list()
for file in files:
signal, sample_rate = torchaudio.load(file)
signal = torch.unsqueeze(torch.mean(signal, dim=0), dim=0)
audio.append(signal)
sample_rates.append(sample_rate)
# check sample rates
if not all(sample_rates[0] == rate for rate in sample_rates):
ValueError("All samplerates must be the same.")
else:
sample_rate = sample_rates[0]
# stack audio by zero padding into a [B, N] tensor,
# where B is the number of files and N is the number of samples
audio = zero_pad_cat(audio)
# spawn estimator
estimator = BeatNet(1, mode='offline', inference_model='DBN', plot=[], thread=false, batch_size=num_files)
# infer beats
beats = estimator.get_beats(audio=audio, sample_rate=sample_rate)
For the input feature extraction and the raw state space generation, Librosa and Madmom libraries are ustilzed respectively. Many thanks for their great jobs. This work has been partially supported by the National Science Foundation grants 1846184 and DGE-1922591.
@inproceedings{heydari2021beatnet,
title={BeatNet: CRNN and Particle Filtering for Online Joint Beat Downbeat and Meter Tracking},
author={Heydari, Mojtaba and Cwitkowitz, Frank and Duan, Zhiyao},
journal={22th International Society for Music Information Retrieval Conference, ISMIR},
year={2021}
}
@inproceedings{heydari2021don,
title={Don’t look back: An online beat tracking method using RNN and enhanced particle filtering},
author={Heydari, Mojtaba and Duan, Zhiyao},
booktitle={ICASSP 2021-2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)},
pages={236--240},
year={2021},
organization={IEEE}
}