Skip to content

kwatcharasupat/bandit

Repository files navigation

BandIt: Cinematic Audio Source Separation

Code for "A Generalized Bandsplit Neural Network for Cinematic Audio Source Separation" by Karn N. Watcharasupat, Chih-Wei Wu, Yiwei Ding, Iroro Orife, Aaron J. Hipple, Phillip A. Williams, Scott Kramer, Alexander Lerch, William Wolcott. [open-access paper]

Cinematic audio source separation is a relatively new subtask of audio source separation, with the aim of extracting the dialogue, music, and effects stems from their mixture. In this work, we developed a model generalizing the Bandsplit RNN for any complete or overcomplete partitions of the frequency axis. Psychoacoustically motivated frequency scales were used to inform the band definitions which are now defined with redundancy for more reliable feature extraction. A loss function motivated by the signal-to-noise ratio and the sparsity-promoting property of the 1-norm was proposed. We additionally exploit the information-sharing property of a common-encoder setup to reduce computational complexity during both training and inference, improve separation performance for hard-to-generalize classes of sounds, and allow flexibility during inference time with detachable decoders. Our best model sets the state of the art on the Divide and Remaster dataset with performance above the ideal ratio mask for the dialogue stem.

For the query-based music source separation model, Banquet, go here.

For Demo

Go here for demo of selected models using the first 10 files from DnR test set. Go here for exhaustive inference on the entire DnR test set for selected models.

For Replication

  • Install required dependencies from environment.yaml.
  • Obtain DnR dataset from here and MUSDB18-HQ from here.
  • Run each dataset's respective proprocess.py.
  • python train.py expt/path-to-the-desired-experiment.yaml.
  • python test.py expt/path-to-the-desired-experiment.yaml --ckpt_path=path/to/checkpoint-from-training.ckpt.

For Inference

  • Get the checkpoints from Zenodo.
  • Get the corresponding yaml config file from expt.
  • Put the checkpoint and the yaml config file into the same subfolder. Rename the config file hparams.yaml.
  • If you run into CUDA OOM, try reducing the batch size in the inference config. Another way without changing the config itself is by setting the system.inference parameter to "file:$PROJECT_ROOT/configs/inference/default16.yaml", or default8.yaml.
  • If you run into a CPU OOM, this is probably due to the resampler. You might want to get your audio file to 44.1 kHz beforehand, especially if it's big. A fix is coming (soon??).
  • Please do not hesitate to report other OOM cases.
python inference.py inference \
  --ckpt_path=path/to/checkpoint.ckpt \
  --file_path=path/to/file.wav \
  --model_name=model_id

or

python inference.py inference_multiple \
  --ckpt_path=path/to/checkpoint.ckpt \
  --file_glob=path/to/glob/*.wav \
  --model_name=model_id

Complexity Benchmark

  • Intel Core i9-11900K CPU + NVIDIA GeForce RTX 3090 GPU.
  • Note that this is benchmarked on one 6-second chunk. The memory usage in practice will scale according to your inference batch size plus some OLA overhead.
Model Band GFlops Params (M) Peak Memory (MB) Batch per second (GPU) Batch per second (CPU) GPU speedup
BSRNN-GRU8 (per stem) Vocals V7 238.2 15.8 416.2 12.35 0.61 20.2
BSRNN-LSTM12 (per stem) Vocals V7 462.2 25.8 505.4 7.99 0.6 13.4
BandIt Bark 48 290.6 64.5 643 10.22 0.39 26.1
BandIt ERB 48 274.2 32.6 519.5 10.31 0.41 25
BandIt Mel 48 274.3 32.8 519.3 10.15 0.38 26.6
BandIt Music 48 274.7 33.5 524.2 10.22 0.43 23.5
BandIt TriBark 48 274.2 32.7 519.9 10.3 0.4 25.5
BandIt Bark 64 387.6 82.6 828.5 8.64 0.4 21.9
BandIt ERB 64 363.5 36 649.1 8.68 0.42 20.7
BandIt Mel 64 363.6 36.1 648.9 8.71 0.32 27.2
BandIt Music 64 364.1 37 653 8.69 0.31 27.7
BandIt TriBark 64 363.5 36 648.7 8.68 0.42 20.6
BandIt Vocals V7 243.2 25.7 454 11.34 0.6 18.8
Hybrid Demucs 85 83.6 552.5 17.04 1.1 15.5
Open-Unmix 5.7 22.1 187.8 52.5 20.77 2.5

Citation

@article{Watcharasupat2023Bandit
  author={Watcharasupat, Karn N. and Wu, Chih-Wei and Ding, Yiwei and Orife, Iroro and Hipple, Aaron J. and Williams, Phillip A. and Kramer, Scott and Lerch, Alexander and Wolcott, William},
  journal={IEEE Open Journal of Signal Processing}, 
  title={A Generalized Bandsplit Neural Network for Cinematic Audio Source Separation}, 
  year={2024},
  volume={5},
  number={},
  pages={73-81},
  doi={10.1109/OJSP.2023.3339428}}

About

BandIt: Cinematic Audio Source Separation

Resources

License

Stars

Watchers

Forks

Releases

No releases published

Sponsor this project

 

Packages

No packages published

Languages