This repository is the official PyTorch implementation of our ACL-2022 paper.
- You can download the dataset from here. Please send us an email for registration (See in apply_form).
- Dataset preview.
NeuralSVB does not need text as input, but the ASR model to extract PPG needs text. Thus we also provide the text labels of PopBuTFy.
Most of the required packages are in https://github.com/NATSpeech/NATSpeech/blob/main/requirements.txt
Or you can prepare environments with the Requirements.txt file in the repository directory.
pip install Requirements.txt
- Extract embeddings of vocal timbre:
CUDA_VISIBLE_DEVICES=0 python data_gen/tts/bin/binarize.py --config egs/datasets/audio/PopBuTFy/save_emb.yaml
- Pack the dataset:
CUDA_VISIBLE_DEVICES=0 python data_gen/tts/bin/binarize.py --config egs/datasets/audio/PopBuTFy/para_bin.yaml
We provide the pre-trained model of HifiGAN-Singing which is specially designed for SVS with NSF mechanism.
Please unzip pre-trained vocoder into checkpoints
before training your acoustic model.
This singing vocoder is trained on 100+ hours singing data (including Chinese and English songs).
We provide the pre-trained model of PPG Extractor.
Please unzip pre-trained PPG extractor into checkpoints
before training your acoustic model.
After the instructions above, the directory structure should be as follows:
.
|--data
|--processed
|--PopBuTFy (unzip PopBuTFy.zip)
|--data
|--directories containing wavs
|--binary
|--PopBuTFyENSpkEM
|--checkpoints
|--1009_pretrain_asr_english
|--
|--config.yaml
|--1012_hifigan_all_songs_nsf
|--
|--config.yaml
CUDA_VISIBLE_DEVICES=0,1 python tasks/run.py --config egs/datasets/audio/PopBuTFy/vae_global_mle_eng.yaml --exp_name exp_name --reset
CUDA_VISIBLE_DEVICES=0,1 python tasks/run.py --config egs/datasets/audio/PopBuTFy/vae_global_mle_eng.yaml --exp_name exp_name --reset --infer
Inference results will be saved in ./checkpoints/EXP_NAME/generated_
by default.
We provided:
- the pre-trained model of NSVB (en version);
Remember to put the pre-trained models in checkpoints
directory.
WIP.
See Appendix D "Limitations and Solutions" in our paper.
If this repository helps your research, please cite:
@inproceedings{liu-etal-2022-learning-beauty,
title = "Learning the Beauty in Songs: Neural Singing Voice Beautifier",
author = "Liu, Jinglin and
Li, Chengxi and
Ren, Yi and
Zhu, Zhiying and
Zhao, Zhou",
booktitle = "Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)",
month = may,
year = "2022",
address = "Dublin, Ireland",
publisher = "Association for Computational Linguistics",
url = "https://aclanthology.org/2022.acl-long.549",
pages = "7970--7983",}
- Before raising a issue, please check our Readme and other issues for possible solutions.
- We will try to handle your problem in time but we could not guarantee a satisfying solution.
- Please be friendly.
- r9y9's wavenet_vocoder
- Po-Hsun-Su's ssim
- descriptinc's melgan
- Official espnet
- Official PyTorch Lightning
The framework of this repository is based on DiffSinger, and is a predecessor of NATSpeech.