F5-TTS: Diffusion Transformer with ConvNeXt V2, faster trained and inference.
E2 TTS: Flat-UNet Transformer, closest reproduction.
Sway Sampling: Inference-time flow step sampling strategy, greatly improves performance

Clone this repository.

git clone [email protected]:SWivid/F5-TTS.git
cd F5-TTS

Install packages.

pip install -r requirements.txt

Example data processing scripts for Emilia and Wenetspeech4TTS, and you may tailor your own one along with a Dataset class in model/dataset.py.

# prepare custom dataset up to your need
# download corresponding dataset first, and fill in the path in scripts

# Prepare the Emilia dataset
python scripts/prepare_emilia.py

# Prepare the Wenetspeech4TTS dataset
python scripts/prepare_wenetspeech4tts.py

Once your datasets are prepared, you can start the training process.

# setup accelerate config, e.g. use multi-gpu ddp, fp16
# will be to: ~/.cache/huggingface/accelerate/default_config.yaml     
accelerate config
accelerate launch test_train.py

To inference with pretrained models, download the checkpoints from 🤗.

You can test single inference using the following command. Before running the command, modify the config up to your need.

# modify the config up to your need,
# e.g. fix_duration (the total length of prompt + to_generate, currently support up to 30s)
#      nfe_step     (larger takes more time to do more precise inference ode)
#      ode_method   (switch to 'midpoint' for better compatibility with small nfe_step, )
#                   ( though 'midpoint' is 2nd-order ode solver, slower compared to 1st-order 'Euler')
python test_infer_single.py

To test speech editing capabilities, use the following command.

python test_infer_single_edit.py

1. Seed-TTS test set: Download from seed-tts-eval.
2. LibriSpeech test-clean: Download from OpenSLR.
3. Unzip the downloaded datasets and place them in the data/ directory.
4. Update the path for the test-clean data in test_infer_batch.py
5. Our filtered LibriSpeech-PC 4-10s subset is already under data/ in this repo

To run batch inference for evaluations, execute the following commands:

# batch inference for evaluations
accelerate config  # if not set before
bash test_infer_batch.sh

1. Chinese ASR Model: Paraformer-zh
2. English ASR Model: Faster-Whisper
3. WavLM Model: Download from Google Drive.

Some Notes
For faster-whisper with CUDA 11:
pip install --force-reinstall ctranslate2==3.24.0
(Recommended) To avoid possible ASR failures, such as abnormal repetitions in output:
pip install faster-whisper==0.10.1

Update the path with your batch-inferenced results, and carry out WER / SIM evaluations:

# Evaluation for Seed-TTS test set
python scripts/eval_seedtts_testset.py

# Evaluation for LibriSpeech-PC test-clean (cross-sentence)
python scripts/eval_librispeech_test_clean.py

• E2-TTS brilliant work, simple and effective
• Emilia, WenetSpeech4TTS valuable datasets
• lucidrains initial CFM structure with also bfs18 for discussion
• SD3 & Huggingface diffusers DiT and MMDiT code structure
• torchdiffeq as ODE solver, Vocos as vocoder
• mrfakename huggingface space demo ~
• FunASR, faster-whisper & UniSpeech for evaluation tools
• ctc-forced-aligner for speech edit test

@article{chen-etal-2024-f5tts,
      title={F5-TTS: A Fairytaler that Fakes Fluent and Faithful Speech with Flow Matching}, 
      author={Yushen Chen and Zhikang Niu and Ziyang Ma and Keqi Deng and Chunhui Wang and Jian Zhao and Kai Yu and Xie Chen},
      journal={arXiv preprint arXiv:2410.06885},
      year={2024},
}

Our code is released under MIT License.