readme-en.md

English ｜ 中文

Video to Video Translation with Lip Synchronization

• Video to Video Translation
• Voice Clone
• Lip Sync
• Add Subtitles
• Supports Chinese and English

Demo

Please check res.mp4 for the demo.

Breakdown

• ASR: fast-whisper
• Text Translation:
  • facebook/m2m
  • translators[default]
• Voice Clone:
  • openvoice（For Chinese, we replaced the default TTS model of openvoice to Microsoft's TTS api）
  • TTS[default]
• Lip Sync: videotalking
• Face Restore: gfpgan
• Video Merge: pyvideotrans

Environment

1. Install ffmpeg and add it to the system environment variable, or simply put the executable file in the ffmpeg directory, and the directory structure should be as follows:

ffmpeg
   |- ffmpeg.exe
   |- ffprobe.exe
...

2. Install dependencies

git clone [email protected]:halfzm/v2vt.git

conda create -n v2vt_clone python=3.11.0
conda activate v2vt_clone

cd v2vt_clone
pip install -r requirements.txt
pip install torch torchvision torchaudio --index-url https://download.pytorch.org/whl/cu118

3. Put the openvoice pretrained models in the openvoice_checkpoints directory, and the directory structure should be as follows:[optional]

openvoice_checkpoints
   |- base_speakers
    |- ...
   |- converter
    |- ...
...

4. Put the coqui xtts pretrained modelsin the tts_models directory，and the directory structure should be as follows:

tts_models
   |- tts_models--multilingual--multi-dataset--xtts_v2
    |- config.json
    |- model.pth
    |- speakers_xtts.pth
    |- vocal.json
   |- ...
...

5. Put the video retalking checkpoints in the video-retalking/checkpointsdirectory，and the directory structure should be as follows:

video-retalking
   |- checkpoints
    |- ...
...

Usage

• Quick start

python app.py

• Input file
  The input video should not be too short, otherwise an error will occur when the voice is being cloned (preferably >=5 seconds)

• Output file
  By default, you can find the outfile in current directory, named outpu.mp4, which can also be downloaded directly from the webui.

Others

About licence and code_of_conduct, we follow the video-retalking project.
You can see the details at LICENSE and CODE_OF_CONDUCT.