🚀 Getting Started

Currently supports 3 modes:

1. preset: Uses fixed voice, can preview on Official Playground (opens in a new tab), default is anna.
2. clone(stable): Corresponds to fishtts api's custom, uses voice from uploaded audio, automatically samples first 10 seconds of video for voice, better voice consistency.
3. clone(dynamic): Corresponds to fishtts api's dynamic, uses each sentence as reference audio during TTS, may have inconsistent voice but better effect.

Voice list can be found on the official website (opens in a new tab), such as alloy, echo, nova, etc. Modify openai_tts.voice in config.yaml.

Recommended to try voices in the online demo (opens in a new tab). You can find the voice code in the code on the right, e.g. zh-CN-XiaoxiaoMultilingualNeural

Go to the official website (opens in a new tab) to listen and choose voices. Find the voice code in the URL, e.g. Dingzhen is 54a5170264694bfc8e9ad98df7bd89c3. Popular voices are already added in config.yaml. To use other voices, modify the fish_tts.character_id_dict dictionary in config.yaml.

1. Check requirements and download the package from official Yuque docs (opens in a new tab).

2. Place GPT-SoVITS-v2-xxx and VideoLingo in the same directory. Note they should be parallel folders.

3. Choose one of the following ways to configure the model:

   a. Self-trained model:

   • After training, tts_infer.yaml under GPT-SoVITS-v2-xxx\GPT_SoVITS\configs will have your model path auto-filled. Copy and rename it to your_preferred_english_character_name.yaml
   • In the same directory as the yaml file, place reference audio named your_preferred_english_character_name_reference_audio_text.wav or .mp3, e.g. Huanyuv2_Hello, this is a test audio.wav
   • In VideoLingo's sidebar, set GPT-SoVITS Character to your_preferred_english_character_name.

   b. Use pre-trained model:

   • Download my model from here (opens in a new tab), extract and overwrite to GPT-SoVITS-v2-xxx.
   • Set GPT-SoVITS Character to Huanyuv2.

   c. Use other trained models:

   • Place xxx.ckpt in GPT_weights_v2 folder and xxx.pth in SoVITS_weights_v2 folder.

   • Following method a, rename tts_infer.yaml and modify t2s_weights_path and vits_weights_path under custom to point to your models, e.g.:

     # Example config for method b:
     t2s_weights_path: GPT_weights_v2/Huanyu_v2-e10.ckpt
     version: v2
     vits_weights_path: SoVITS_weights_v2/Huanyu_v2_e10_s150.pth

   • Following method a, place reference audio in the same directory as the yaml file, named your_preferred_english_character_name_reference_audio_text.wav or .mp3, e.g. Huanyuv2_Hello, this is a test audio.wav. The program will auto-detect and use it.

   • ⚠️ Warning: Please use English for character_name to avoid errors. reference_audio_text can be in Chinese. Currently in beta, may produce errors.

   # Expected directory structure:
   .
   ├── VideoLingo
   │   └── ...
   └── GPT-SoVITS-v2-xxx
       ├── GPT_SoVITS
       │   └── configs
       │       ├── tts_infer.yaml
       │       ├── your_preferred_english_character_name.yaml
       │       └── your_preferred_english_character_name_reference_audio_text.wav
       ├── GPT_weights_v2
       │   └── [your GPT model file]
       └── SoVITS_weights_v2
           └── [your SoVITS model file]

After configuration, select Reference Audio Mode in the sidebar (see Yuque docs for details). During dubbing, VideoLingo will automatically open GPT-SoVITS inference API port in the command line, which can be closed manually after completion. Note that stability depends on the base model chosen.