ViGent/models/MuseTalk/DEPLOY.md

MuseTalk 部署指南

硬件要求

配置	最低要求	推荐配置
GPU	8GB VRAM (如 RTX 3060)	24GB VRAM (如 RTX 3090)
内存	32GB	64GB
CUDA	11.7+	12.0+

---

📦 安装步骤

1. 克隆 MuseTalk 仓库

# 进入 ViGent 项目的 models 目录
cd /home/rongye/ProgramFiles/ViGent/models

# 克隆 MuseTalk 仓库
git clone https://github.com/TMElyralab/MuseTalk.git MuseTalk_repo

# 保留我们的自定义文件 (如果有)
# cp MuseTalk/DEPLOY.md MuseTalk_repo/
# cp MuseTalk/musetalk_api.py MuseTalk_repo/

# 替换目录
rm -rf MuseTalk
mv MuseTalk_repo MuseTalk

2. 创建虚拟环境

cd /home/rongye/ProgramFiles/ViGent/models/MuseTalk
conda create -n musetalk python=3.10 -y
conda activate musetalk

3. 安装 PyTorch (稳定版 2.0.1)

⚠️ 注意：必须使用 PyTorch 2.0.1 配合 CUDA 11.8，因为这是 mmcv 预编译包支持的版本。

pip install torch==2.0.1 torchvision==0.15.2 torchaudio==2.0.2 --index-url https://download.pytorch.org/whl/cu118

4. 安装 MuseTalk 依赖 (MMLab)

严格按照以下顺序和版本安装：

pip install -r requirements.txt

# 安装 mmlab 系列
pip install --no-cache-dir -U openmim
mim install mmengine
mim install "mmcv==2.0.1"
mim install "mmdet==3.1.0"
pip install chumpy --no-build-isolation
pip install "mmpose==1.1.0" --no-deps

5. 下载模型权重 ⬇️

注意：模型目录结构极其重要，必须严格按照以下步骤操作。

cd /home/rongye/ProgramFiles/ViGent/models/MuseTalk/models

# 1. 准备目录
mkdir -p musetalk musetalkV15 dwpose syncnet face-parse-bisent

# 2. 从 HuggingFace 下载基础模型
pip install huggingface_hub

# MuseTalk v1
huggingface-cli download TMElyralab/MuseTalk --local-dir ./musetalk_tmp --include "*.pth" "*.json" "*.bin"
mv musetalk_tmp/* musetalk/ && rm -rf musetalk_tmp

# MuseTalk v1.5
huggingface-cli download TMElyralab/MuseTalk --local-dir ./musetalkV15_tmp --include "unet.pth"
mv musetalkV15_tmp/* musetalkV15/ && rm -rf musetalkV15_tmp

# SD-VAE
huggingface-cli download stabilityai/sd-vae-ft-mse --local-dir ./sd-vae-ft-mse

# Whisper
huggingface-cli download openai/whisper-tiny --local-dir ./whisper

# DWPose
huggingface-cli download yzd-v/DWPose dw-ll_ucoco_384.pth --local-dir ./dwpose

# SyncNet
huggingface-cli download ByteDance/LatentSync latentsync_syncnet.pt --local-dir ./syncnet

# 3. 下载 Face Parse 模型 (需从 Google Drive 或 PyTorch 官网)
cd face-parse-bisent
wget https://download.pytorch.org/models/resnet18-5c106cde.pth -O resnet18-5c106cde.pth
# 79999_iter.pth主要从Google Drive下载，或手动上传
# pip install gdown && gdown 154JgKpzCPW82qINcVieuPH3fZ2e0P812 -O 79999_iter.pth
cd ..

# 4. === 关键修复步骤 ===
# 创建必要的符号链接以匹配代码路径
ln -s sd-vae-ft-mse sd-vae
cd musetalk && ln -s musetalk.json config.json && cd ..

---

📂 最终目录结构验证

确保 models/MuseTalk/models 目录如下所示：

models/
├── musetalk/
│   ├── config.json -> musetalk.json  # ⚠️ 必须有此软链
│   ├── musetalk.json
│   └── pytorch_model.bin
├── musetalkV15/
│   ├── musetalk.json
│   └── unet.pth
├── sd-vae -> sd-vae-ft-mse           # ⚠️ 必须有此软链
├── sd-vae-ft-mse/
│   └── diffusion_pytorch_model.bin
├── whisper/
│   └── pytorch_model.bin ...
├── dwpose/
│   └── dw-ll_ucoco_384.pth
├── syncnet/
│   └── latentsync_syncnet.pt
└── face-parse-bisent/
    ├── 79999_iter.pth
    └── resnet18-5c106cde.pth

---

🔧 验证安装

cd /home/rongye/ProgramFiles/ViGent/models/MuseTalk
conda activate musetalk

# 运行测试脚本 (需先准备测试素材)
# 确保 inference_config.yaml 格式正确
CUDA_VISIBLE_DEVICES=1 python -m scripts.inference \
    --version v15 \
    --inference_config configs/inference/test.yaml \
    --result_dir ./results \
    --use_float16