ViGent/models/MuseTalk/DEPLOY.md

MuseTalk 部署指南

更新时间：2026-01-16
适用版本：MuseTalk v1.5

---

硬件要求

配置	最低要求	推荐配置
GPU	8GB VRAM (RTX 3060)	24GB VRAM (RTX 3090)
内存	32GB	64GB
CUDA	11.7+	11.8

---

📦 安装步骤

1. 创建 Conda 环境

cd /home/rongye/ProgramFiles/ViGent/models/MuseTalk
conda create -n musetalk python=3.10 -y
conda activate musetalk

2. 安装 PyTorch (固定版本 2.0.1)

⚠️ 重要：必须使用 PyTorch 2.0.1 + CUDA 11.8，这是 mmcv 预编译包支持的版本。

pip install torch==2.0.1 torchvision==0.15.2 torchaudio==2.0.2 --index-url https://download.pytorch.org/whl/cu118

3. 安装 MMLab 依赖

严格按顺序执行：

pip install -r requirements.txt

# MMLab 系列
pip install --no-cache-dir -U openmim
mim install mmengine
mim install "mmcv==2.0.1"
mim install "mmdet==3.1.0"
pip install chumpy --no-build-isolation
pip install "mmpose==1.1.0" --no-deps

---

⬇️ 模型权重下载

权重路径总览

ViGent/models/MuseTalk/models/
├── musetalk/                   ← MuseTalk v1 基础模型
│   ├── config.json -> musetalk.json  ⚠️ 软链接
│   ├── musetalk.json
│   ├── musetalkV15 -> ../musetalkV15  ⚠️ 软链接 (供 lipsync_service 检测)
│   └── pytorch_model.bin       (~3.2GB)
│
├── musetalkV15/                ← MuseTalk v1.5 UNet 模型
│   ├── musetalk.json
│   └── unet.pth                (~3.2GB)
│
├── sd-vae -> sd-vae-ft-mse     ⚠️ 软链接
├── sd-vae-ft-mse/              ← Stable Diffusion VAE
│   ├── config.json
│   └── diffusion_pytorch_model.bin
│
├── whisper/                    ← OpenAI Whisper Tiny
│   ├── config.json
│   ├── pytorch_model.bin       (~151MB)
│   └── ...
│
├── dwpose/                     ← DWPose 人体姿态检测
│   └── dw-ll_ucoco_384.pth     (~387MB)
│
├── syncnet/                    ← SyncNet 唇形同步评估
│   └── latentsync_syncnet.pt
│
└── face-parse-bisent/          ← 人脸解析模型
    ├── 79999_iter.pth          (~53MB, 需从 Google Drive 下载)
    └── resnet18-5c106cde.pth   (~45MB)

下载步骤

cd /home/rongye/ProgramFiles/ViGent/models/MuseTalk/models

# 安装下载工具
pip install huggingface_hub

# 1. MuseTalk v1 模型
huggingface-cli download TMElyralab/MuseTalk \
    --include "musetalk/musetalk.json" "musetalk/pytorch_model.bin" \
    --local-dir ./musetalk_hf
mv musetalk_hf/musetalk/* musetalk/ 2>/dev/null || true
rm -rf musetalk_hf

# 2. MuseTalk v1.5 UNet
huggingface-cli download TMElyralab/MuseTalk \
    --include "musetalkV15/unet.pth" "musetalkV15/musetalk.json" \
    --local-dir ./mt15_tmp
mv mt15_tmp/musetalkV15/* musetalkV15/ 2>/dev/null || true
rm -rf mt15_tmp

# 3. SD-VAE
huggingface-cli download stabilityai/sd-vae-ft-mse --local-dir ./sd-vae-ft-mse

# 4. Whisper Tiny
huggingface-cli download openai/whisper-tiny --local-dir ./whisper

# 5. DWPose
mkdir -p dwpose
huggingface-cli download yzd-v/DWPose dw-ll_ucoco_384.pth --local-dir ./dwpose

# 6. SyncNet
mkdir -p syncnet
huggingface-cli download ByteDance/LatentSync latentsync_syncnet.pt --local-dir ./syncnet

# 7. Face Parse BiSeNet
mkdir -p face-parse-bisent
cd face-parse-bisent
wget https://download.pytorch.org/models/resnet18-5c106cde.pth -O resnet18-5c106cde.pth
# 79999_iter.pth 需要从 Google Drive 下载
pip install gdown
gdown 154JgKpzCPW82qINcVieuPH3fZ2e0P812 -O 79999_iter.pth
cd ..

⚠️ 创建必要的软链接

cd /home/rongye/ProgramFiles/ViGent/models/MuseTalk/models

# SD-VAE 路径兼容
ln -sf sd-vae-ft-mse sd-vae

# MuseTalk 配置文件
cd musetalk
ln -sf musetalk.json config.json

# ⚠️ 关键：供 lipsync_service.py 权重检测使用
ln -sf ../musetalkV15 musetalkV15
cd ..

---

<EFBFBD> 验证安装

1. 检查 Python 环境

conda activate musetalk
python -c "import torch; print('PyTorch:', torch.__version__); print('CUDA:', torch.cuda.is_available())"
python -c "import mmcv; print('mmcv:', mmcv.__version__)"

2. 检查模型权重

cd /home/rongye/ProgramFiles/ViGent/models/MuseTalk/models

# 检查关键文件
ls -la musetalk/pytorch_model.bin
ls -la musetalkV15/unet.pth
ls -la whisper/pytorch_model.bin
ls -la dwpose/dw-ll_ucoco_384.pth
ls -la musetalk/musetalkV15  # 应显示软链接

# 检查软链接
ls -la sd-vae
ls -la musetalk/config.json
ls -la musetalk/musetalkV15

3. 运行推理测试

cd /home/rongye/ProgramFiles/ViGent/models/MuseTalk
conda activate musetalk

# 使用命令行参数直接测试
CUDA_VISIBLE_DEVICES=1 python -m scripts.inference \
    --video_path /path/to/your/video.mp4 \
    --audio_path /path/to/your/audio.mp3 \
    --output_path /tmp/test_output.mp4 \
    --version v15 \
    --gpu_id 0 \
    --batch_size 8 \
    --use_float16

---

🐛 常见问题

mmcv 导入失败

ImportError: cannot import name 'Config' from 'mmcv'

解决：重新安装 mmcv

pip uninstall mmcv mmcv-full -y
mim install "mmcv==2.0.1"

CUDA 版本不匹配

RuntimeError: CUDA error: no kernel image is available

解决：确保 PyTorch 版本与 CUDA 驱动兼容

nvidia-smi  # 查看驱动支持的 CUDA 版本
pip install torch==2.0.1 --index-url https://download.pytorch.org/whl/cu118

音视频长度不匹配导致推理失败

Error occurred: 
whisper_feature.shape: torch.Size([1, 275, 5, 384])

解决：确保使用了更新后的 musetalk/utils/audio_processor.py（包含零填充逻辑）

---

📝 与 ViGent 后端集成

MuseTalk 通过 lipsync_service.py 以 subprocess 方式被调用：

1. 后端使用 MUSETALK_GPU_ID=1 环境变量指定 GPU
2. 权重检测路径：models/musetalk/musetalkV15 (需要软链接)
3. Conda 环境路径：~/ProgramFiles/miniconda3/envs/musetalk/bin/python

配置文件：backend/.env

MUSETALK_GPU_ID=1
MUSETALK_LOCAL=true
MUSETALK_VERSION=v15