Originals/ViGent
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases 2 Wiki Activity

更新MuseTalk的部署

Browse Source
This commit is contained in:
Kevin Wong
2026-01-15 15:50:23 +08:00
parent fcc1a99778
commit 30a7677837
2 changed files with 90 additions and 156 deletions
Download Patch File Download Diff File Expand all files Collapse all files

55
Docs/DEPLOY_MANUAL.md
View File

@@ -69,55 +69,20 @@ playwright install chromium
---
## 步骤 4: 安装 MMPose (唇形检测)
## 步骤 4: 部署 AI 模型 (MuseTalk)
```bash
source /home/rongye/ProgramFiles/ViGent/backend/venv/bin/activate
> ⚠️ **重要**：MuseTalk 需要独立的 Conda 环境和特定的 PyTorch 版本。请**不要**直接安装在后端环境中。
pip install -U openmim
mim install mmengine
mim install "mmcv>=2.0.1"
mim install "mmdet>=3.1.0"
mim install "mmpose>=1.1.0"
```
请参考详细的独立部署指南：
**[MuseTalk 部署指南](../models/MuseTalk/DEPLOY.md)**
---
该指南包含以下关键步骤，请务必严格按照文档操作：
1. 创建独立的 `musetalk` Conda 环境
2. 安装特定版本的 PyTorch (2.0.1) 和 MMLab 依赖
3. 按照特定结构下载模型权重
4. 验证推理脚本
## 步骤 5: 安装 MuseTalk
```bash
cd /home/rongye/ProgramFiles/ViGent/models
# 克隆仓库
git clone https://github.com/TMElyralab/MuseTalk.git
cd MuseTalk
# 激活虚拟环境
source /home/rongye/ProgramFiles/ViGent/backend/venv/bin/activate
# 安装依赖
pip install -r requirements.txt
```
---
## 步骤 6: 下载 MuseTalk 模型权重
从 HuggingFace 下载模型:
- 地址: https://huggingface.co/TMElyralab/MuseTalk
```bash
cd /home/rongye/ProgramFiles/ViGent/models/MuseTalk
# 使用 huggingface-cli 下载 (需要安装 huggingface_hub)
pip install huggingface_hub
huggingface-cli download TMElyralab/MuseTalk --local-dir ./models
```
或手动下载后放到:
```
/home/rongye/ProgramFiles/ViGent/models/MuseTalk/models/
```
确保 MuseTalk 部署成功后，再继续后续步骤。
---

191
models/MuseTalk/DEPLOY.md
View File

@@ -21,9 +21,9 @@ cd /home/rongye/ProgramFiles/ViGent/models
# 克隆 MuseTalk 仓库
git clone https://github.com/TMElyralab/MuseTalk.git MuseTalk_repo
# 保留我们的自定义文件
cp MuseTalk/DEPLOY.md MuseTalk_repo/
cp MuseTalk/musetalk_api.py MuseTalk_repo/
# 保留我们的自定义文件 (如果有)
# cp MuseTalk/DEPLOY.md MuseTalk_repo/
# cp MuseTalk/musetalk_api.py MuseTalk_repo/
# 替换目录
rm -rf MuseTalk
@@ -38,149 +38,118 @@ conda create -n musetalk python=3.10 -y
conda activate musetalk
```
### 3. 安装 PyTorch (CUDA 12.1)
### 3. 安装 PyTorch (稳定版 2.0.1)
⚠️ **注意**：必须使用 PyTorch 2.0.1 配合 CUDA 11.8，因为这是 mmcv 预编译包支持的版本。
```bash
# CUDA 12.1 (适配服务器 CUDA 12.8)
pip install torch==2.1.0 torchvision==0.16.0 torchaudio==2.1.0 --index-url https://download.pytorch.org/whl/cu121
pip install torch==2.0.1 torchvision==0.15.2 torchaudio==2.0.2 --index-url https://download.pytorch.org/whl/cu118
```
### 4. 安装 MuseTalk 依赖
### 4. 安装 MuseTalk 依赖 (MMLab)
严格按照以下顺序和版本安装：
```bash
pip install -r requirements.txt
# 安装 mmlab 系列 (MuseTalk 必需)
# 安装 mmlab 系列
pip install --no-cache-dir -U openmim
mim install mmengine
mim install "mmcv>=2.0.1"
mim install "mmdet>=3.1.0"
mim install "mmpose>=1.1.0"
mim install "mmcv==2.0.1"
mim install "mmdet==3.1.0"
pip install chumpy --no-build-isolation
pip install "mmpose==1.1.0" --no-deps
```
### 5. 下载模型权重 ⬇️
> **权重文件较大（约 5GB），请确保网络稳定**
#### 方式一：从 Hugging Face 下载 (推荐)
> **注意**：模型目录结构极其重要，必须严格按照以下步骤操作。
```bash
cd /home/rongye/ProgramFiles/ViGent/models/MuseTalk
cd /home/rongye/ProgramFiles/ViGent/models/MuseTalk/models
# 安装 huggingface-cli
# 1. 准备目录
mkdir -p musetalk musetalkV15 dwpose syncnet face-parse-bisent
# 2. 从 HuggingFace 下载基础模型
pip install huggingface_hub
# 下载 MuseTalk 权重 (v1.5)
huggingface-cli download TMElyralab/MuseTalk \
--local-dir ./models/musetalk \
--include "*.pth" "*.json"
# MuseTalk v1
huggingface-cli download TMElyralab/MuseTalk --local-dir ./musetalk_tmp --include "*.pth" "*.json" "*.bin"
mv musetalk_tmp/* musetalk/ && rm -rf musetalk_tmp
# 下载 MuseTalk V15 权重
huggingface-cli download TMElyralab/MuseTalk \
--local-dir ./models/musetalkV15 \
--include "unet.pth"
# MuseTalk v1.5
huggingface-cli download TMElyralab/MuseTalk --local-dir ./musetalkV15_tmp --include "unet.pth"
mv musetalkV15_tmp/* musetalkV15/ && rm -rf musetalkV15_tmp
# 下载 SD-VAE 模型 (Stable Diffusion VAE)
huggingface-cli download stabilityai/sd-vae-ft-mse \
--local-dir ./models/sd-vae-ft-mse
# SD-VAE
huggingface-cli download stabilityai/sd-vae-ft-mse --local-dir ./sd-vae-ft-mse
# 下载 Whisper 模型 (音频特征提取)
# MuseTalk 使用 whisper-tiny
huggingface-cli download openai/whisper-tiny \
--local-dir ./models/whisper
# Whisper
huggingface-cli download openai/whisper-tiny --local-dir ./whisper
# DWPose
huggingface-cli download yzd-v/DWPose dw-ll_ucoco_384.pth --local-dir ./dwpose
# SyncNet
huggingface-cli download ByteDance/LatentSync latentsync_syncnet.pt --local-dir ./syncnet
# 3. 下载 Face Parse 模型 (需从 Google Drive 或 PyTorch 官网)
cd face-parse-bisent
wget https://download.pytorch.org/models/resnet18-5c106cde.pth -O resnet18-5c106cde.pth
# 79999_iter.pth主要从Google Drive下载，或手动上传
# pip install gdown && gdown 154JgKpzCPW82qINcVieuPH3fZ2e0P812 -O 79999_iter.pth
cd ..
# 4. === 关键修复步骤 ===
# 创建必要的符号链接以匹配代码路径
ln -s sd-vae-ft-mse sd-vae
cd musetalk && ln -s musetalk.json config.json && cd ..
```
#### 方式二：手动下载
---
从以下链接下载并放到对应目录：
## 📂 最终目录结构验证
| 模型 | 下载链接 | 存放路径 |
|------|----------|----------|
| MuseTalk | [Hugging Face](https://huggingface.co/TMElyralab/MuseTalk) | `models/MuseTalk/models/musetalk/` |
| MuseTalk V15 | 同上 | `models/MuseTalk/models/musetalkV15/` |
| SD-VAE | [Hugging Face](https://huggingface.co/stabilityai/sd-vae-ft-mse) | `models/MuseTalk/models/sd-vae-ft-mse/` |
| Whisper | [Hugging Face](https://huggingface.co/openai/whisper-tiny) | `models/MuseTalk/models/whisper/` |
| DWPose | 按官方 README | `models/MuseTalk/models/dwpose/` |
| Face Parse | 按官方 README | `models/MuseTalk/models/face-parse-bisent/` |
确保 `models/MuseTalk/models` 目录如下所示：
### 6. 验证安装
```
models/
├── musetalk/
│ ├── config.json -> musetalk.json # ⚠️ 必须有此软链
│ ├── musetalk.json
│ └── pytorch_model.bin
├── musetalkV15/
│ ├── musetalk.json
│ └── unet.pth
├── sd-vae -> sd-vae-ft-mse # ⚠️ 必须有此软链
├── sd-vae-ft-mse/
│ └── diffusion_pytorch_model.bin
├── whisper/
│ └── pytorch_model.bin ...
├── dwpose/
│ └── dw-ll_ucoco_384.pth
├── syncnet/
│ └── latentsync_syncnet.pt
└── face-parse-bisent/
├── 79999_iter.pth
└── resnet18-5c106cde.pth
```
---
## 🔧 验证安装
```bash
cd /home/rongye/ProgramFiles/ViGent/models/MuseTalk
conda activate musetalk
# 测试推理 (使用 GPU1)
# 运行测试脚本 (需先准备测试素材)
# 确保 inference_config.yaml 格式正确
CUDA_VISIBLE_DEVICES=1 python -m scripts.inference \
--version v15 \
--inference_config configs/inference/test.yaml \
--result_dir ./results \
--use_float16
```
---
## 📂 目录结构
安装完成后目录结构：
```
models/MuseTalk/
├── configs/
│ └── inference/
├── models/ # ⬅️ 权重文件目录
│ ├── musetalk/ # MuseTalk 基础权重
│ │ ├── config.json
│ │ └── pytorch_model.bin
│ ├── musetalkV15/ # V1.5 版本 UNet
│ │ └── unet.pth
│ ├── sd-vae-ft-mse/ # Stable Diffusion VAE
│ │ └── diffusion_pytorch_model.bin
│ ├── whisper/ # Whisper 模型
│ ├── dwpose/ # 姿态检测
│ └── face-parse-bisent/ # 人脸解析
├── musetalk/ # MuseTalk 源码
├── scripts/
│ └── inference.py
├── DEPLOY.md # 本文档
└── musetalk_api.py # API 服务
```
---
## 🔧 ViGent 集成配置
### 环境变量配置
在 `/home/rongye/ProgramFiles/ViGent/backend/.env` 中设置：
```bash
# MuseTalk 配置
MUSETALK_LOCAL=true
MUSETALK_GPU_ID=1
MUSETALK_VERSION=v15
MUSETALK_USE_FLOAT16=true
MUSETALK_BATCH_SIZE=8
```
### 启动后端服务
```bash
cd /home/rongye/ProgramFiles/ViGent/backend
source venv/bin/activate
# 设置 GPU 并启动
CUDA_VISIBLE_DEVICES=1 uvicorn app.main:app --host 0.0.0.0 --port 8000
```
---
## 🚨 常见问题
### Q1: CUDA out of memory
**解决**：减小 `MUSETALK_BATCH_SIZE` 或启用 `MUSETALK_USE_FLOAT16=true`
### Q2: mmcv 安装失败
**解决**：确保 CUDA 版本匹配，使用 `mim install mmcv==2.0.1`
### Q3: Whisper 加载失败
**解决**：检查 `models/whisper/` 目录是否包含完整模型文件