Init: 初版代码
This commit is contained in:
Kevin Wong
186
models/MuseTalk/DEPLOY.md
Normal file
186
models/MuseTalk/DEPLOY.md
Normal file
@@ -0,0 +1,186 @@
|
||||
# MuseTalk 部署指南
|
||||
|
||||
## 硬件要求
|
||||
|
||||
| 配置 | 最低要求 | 推荐配置 |
|
||||
|------|----------|----------|
|
||||
| GPU | 8GB VRAM (如 RTX 3060) | 24GB VRAM (如 RTX 3090) |
|
||||
| 内存 | 32GB | 64GB |
|
||||
| CUDA | 11.7+ | 12.0+ |
|
||||
|
||||
---
|
||||
|
||||
## 📦 安装步骤
|
||||
|
||||
### 1. 克隆 MuseTalk 仓库
|
||||
|
||||
```bash
|
||||
# 进入 ViGent 项目的 models 目录
|
||||
cd /home/rongye/ProgramFiles/ViGent/models
|
||||
|
||||
# 克隆 MuseTalk 仓库
|
||||
git clone https://github.com/TMElyralab/MuseTalk.git MuseTalk_repo
|
||||
|
||||
# 保留我们的自定义文件
|
||||
cp MuseTalk/DEPLOY.md MuseTalk_repo/
|
||||
cp MuseTalk/musetalk_api.py MuseTalk_repo/
|
||||
|
||||
# 替换目录
|
||||
rm -rf MuseTalk
|
||||
mv MuseTalk_repo MuseTalk
|
||||
```
|
||||
|
||||
### 2. 创建虚拟环境
|
||||
|
||||
```bash
|
||||
cd /home/rongye/ProgramFiles/ViGent/models/MuseTalk
|
||||
conda create -n musetalk python=3.10 -y
|
||||
conda activate musetalk
|
||||
```
|
||||
|
||||
### 3. 安装 PyTorch (CUDA 12.1)
|
||||
|
||||
```bash
|
||||
# CUDA 12.1 (适配服务器 CUDA 12.8)
|
||||
pip install torch==2.1.0 torchvision==0.16.0 torchaudio==2.1.0 --index-url https://download.pytorch.org/whl/cu121
|
||||
```
|
||||
|
||||
### 4. 安装 MuseTalk 依赖
|
||||
|
||||
```bash
|
||||
pip install -r requirements.txt
|
||||
|
||||
# 安装 mmlab 系列 (MuseTalk 必需)
|
||||
pip install --no-cache-dir -U openmim
|
||||
mim install mmengine
|
||||
mim install "mmcv>=2.0.1"
|
||||
mim install "mmdet>=3.1.0"
|
||||
mim install "mmpose>=1.1.0"
|
||||
```
|
||||
|
||||
### 5. 下载模型权重 ⬇️
|
||||
|
||||
> **权重文件较大（约 5GB），请确保网络稳定**
|
||||
|
||||
#### 方式一：从 Hugging Face 下载 (推荐)
|
||||
|
||||
```bash
|
||||
cd /home/rongye/ProgramFiles/ViGent/models/MuseTalk
|
||||
|
||||
# 安装 huggingface-cli
|
||||
pip install huggingface_hub
|
||||
|
||||
# 下载 MuseTalk 权重 (v1.5)
|
||||
huggingface-cli download TMElyralab/MuseTalk \
|
||||
--local-dir ./models/musetalk \
|
||||
--include "*.pth" "*.json"
|
||||
|
||||
# 下载 MuseTalk V15 权重
|
||||
huggingface-cli download TMElyralab/MuseTalk \
|
||||
--local-dir ./models/musetalkV15 \
|
||||
--include "unet.pth"
|
||||
|
||||
# 下载 SD-VAE 模型 (Stable Diffusion VAE)
|
||||
huggingface-cli download stabilityai/sd-vae-ft-mse \
|
||||
--local-dir ./models/sd-vae-ft-mse
|
||||
|
||||
# 下载 Whisper 模型 (音频特征提取)
|
||||
# MuseTalk 使用 whisper-tiny
|
||||
huggingface-cli download openai/whisper-tiny \
|
||||
--local-dir ./models/whisper
|
||||
```
|
||||
|
||||
#### 方式二：手动下载
|
||||
|
||||
从以下链接下载并放到对应目录：
|
||||
|
||||
| 模型 | 下载链接 | 存放路径 |
|
||||
|------|----------|----------|
|
||||
| MuseTalk | [Hugging Face](https://huggingface.co/TMElyralab/MuseTalk) | `models/MuseTalk/models/musetalk/` |
|
||||
| MuseTalk V15 | 同上 | `models/MuseTalk/models/musetalkV15/` |
|
||||
| SD-VAE | [Hugging Face](https://huggingface.co/stabilityai/sd-vae-ft-mse) | `models/MuseTalk/models/sd-vae-ft-mse/` |
|
||||
| Whisper | [Hugging Face](https://huggingface.co/openai/whisper-tiny) | `models/MuseTalk/models/whisper/` |
|
||||
| DWPose | 按官方 README | `models/MuseTalk/models/dwpose/` |
|
||||
| Face Parse | 按官方 README | `models/MuseTalk/models/face-parse-bisent/` |
|
||||
|
||||
### 6. 验证安装
|
||||
|
||||
```bash
|
||||
cd /home/rongye/ProgramFiles/ViGent/models/MuseTalk
|
||||
conda activate musetalk
|
||||
|
||||
# 测试推理 (使用 GPU1)
|
||||
CUDA_VISIBLE_DEVICES=1 python -m scripts.inference \
|
||||
--version v15 \
|
||||
--inference_config configs/inference/test.yaml \
|
||||
--result_dir ./results \
|
||||
--use_float16
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 📂 目录结构
|
||||
|
||||
安装完成后目录结构：
|
||||
|
||||
```
|
||||
models/MuseTalk/
|
||||
├── configs/
|
||||
│ └── inference/
|
||||
├── models/ # ⬅️ 权重文件目录
|
||||
│ ├── musetalk/ # MuseTalk 基础权重
|
||||
│ │ ├── config.json
|
||||
│ │ └── pytorch_model.bin
|
||||
│ ├── musetalkV15/ # V1.5 版本 UNet
|
||||
│ │ └── unet.pth
|
||||
│ ├── sd-vae-ft-mse/ # Stable Diffusion VAE
|
||||
│ │ └── diffusion_pytorch_model.bin
|
||||
│ ├── whisper/ # Whisper 模型
|
||||
│ ├── dwpose/ # 姿态检测
|
||||
│ └── face-parse-bisent/ # 人脸解析
|
||||
├── musetalk/ # MuseTalk 源码
|
||||
├── scripts/
|
||||
│ └── inference.py
|
||||
├── DEPLOY.md # 本文档
|
||||
└── musetalk_api.py # API 服务
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 🔧 ViGent 集成配置
|
||||
|
||||
### 环境变量配置
|
||||
|
||||
在 `/home/rongye/ProgramFiles/ViGent/backend/.env` 中设置：
|
||||
|
||||
```bash
|
||||
# MuseTalk 配置
|
||||
MUSETALK_LOCAL=true
|
||||
MUSETALK_GPU_ID=1
|
||||
MUSETALK_VERSION=v15
|
||||
MUSETALK_USE_FLOAT16=true
|
||||
MUSETALK_BATCH_SIZE=8
|
||||
```
|
||||
|
||||
### 启动后端服务
|
||||
|
||||
```bash
|
||||
cd /home/rongye/ProgramFiles/ViGent/backend
|
||||
source venv/bin/activate
|
||||
|
||||
# 设置 GPU 并启动
|
||||
CUDA_VISIBLE_DEVICES=1 uvicorn app.main:app --host 0.0.0.0 --port 8000
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 🚨 常见问题
|
||||
|
||||
### Q1: CUDA out of memory
|
||||
**解决**：减小 `MUSETALK_BATCH_SIZE` 或启用 `MUSETALK_USE_FLOAT16=true`
|
||||
|
||||
### Q2: mmcv 安装失败
|
||||
**解决**：确保 CUDA 版本匹配，使用 `mim install mmcv==2.0.1`
|
||||
|
||||
### Q3: Whisper 加载失败
|
||||
**解决**：检查 `models/whisper/` 目录是否包含完整模型文件
|
||||
Reference in New Issue
Block a user