From 30a7677837951cae34610668dd5c97510d662d94 Mon Sep 17 00:00:00 2001 From: Kevin Wong Date: Thu, 15 Jan 2026 15:50:23 +0800 Subject: [PATCH] =?UTF-8?q?=E6=9B=B4=E6=96=B0MuseTalk=E7=9A=84=E9=83=A8?= =?UTF-8?q?=E7=BD=B2?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit --- Docs/DEPLOY_MANUAL.md | 55 ++--------- models/MuseTalk/DEPLOY.md | 191 ++++++++++++++++---------------------- 2 files changed, 90 insertions(+), 156 deletions(-) diff --git a/Docs/DEPLOY_MANUAL.md b/Docs/DEPLOY_MANUAL.md index 1799642..ff70f62 100644 --- a/Docs/DEPLOY_MANUAL.md +++ b/Docs/DEPLOY_MANUAL.md @@ -69,55 +69,20 @@ playwright install chromium --- -## 步骤 4: 安装 MMPose (唇形检测) +## 步骤 4: 部署 AI 模型 (MuseTalk) -```bash -source /home/rongye/ProgramFiles/ViGent/backend/venv/bin/activate +> ⚠️ **重要**：MuseTalk 需要独立的 Conda 环境和特定的 PyTorch 版本。请**不要**直接安装在后端环境中。 -pip install -U openmim -mim install mmengine -mim install "mmcv>=2.0.1" -mim install "mmdet>=3.1.0" -mim install "mmpose>=1.1.0" -``` +请参考详细的独立部署指南： +**[MuseTalk 部署指南](../models/MuseTalk/DEPLOY.md)** ---- +该指南包含以下关键步骤，请务必严格按照文档操作： +1. 创建独立的 `musetalk` Conda 环境 +2. 安装特定版本的 PyTorch (2.0.1) 和 MMLab 依赖 +3. 按照特定结构下载模型权重 +4. 验证推理脚本 -## 步骤 5: 安装 MuseTalk - -```bash -cd /home/rongye/ProgramFiles/ViGent/models - -# 克隆仓库 -git clone https://github.com/TMElyralab/MuseTalk.git -cd MuseTalk - -# 激活虚拟环境 -source /home/rongye/ProgramFiles/ViGent/backend/venv/bin/activate - -# 安装依赖 -pip install -r requirements.txt -``` - ---- - -## 步骤 6: 下载 MuseTalk 模型权重 - -从 HuggingFace 下载模型: -- 地址: https://huggingface.co/TMElyralab/MuseTalk - -```bash -cd /home/rongye/ProgramFiles/ViGent/models/MuseTalk - -# 使用 huggingface-cli 下载 (需要安装 huggingface_hub) -pip install huggingface_hub -huggingface-cli download TMElyralab/MuseTalk --local-dir ./models -``` - -或手动下载后放到: -``` -/home/rongye/ProgramFiles/ViGent/models/MuseTalk/models/ -``` +确保 MuseTalk 部署成功后，再继续后续步骤。 --- diff --git a/models/MuseTalk/DEPLOY.md b/models/MuseTalk/DEPLOY.md index 1db4459..2595299 100644 --- a/models/MuseTalk/DEPLOY.md +++ b/models/MuseTalk/DEPLOY.md @@ -21,9 +21,9 @@ cd /home/rongye/ProgramFiles/ViGent/models # 克隆 MuseTalk 仓库 git clone https://github.com/TMElyralab/MuseTalk.git MuseTalk_repo -# 保留我们的自定义文件 -cp MuseTalk/DEPLOY.md MuseTalk_repo/ -cp MuseTalk/musetalk_api.py MuseTalk_repo/ +# 保留我们的自定义文件 (如果有) +# cp MuseTalk/DEPLOY.md MuseTalk_repo/ +# cp MuseTalk/musetalk_api.py MuseTalk_repo/ # 替换目录 rm -rf MuseTalk @@ -38,149 +38,118 @@ conda create -n musetalk python=3.10 -y conda activate musetalk ``` -### 3. 安装 PyTorch (CUDA 12.1) +### 3. 安装 PyTorch (稳定版 2.0.1) + +⚠️ **注意**：必须使用 PyTorch 2.0.1 配合 CUDA 11.8，因为这是 mmcv 预编译包支持的版本。 ```bash -# CUDA 12.1 (适配服务器 CUDA 12.8) -pip install torch==2.1.0 torchvision==0.16.0 torchaudio==2.1.0 --index-url https://download.pytorch.org/whl/cu121 +pip install torch==2.0.1 torchvision==0.15.2 torchaudio==2.0.2 --index-url https://download.pytorch.org/whl/cu118 ``` -### 4. 安装 MuseTalk 依赖 +### 4. 安装 MuseTalk 依赖 (MMLab) + +严格按照以下顺序和版本安装： ```bash pip install -r requirements.txt -# 安装 mmlab 系列 (MuseTalk 必需) +# 安装 mmlab 系列 pip install --no-cache-dir -U openmim mim install mmengine -mim install "mmcv>=2.0.1" -mim install "mmdet>=3.1.0" -mim install "mmpose>=1.1.0" +mim install "mmcv==2.0.1" +mim install "mmdet==3.1.0" +pip install chumpy --no-build-isolation +pip install "mmpose==1.1.0" --no-deps ``` ### 5. 下载模型权重 ⬇️ -> **权重文件较大（约 5GB），请确保网络稳定** - -#### 方式一：从 Hugging Face 下载 (推荐) +> **注意**：模型目录结构极其重要，必须严格按照以下步骤操作。 ```bash -cd /home/rongye/ProgramFiles/ViGent/models/MuseTalk +cd /home/rongye/ProgramFiles/ViGent/models/MuseTalk/models -# 安装 huggingface-cli +# 1. 准备目录 +mkdir -p musetalk musetalkV15 dwpose syncnet face-parse-bisent + +# 2. 从 HuggingFace 下载基础模型 pip install huggingface_hub -# 下载 MuseTalk 权重 (v1.5) -huggingface-cli download TMElyralab/MuseTalk \ - --local-dir ./models/musetalk \ - --include "*.pth" "*.json" +# MuseTalk v1 +huggingface-cli download TMElyralab/MuseTalk --local-dir ./musetalk_tmp --include "*.pth" "*.json" "*.bin" +mv musetalk_tmp/* musetalk/ && rm -rf musetalk_tmp -# 下载 MuseTalk V15 权重 -huggingface-cli download TMElyralab/MuseTalk \ - --local-dir ./models/musetalkV15 \ - --include "unet.pth" +# MuseTalk v1.5 +huggingface-cli download TMElyralab/MuseTalk --local-dir ./musetalkV15_tmp --include "unet.pth" +mv musetalkV15_tmp/* musetalkV15/ && rm -rf musetalkV15_tmp -# 下载 SD-VAE 模型 (Stable Diffusion VAE) -huggingface-cli download stabilityai/sd-vae-ft-mse \ - --local-dir ./models/sd-vae-ft-mse +# SD-VAE +huggingface-cli download stabilityai/sd-vae-ft-mse --local-dir ./sd-vae-ft-mse -# 下载 Whisper 模型 (音频特征提取) -# MuseTalk 使用 whisper-tiny -huggingface-cli download openai/whisper-tiny \ - --local-dir ./models/whisper +# Whisper +huggingface-cli download openai/whisper-tiny --local-dir ./whisper + +# DWPose +huggingface-cli download yzd-v/DWPose dw-ll_ucoco_384.pth --local-dir ./dwpose + +# SyncNet +huggingface-cli download ByteDance/LatentSync latentsync_syncnet.pt --local-dir ./syncnet + +# 3. 下载 Face Parse 模型 (需从 Google Drive 或 PyTorch 官网) +cd face-parse-bisent +wget https://download.pytorch.org/models/resnet18-5c106cde.pth -O resnet18-5c106cde.pth +# 79999_iter.pth主要从Google Drive下载，或手动上传 +# pip install gdown && gdown 154JgKpzCPW82qINcVieuPH3fZ2e0P812 -O 79999_iter.pth +cd .. + +# 4. === 关键修复步骤 === +# 创建必要的符号链接以匹配代码路径 +ln -s sd-vae-ft-mse sd-vae +cd musetalk && ln -s musetalk.json config.json && cd .. ``` -#### 方式二：手动下载 +--- -从以下链接下载并放到对应目录： +## 📂 最终目录结构验证 -| 模型 | 下载链接 | 存放路径 | -|------|----------|----------| -| MuseTalk | [Hugging Face](https://huggingface.co/TMElyralab/MuseTalk) | `models/MuseTalk/models/musetalk/` | -| MuseTalk V15 | 同上 | `models/MuseTalk/models/musetalkV15/` | -| SD-VAE | [Hugging Face](https://huggingface.co/stabilityai/sd-vae-ft-mse) | `models/MuseTalk/models/sd-vae-ft-mse/` | -| Whisper | [Hugging Face](https://huggingface.co/openai/whisper-tiny) | `models/MuseTalk/models/whisper/` | -| DWPose | 按官方 README | `models/MuseTalk/models/dwpose/` | -| Face Parse | 按官方 README | `models/MuseTalk/models/face-parse-bisent/` | +确保 `models/MuseTalk/models` 目录如下所示： -### 6. 验证安装 +``` +models/ +├── musetalk/ +│ ├── config.json -> musetalk.json # ⚠️ 必须有此软链 +│ ├── musetalk.json +│ └── pytorch_model.bin +├── musetalkV15/ +│ ├── musetalk.json +│ └── unet.pth +├── sd-vae -> sd-vae-ft-mse # ⚠️ 必须有此软链 +├── sd-vae-ft-mse/ +│ └── diffusion_pytorch_model.bin +├── whisper/ +│ └── pytorch_model.bin ... +├── dwpose/ +│ └── dw-ll_ucoco_384.pth +├── syncnet/ +│ └── latentsync_syncnet.pt +└── face-parse-bisent/ + ├── 79999_iter.pth + └── resnet18-5c106cde.pth +``` + +--- + +## 🔧 验证安装 ```bash cd /home/rongye/ProgramFiles/ViGent/models/MuseTalk conda activate musetalk -# 测试推理 (使用 GPU1) +# 运行测试脚本 (需先准备测试素材) +# 确保 inference_config.yaml 格式正确 CUDA_VISIBLE_DEVICES=1 python -m scripts.inference \ --version v15 \ --inference_config configs/inference/test.yaml \ --result_dir ./results \ --use_float16 ``` - ---- - -## 📂 目录结构 - -安装完成后目录结构： - -``` -models/MuseTalk/ -├── configs/ -│ └── inference/ -├── models/ # ⬅️ 权重文件目录 -│ ├── musetalk/ # MuseTalk 基础权重 -│ │ ├── config.json -│ │ └── pytorch_model.bin -│ ├── musetalkV15/ # V1.5 版本 UNet -│ │ └── unet.pth -│ ├── sd-vae-ft-mse/ # Stable Diffusion VAE -│ │ └── diffusion_pytorch_model.bin -│ ├── whisper/ # Whisper 模型 -│ ├── dwpose/ # 姿态检测 -│ └── face-parse-bisent/ # 人脸解析 -├── musetalk/ # MuseTalk 源码 -├── scripts/ -│ └── inference.py -├── DEPLOY.md # 本文档 -└── musetalk_api.py # API 服务 -``` - ---- - -## 🔧 ViGent 集成配置 - -### 环境变量配置 - -在 `/home/rongye/ProgramFiles/ViGent/backend/.env` 中设置： - -```bash -# MuseTalk 配置 -MUSETALK_LOCAL=true -MUSETALK_GPU_ID=1 -MUSETALK_VERSION=v15 -MUSETALK_USE_FLOAT16=true -MUSETALK_BATCH_SIZE=8 -``` - -### 启动后端服务 - -```bash -cd /home/rongye/ProgramFiles/ViGent/backend -source venv/bin/activate - -# 设置 GPU 并启动 -CUDA_VISIBLE_DEVICES=1 uvicorn app.main:app --host 0.0.0.0 --port 8000 -``` - ---- - -## 🚨 常见问题 - -### Q1: CUDA out of memory -**解决**：减小 `MUSETALK_BATCH_SIZE` 或启用 `MUSETALK_USE_FLOAT16=true` - -### Q2: mmcv 安装失败 -**解决**：确保 CUDA 版本匹配，使用 `mim install mmcv==2.0.1` - -### Q3: Whisper 加载失败 -**解决**：检查 `models/whisper/` 目录是否包含完整模型文件