文档重构
This commit is contained in:
35
DOC_RULES.md
35
DOC_RULES.md
@@ -68,23 +68,28 @@ NaviGlass/Docs/
|
||||
|
||||
> ⚠️ **仅当用户明确要求更新 `task_complete.md` 时才更新**
|
||||
|
||||
### 更新内容
|
||||
```markdown
|
||||
## ✅ 已完成任务
|
||||
### 更新原则
|
||||
- **格式一致性**:直接参考 `task_complete.md` 现有格式(列表/表格)追加内容。
|
||||
- **进度更新**:仅在阶段性里程碑时更新进度百分比。
|
||||
|
||||
### [模块名称]
|
||||
- [x] 功能描述 (Day N)
|
||||
- [x] 子功能 1
|
||||
```
|
||||
### 🔍 完整性检查清单 (必做)
|
||||
每次更新 `task_complete.md` 时,必须**逐一检查**以下所有板块:
|
||||
|
||||
### 进度表更新
|
||||
```markdown
|
||||
| 模块 | 完成度 | 状态 |
|
||||
|------|--------|------|
|
||||
| 网络通信 | 100% | ✅ 完成 |
|
||||
```
|
||||
1. **文件头部 & 导航**
|
||||
- [ ] `更新时间`:必须是当天日期
|
||||
- [ ] `整体进度`:简述当前状态 (e.g. "优化交付")
|
||||
- [ ] `快速导航`:Day 范围与文档一致
|
||||
2. **核心任务区**
|
||||
- [ ] `已完成任务`:添加新的 [x] 项目
|
||||
- [ ] `后续规划与遗留问题`:管理三色板块 (优先/债务/未来),移除已完成项
|
||||
3. **统计与回顾**
|
||||
- [ ] `进度统计`:更新对应模块的状态描述和百分比
|
||||
- [ ] `里程碑`:若有重大进展,追加 `## Milestone N` (注意用 **完成时间**)
|
||||
4. **底部链接**
|
||||
- [ ] `时间线`:追加今日概括,确保与 DayLog 一致
|
||||
- [ ] `相关文档`:更新 DayLog 链接范围 (e.g. `Day1-Day28`)
|
||||
|
||||
- 仅在阶段性里程碑时更新进度百分比
|
||||
> **口诀**:头尾时间要对齐,任务规划两手抓,里程碑上别落下。
|
||||
|
||||
---
|
||||
|
||||
@@ -106,4 +111,4 @@ NaviGlass/Docs/
|
||||
|
||||
---
|
||||
|
||||
**最后更新**:2025-12-25
|
||||
**最后更新**:2026-01-06
|
||||
|
||||
60
DevLogs/Day28.md
Normal file
60
DevLogs/Day28.md
Normal file
@@ -0,0 +1,60 @@
|
||||
# Day 28: 室内导航优化与 Bug 修复 (2026-01-06)
|
||||
|
||||
## 🎯 今日目标
|
||||
1. **修复室内导航无语音播报问题**。
|
||||
2. **优化“未检测到可行走区域”的播报逻辑,消除闪烁**。
|
||||
3. **切换到更精准的室内分割模型 (14类)**。
|
||||
4. **解决 TTS 音频末尾的“啪”声**。
|
||||
5. **排查终端 WiFi 断连后无法重连的问题**。
|
||||
|
||||
## 🔧 实施细节
|
||||
|
||||
### 1. 模型切换与优化
|
||||
- **切换模型**: 从原 `yolo11l-seg-indoor.engine` (22类) 切换为 `yolo11l-seg-indoor14.engine` (14类,专为室内地板优化)。
|
||||
- **类别映射更新**: 修改 `workflow_indoor.py` 以适配 14 类数据集 (MIT Indoor Subset):
|
||||
- 可行走区域:`floor(0)`, `corridor(1)`, `sidewalk(2)`
|
||||
- 静态障碍物:`chair(3)`, `table(4)`, `sofa_bed(5)`, `cabinet(11)`, `trash_can(12)`
|
||||
- 兴趣点:`door(6)`, `elevator(7)`, `stairs(8)`
|
||||
- 动态障碍:`person(10)`
|
||||
- **效果**: 地板 (`floor`) 检测置信度显著提升(0.85+),彻底解决了“无法识别地面”的问题。
|
||||
- **识别逻辑优化**:
|
||||
- **视觉持久化**: 引入可行走区域 Mask 缓冲机制,防止因单帧检测失败导致的“绿道闪烁”现象。
|
||||
- **阈值微调**: 将 `WALKABLE_MIN_AREA` 降至 50,提高对狭窄路段或远距离地面的检出灵敏度。
|
||||
- **可视化增强**: 完善 `_draw_indoor_visualizations`,确保实时渲染分割掩膜与避障框。
|
||||
|
||||
### 2. 语音播报修复
|
||||
- **强制首播机制**: 修复了 `IndoorNavigator` 中“未检测到可行走区域”的节流逻辑。
|
||||
- **问题**: 初始化时间 `last_no_walkable_time` 为 0,导致首次检测失败时由于时间差过大或逻辑判断失误被跳过。
|
||||
- **修复**: 增加显式判断 `if self.last_no_walkable_time == 0`,确保首次检测到无路时**立即播报**,后续保持 8 秒间隔。
|
||||
- **启动即时反馈**: 在 `app_main.py` 中,收到“室内导航”指令后,立即播报“**室内导航已启动,正在寻找可行走区域**”,消除用户等待时的不确定感。
|
||||
- **崩溃修复**: 移除了新模型中不存在的 `CLASS_TOILET` 和 `CLASS_SINK` 引用,解决了后台报错导致的导航中断。
|
||||
- **语音识别优化**:
|
||||
- **别名扩展**: 针对 SenseVoice 的同音误识,增加了 `["四内导航", "思维导航", "失内导航", "室类导航"]` 等别名映射,确保“室内导航”指令的高唤醒率。
|
||||
- **短指令支持**: 允许单独说“导航”触发盲道导航,防止吞字导致命令失效。
|
||||
|
||||
### 3. 音频体验优化
|
||||
- **消除“啪”声**: 在 `app_main.py` 的 TTS 播放流末尾,将静音填充时长从 60ms 增加到 **200ms**,有效消除了音频截断带来的爆音。
|
||||
|
||||
### 4. 终端 WiFi 重连问题分析
|
||||
- **现象**: 终端离开 WiFi 范围后断网,返回后无法自动重连。
|
||||
- **分析**: 客户端程序 (`NaviGlassClient`) 的 WebSocket 连接逻辑已有无限重试和指数退避机制。无法重连是因为底层操作系统 (Tina Linux) 的 WiFi 接口在断开后未自动重新关联 AP。
|
||||
- **结论**: 此为系统配置问题,非 App 代码 Bug。
|
||||
- **建议**: 在终端部署后台脚本 (`wifi_check.sh`),定期检测网关连通性并在断网时重启 WiFi 接口。
|
||||
|
||||
## 📊 结果验证
|
||||
- ✅ **地面检测**: 14类模型能稳定识别大面积 `floor` (area > 60000)。
|
||||
- ✅ **语音反馈**: 启动导航和无路警告均能正常听到,不再静默。
|
||||
- ✅ **稳定性**: 修复未定义变量错误后,导航进程稳定运行。
|
||||
- ⚠️ **遗留问题**: 终端 WiFi 自动重连需在设备端通过脚本解决。
|
||||
|
||||
### 5. 文档体系重构与完善
|
||||
- **系统架构图更新**: 重写 `implementation_plan_complete.md`,新增基于 Mermaid 的端云协同架构图,更新至 Day 28 状态。
|
||||
- **任务清单优化**: 全面整理 `task_complete.md`,合并“技术债务/未来规划”板块,优化快速导航与排版。
|
||||
- **技术文档刷新**: 更新《技术栈对比》与《服务器技术栈》,纳入室内导盲 (14类) 与新 AI 管道信息。
|
||||
- **更新规则精简**: 优化 `DOC_RULES.md`,移除冗余示例,确立“轻量化检查清单”原则。
|
||||
|
||||
---
|
||||
|
||||
## 📝 下一步计划
|
||||
1. 在真实环境中实测 14 类模型的避障效果。
|
||||
2. (终端侧) 部署 WiFi 保活脚本。
|
||||
@@ -1,693 +1,105 @@
|
||||
# Avaota F1 <20><><EFBFBD>臬<EFBFBD><E887AC>唳<EFBFBD>餌<EFBFBD>
|
||||
# NaviGlass 系统架构与实现详情
|
||||
|
||||
**<EFBFBD><EFBFBD>𧋦**嚗鯝3.5
|
||||
**<EFBFBD>交<EFBFBD>**嚗?025-12-17
|
||||
**撟喳蝱**嚗鋫vaota F1 (<28>典<EFBFBD> V821 / 32-bit RISC-V)
|
||||
**憿寧𤌍餈𥕦漲**嚗?00% - [<EFBFBD>亦<EFBFBD>隞餃𦛚皜<EFBFBD><EFBFBD>](task_complete.md)
|
||||
**更新日期**: 2026-01-06 (Day 28)
|
||||
**项目状态**: ✅ 核心功能完备 / 室内导盲优化交付
|
||||
|
||||
---
|
||||
|
||||
## <EFBFBD><EFBFBD> <20><>﹝霂湔<E99C82>
|
||||
|
||||
<EFBFBD>祆<EFBFBD>獢<EFBFBD><EFBFBD>瘜其<EFBFBD>**<2A><><EFBFBD>臬<EFBFBD><E887AC>啁<EFBFBD><E59581><EFBFBD><EFBFBD><EFBFBD>𠗕<EFBFBD>寡圾<E5AFA1>喳<EFBFBD>蝏誯<E89D8F><E8AAAF>餌<EFBFBD>**<2A>?
|
||||
憒<EFBFBD><EFBFBD><EFBFBD>亦<EFBFBD>嚗?- <20><> **隞餃𦛚餈𥕦漲<F0A595A6><E6BCB2><EFBFBD>蝔讠<E89D94><E8AEA0><EFBFBD>𧒄<EFBFBD>渡瑪** <20>?`task_complete.md`
|
||||
- <20><> **撘<><E69298>烐𠯫敹𡑒祕<F0A19192>?* <20>?`Day1-8.md`
|
||||
|
||||
---
|
||||
|
||||
## <20>圲 <20><><EFBFBD>舀<EFBFBD><E88880>餌<EFBFBD>
|
||||
|
||||
### 蝖砌辣憭𤥁挽
|
||||
|
||||
| 璅∪<E79285> | <20>见噡/<2F>亙藁 | 摰䂿緵<E482BF>孵<EFBFBD> | <20>喲睸<E596B2><E79DB8>㺭 |
|
||||
|------|-----------|---------|----------|
|
||||
| **<EFBFBD>喲<EFBFBD>颲枏枂** | MAX98357A (I2S) | Device Tree + ALSA | PD12/PD13/PD15, 16kHz |
|
||||
| **<EFBFBD>喲<EFBFBD>颲枏<EFBFBD>** | <20>輯蝸璅⊥<E79285>暻血<E69ABB>憌?| Audio Codec ADC | MIC Gain=25, 16kHz Mono |
|
||||
| **IMU** | ICM-42688-P | GPIO 璅⊥<E79285> SPI | 簣16g, 簣2000簞/s, ~500kHz |
|
||||
| **<EFBFBD><EFBFBD><EFBFBD>憭?* | GC2083 (MIPI) | MPP 獢<>沲 | 1280x720@20fps, JPEG Q80 |
|
||||
| **蝵𤑳<E89DB5>** | 隞亙云蝵?WiFi | <20>輯蝸 | - |
|
||||
|
||||
### 頧臭辣<E887AD>嗆<EFBFBD>
|
||||
|
||||
| 蝏<>辣 | <20><><EFBFBD>舀<EFBFBD> | 霂湔<E99C82> |
|
||||
|------|--------|------|
|
||||
| **AudioCapture** | ALSA API | 暻血<E69ABB>憌𤾸<E6868C><F0A4BEB8>喉<EFBFBD>S16_LE<4C>澆<EFBFBD> |
|
||||
| **AudioPlayer** | ALSA API | <20>砍ㄟ<E7A08D>冽偘<E586BD>?|
|
||||
| **ICM42688** | GPIO + SPI<50>讛悅 | <20>芸<EFBFBD><E88AB8>訕PI撽勗𢆡 |
|
||||
| **Camera** | MPP (VI/ISP/VENC) | JPEG蝖砌辣蝻𣇉<E89DBB> |
|
||||
| **UDPSender** | BSD Socket | <20>蹱<EFBFBD><E8B9B1>曎<EFBFBD>?|
|
||||
| **WSClient** | <20>芸<EFBFBD><E88AB8>?+ BSD Socket | WebSocket摰X<E691B0>蝡荔<E89DA1><E88D94>舀<EFBFBD>鈭諹<E988AD><E8ABB9>嗆㺭<E59786>殷<EFBFBD> |
|
||||
| **HTTPClient** | libcurl | HTTP摰X<E691B0>蝡?|
|
||||
|
||||
### 蝻𤥁<E89DBB>撌亙<E6928C><E4BA99>橘<EFBFBD><E6A998>湔鰵嚗帋蝙<E5B88B>?musl嚗?
|
||||
```makefile
|
||||
SDK_ROOT := ~/ProgramFiles/AvaotaF1/avaota_sdk/tina-v821-release
|
||||
|
||||
# <20>𩤃<EFBFBD> <20>喲睸靽格迤嚗帋蝙<E5B88B>?musl 撌亙<E6928C><E4BA99>橘<EFBFBD>Day 9嚗?TOOLCHAIN_DIR := $(SDK_ROOT)/prebuilt/rootfsbuilt/riscv/nds32le-linux-musl-v5d/bin
|
||||
CROSS_COMPILE := riscv32-linux-musl-
|
||||
|
||||
# <20>折<EFBFBD>蝵殷<E89DB5>銝滚<E98A9D>摰孵<E691B0><E5ADB5>烐踎嚗?# TOOLCHAIN_DIR := $(SDK_ROOT)/out/toolchain/nds32le-linux-glibc-v5d/bin
|
||||
# CROSS_COMPILE := riscv32-unknown-linux-
|
||||
|
||||
CC := $(TOOLCHAIN_DIR)/$(CROSS_COMPILE)gcc
|
||||
CXX := $(TOOLCHAIN_DIR)/$(CROSS_COMPILE)g++
|
||||
```
|
||||
|
||||
**<EFBFBD>喲睸<EFBFBD>暹𦻖摨?*嚗?0+ <20>蹱<EFBFBD><E8B9B1><EFBFBD>嚗㚁<E59A97>
|
||||
- MPP獢<50>沲嚗惨w_mpp, media_utils, awion
|
||||
- ISP憭<50><E686AD>嚗?2銝杷SP摨?- 閫<><E996AB>蝻𤥁圾<F0A4A581><E59CBE><EFBFBD>vencoder, vdecoder, Cedar摨?- <20>喲<EFBFBD>憭<EFBFBD><E686AD>嚗惨decoder, aencoder, AGC, AEC
|
||||
- <20><>辣<EFBFBD>澆<EFBFBD>嚗饢uxers, demuxer, parser
|
||||
|
||||
---
|
||||
|
||||
## <20>働 <20><><EFBFBD>舫𠗕<E888AB>嫣<EFBFBD>閫<EFBFBD><E996AB><EFBFBD>寞<EFBFBD>
|
||||
|
||||
### 1. I2S <20>喲<EFBFBD>颲枏枂<E69E8F>滨蔭
|
||||
|
||||
#### <20>桅<EFBFBD>
|
||||
- Device Tree <20>嘥<EFBFBD><E598A5>滨蔭<E6BBA8>躰秤
|
||||
- 撘閗<E69298><E99697>蠘<EFBFBD><E8A098>脩<EFBFBD>
|
||||
- `simple-audio-card` <20>䭾<EFBFBD>甇<EFBFBD>虜撌乩<E6928C>
|
||||
|
||||
#### 閫<><E996AB><EFBFBD>寞<EFBFBD>
|
||||
```dts
|
||||
// <20><><EFBFBD>?LVDS <20>𣂼<EFBFBD><F0A382BC>滨蔭嚗䔶蛹瘥譍葵撘閗<E69298><E99697>𥕦遣<F0A595A6>祉<EFBFBD><E7A589><EFBFBD><EFBFBD>
|
||||
&pio {
|
||||
sndcodec_pins_a: sndcodec@0 {
|
||||
pins = "PD12";
|
||||
function = "i2s0";
|
||||
};
|
||||
sndcodec_pins_b: sndcodec@1 {
|
||||
pins = "PD13";
|
||||
function = "i2s0";
|
||||
};
|
||||
sndcodec_pins_c: sndcodec@2 {
|
||||
pins = "PD15";
|
||||
function = "i2s0";
|
||||
};
|
||||
};
|
||||
```
|
||||
|
||||
**<EFBFBD>喲睸蝏誯<EFBFBD>**嚗?- 雿輻鍂 `pins` + `function` <20><><EFBFBD>撅墧<E69285>?- <20>踹<EFBFBD> `allwinner,pins` 蝑㕑䌊摰帋<E691B0>撅墧<E69285>?- 瘥譍葵撘閗<E69298><E99697>祉<EFBFBD><E7A589><EFBFBD><EFBFBD><EFBFBD>湔<EFBFBD><E6B994>?
|
||||
---
|
||||
|
||||
### 2. ICM-42688 <20>帋縑<E5B88B>寞<EFBFBD><E5AF9E>匧<EFBFBD>
|
||||
|
||||
#### I2C <20>寞<EFBFBD>撠肽<E692A0>嚗<EFBFBD>仃韐伐<E99F90><E4BC90>?
|
||||
**<2A>桅<EFBFBD>**嚗?- PD1 鋡怠<E98BA1>隞硋<E99A9E><E7A18B>賢<EFBFBD><E8B3A2>?- PL2/PL3 銝齿𣈲<E9BDBF>?TWI0
|
||||
- 霈曉<E99C88><E69B89>滚<EFBFBD><E6BB9A>啣<EFBFBD> (0x68) 雿<><E99BBF>蝏嘥<E89D8F>摮睃膥霂餃<E99C82>
|
||||
|
||||
**撠肽<E692A0>餈<EFBFBD><E9A488><EFBFBD>寞<EFBFBD>**嚗?1. GPIO 璅⊥<E79285> I2C
|
||||
2. <20>湔揢<E6B994>啣<EFBFBD>撘閗<E69298> (0x69)
|
||||
3. 靚<>㟲<EFBFBD>嗅<EFBFBD>撱嗉<E692B1>
|
||||
4. Bank 撖<><E69296><EFBFBD>典<EFBFBD><E585B8>?
|
||||
**蝏栞捏**嚗𡁜虾<F0A1819C>賡<EFBFBD>閬<EFBFBD>鸌畾𠰴<E795BE>憪见<E686AA>摨誩<E691A8>嚗峕𦆮撘<F0A686AE>2C<32>寞<EFBFBD>
|
||||
|
||||
#### SPI <20>寞<EFBFBD>摰䂿緵嚗<E7B7B5><E59A97><EFBFBD><EFBFBD><EFBFBD><EFBFBD>?
|
||||
**摰䂿緵蝏<E7B7B5><E89D8F>**嚗?```cpp
|
||||
// GPIO 璅⊥<E79285> SPI Mode 0 (CPOL=0, CPHA=0)
|
||||
void spiTransfer(uint8_t data) {
|
||||
for (int i = 7; i >= 0; i--) {
|
||||
gpio_set_value(MOSI, (data >> i) & 0x01);
|
||||
usleep(1); // ~500kHz
|
||||
gpio_set_value(SCLK, 1); // 銝𠰴<E98A9D>瘝? usleep(1);
|
||||
gpio_set_value(SCLK, 0);
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
**撘閗<E69298><E99697>滨蔭**嚗?- SCLK: PD3 (GPIO 99)
|
||||
- MOSI: PD2 (GPIO 98)
|
||||
- MISO: PD4 (GPIO 100)
|
||||
- CS: PD5 (GPIO 101)
|
||||
|
||||
**<2A>扯<EFBFBD>**嚗?- <20>笔漲嚗鰺500kHz
|
||||
- WHO_AM_I 霂<><E99C82>嚗?x47 <20>?- <20>唳旿蝔喳<E89D94><E596B3>改<EFBFBD>隡条<E99AA1>
|
||||
|
||||
---
|
||||
|
||||
### 3. <20>蹱<EFBFBD><E8B9B1><EFBFBD>霂煾曎<E785BE>仿◇摨?
|
||||
#### <20>桅<EFBFBD>
|
||||
```
|
||||
undefined reference to `pthread_create`
|
||||
```
|
||||
|
||||
#### 閫<><E996AB><EFBFBD>寞<EFBFBD>
|
||||
```makefile
|
||||
# <20>躰秤憿箏<E686BF>
|
||||
LDFLAGS += -lpthread -lssl -lcrypto
|
||||
|
||||
# 甇<>&憿箏<E686BF>
|
||||
LDFLAGS += -lssl -lcrypto -lpthread -lm -lstdc++
|
||||
```
|
||||
|
||||
**閫<><E996AB>**嚗?1. 銝𡁜𦛚摨枏銁<E69E8F>?2. 蝟餌<E89D9F>摨枏銁<E69E8F>?3. `-lpthread` `-lm` `-lstdc++` <20>暹<EFBFBD><E69AB9>?
|
||||
---
|
||||
|
||||
### 4. MPP 獢<>沲<EFBFBD><E6B2B2><EFBFBD>
|
||||
|
||||
#### 蝻枏<E89DBB><E69E8F>粹<EFBFBD>蝵桐<E89DB5><E6A190>?
|
||||
**<2A>桅<EFBFBD>**嚗间BV 蝻枏<E89DBB><E69E8F>箸滯<E7AEB8>箏紡<E7AE8F>游葷銝W仃
|
||||
|
||||
**閫<><E996AB><EFBFBD>寞<EFBFBD>**嚗?```cpp
|
||||
// VI 蝻枏<E89DBB><E69E8F>粹<EFBFBD>蝵?vipp_attr.nbufs = 5; // 5銝杼I蝻枏<E89DBB><E69E8F>?vipp_attr.nplanes = 1;
|
||||
|
||||
// VBV <20>滨蔭
|
||||
aw_enc_attr.VeAttr.mMaxKeyInterval = 30;
|
||||
aw_enc_attr.VeAttr.mVbvBufferSize = 4 * 1024 * 1024; // 4MB
|
||||
```
|
||||
|
||||
**<2A>喲睸蝏誯<E89D8F>**嚗?- VI蝻枏<E89DBB><E69E8F>箔<EFBFBD>摰𡏭<E691B0>憭改<E686AD><E694B9><EFBFBD><EFBFBD><EFBFBD>厰<EFBFBD>嚗?- VBV<42><56><EFBFBD>寞旿<E5AF9E><E697BF><EFBFBD>靚<EFBFBD>㟲
|
||||
- JPEG蝻𣇉<E89DBB>韐券<E99F90>80銝箸<E98A9D>雿喳像銵∠<E98AB5>
|
||||
|
||||
---
|
||||
|
||||
### 5. 撌亙<E6928C><E4BA99>暸<EFBFBD>蝵格𤣰<E6A0BC>?(Day 7-8)
|
||||
|
||||
#### <20>桅<EFBFBD>
|
||||
- README.md <20>𣂼<EFBFBD> `riscv32-linux-musl` 撌亙<E6928C><E4BA99>曆<EFBFBD>摮睃銁
|
||||
- 憭帋葵Makefile<6C>滨蔭銝滢<E98A9D><E6BBA2>?- <20><>遣<EFBFBD>𡁏𧋦頝臬<E9A09D>瘛瑚僚
|
||||
|
||||
#### 閫<><E996AB>餈<EFBFBD><E9A488>
|
||||
1. <20>亥砭 `tina_files_clean.csv` <20><>辣蝝W<E89D9D>
|
||||
2. <20>冽<EFBFBD><E586BD>∪膥銝𢠃<E98A9D>霂<EFBFBD><E99C82><EFBFBD><EFBFBD>楝敺?3. <20>𤑳緵 musl 撌亙<E6928C><E4BA99>曆蛹<E69B86>讠憬<E8AEA0><E686AC><EFBFBD><EFBFBD>芾圾<E88ABE>页<EFBFBD>
|
||||
4. 蝖株恕雿輻鍂 glibc 撌亙<E6928C><E4BA99>橘<EFBFBD>撌脰圾<E884B0>见虾<E8A781>剁<EFBFBD>
|
||||
|
||||
#### <20><>蝏<EFBFBD><E89D8F>蝵?```bash
|
||||
# 摰鮋<E691B0>撌亙<E6928C><E4BA99>曇楝敺?out/toolchain/nds32le-linux-glibc-v5d/bin/
|
||||
|
||||
# 蝻𤥁<E89DBB><F0A4A581>剁<EFBFBD>蝚血噡<E8A180>暹𦻖嚗?riscv32-unknown-linux-g++ -> riscv32-linux-g++
|
||||
|
||||
# C摨?glibc (<28>?musl)
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### 6. Cedar 摨㯄曎<E3AF84>仿䔮憸?(Day 8)
|
||||
|
||||
#### <20>桅<EFBFBD>
|
||||
```
|
||||
undefined reference to `CDC_LOG_LEVEL_NAME'
|
||||
undefined reference to `CDC_GLOBAL_LOG_LEVEL'
|
||||
```
|
||||
|
||||
#### <20>笔<EFBFBD>
|
||||
銝餌<EFBFBD>摨?Makefile 蝻箏<E89DBB>摰峕㟲<E5B395>?Cedar 憭𡁜<E686AD>雿枏<E99BBF><E69E8F>暹𦻖
|
||||
|
||||
#### 閫<><E996AB><EFBFBD>寞<EFBFBD>
|
||||
瘛餃<EFBFBD> 60+ 銝芷<E98A9D><E88AB7><EFBFBD><EFBFBD>嚗?```makefile
|
||||
# Cedar <20>詨<EFBFBD>摨橒<E691A8><E6A992>喲睸嚗<E79DB8><E59A97>
|
||||
LDFLAGS += -lcdc_base -lcdx_base
|
||||
|
||||
# <20>喲<EFBFBD>憭<EFBFBD><E686AD>
|
||||
LDFLAGS += -ladecoder -lResample -lAudioVps -laac -lwav
|
||||
LDFLAGS += -lcedarx_aencoder -laacenc -lAgc -lAec -lAns
|
||||
|
||||
# Muxer/Demuxer
|
||||
LDFLAGS += -lmuxers -lcedarxdemuxer -lcdx_parser
|
||||
LDFLAGS += -lmp4_muxer -lraw_muxer -lmpeg2ts_muxer
|
||||
|
||||
# 閫<><E996AB>蝻𤥁圾<F0A4A581>?LDFLAGS += -lvencoder -lvdecoder -lvideoengine -lawmjpegplus
|
||||
|
||||
# <20>滨蔭閫<E894AD><E996AB>
|
||||
LDFLAGS += -lPluginMpp -lIniParserMpp -lsample_confparser
|
||||
|
||||
# <20>曄內摨?LDFLAGS += -lcedarxrender -lhwdisplay
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### 7. musl/glibc 撌亙<E6928C><E4BA99>曉<EFBFBD>摰寞<E691B0>?(Day 9) <20>?
|
||||
#### <20>桅<EFBFBD>
|
||||
撘<EFBFBD><EFBFBD>烐踎餈鞱<EFBFBD> `musl libc 1.2.4`嚗䔶<E59A97>蝔见<E89D94>雿輻鍂 `glibc` 蝻𤥁<E89DBB>嚗<EFBFBD>紡<EFBFBD>湛<EFBFBD>
|
||||
- <20>冽<EFBFBD><E586BD>曎<EFBFBD>亙膥銝滚龪<E6BB9A>?- 蝚血噡銝滚<E98A9D>摰?(`__register_atfork` 蝑?
|
||||
- 蝔见<E89D94><E8A781>䭾<EFBFBD>餈鞱<E9A488>
|
||||
|
||||
#### 閫<><E996AB><EFBFBD>寞<EFBFBD>
|
||||
```makefile
|
||||
# 靽格迤撌亙<E6928C><E4BA99>曇楝敺<E6A59D><E695BA><EFBFBD>喲睸<E596B2>𤑳緵嚗饢usl <20>?prebuilt <20>屸<EFBFBD> out/toolchain嚗?USE_MUSL := 1
|
||||
|
||||
ifeq ($(USE_MUSL),1)
|
||||
TOOLCHAIN_DIR := $(SDK_ROOT)/prebuilt/rootfsbuilt/riscv/nds32le-linux-musl-v5d/bin
|
||||
CROSS_COMPILE := riscv32-linux-musl-
|
||||
endif
|
||||
```
|
||||
|
||||
**撉諹<E69289><E8ABB9>寞<EFBFBD>**嚗?```bash
|
||||
# 璉<><E79289>亙𢆡<E4BA99><F0A286A1>曎<EFBFBD>亙膥
|
||||
readelf -l avaota_client | grep interpreter
|
||||
# 摨磰<E691A8><E7A3B0>綽<EFBFBD>/lib32/ld.so.1
|
||||
|
||||
# 撘<><E69298>烐踎銝𠰴<E98A9D>撱箇泵<E7AE87>琿曎<E790BF>?ln -s /lib32/ilp32d/libc.so /lib32/ld.so.1
|
||||
```
|
||||
|
||||
**蝏𤘪<E89D8F>**嚗尠<E59A97> 蝔见<E89D94><E8A781>𣂼<EFBFBD>餈鞱<E9A488>嚗峕<E59A97><E5B395>厩′隞嗆芋<E59786>埈<EFBFBD>霂閖<E99C82>朞<EFBFBD>
|
||||
|
||||
---
|
||||
|
||||
### 8. <20>喲<EFBFBD> I/O <20>躰秤憭<E7A7A4><E686AD> (Day 10) <20>?
|
||||
#### <20>桅<EFBFBD>
|
||||
蝔见<EFBFBD>餈鞱<EFBFBD>蝥?10 蝘鍦<E89D98>撏拇<E6928F>嚗?- ALSA 餈𥪜<E9A488> I/O error
|
||||
- `snd_pcm_recover()` <20>W<EFBFBD>憭梯揖
|
||||
- 蝐餃<E89D90>頧祆揢<E7A586>躰秤嚗Ǒsize_t` vs `snd_pcm_sframes_t`嚗?- 蝻箏<E89DBB><E7AE8F>躰秤璉<E7A7A4><E79289>亙紡<E4BA99>?Segmentation fault
|
||||
|
||||
#### 閫<><E996AB><EFBFBD>寞<EFBFBD>
|
||||
|
||||
**audio_capture.cpp - 霈曉<E99C88><E69B89>齿鰵<E9BDBF>嘥<EFBFBD><E598A5>?*嚗?```cpp
|
||||
int err = snd_pcm_recover(m_pcm_handle, frames_read, 0);
|
||||
if (err < 0) {
|
||||
LOG_ERROR("[AudioCapture] Cannot recover: %s, attempting to reinitialize...",
|
||||
snd_strerror(err));
|
||||
|
||||
// <20>喲𡡒霈曉<E99C88>
|
||||
snd_pcm_close(m_pcm_handle);
|
||||
m_pcm_handle = nullptr;
|
||||
|
||||
// 蝑匧<E89D91>撟園<E6929F><E59C92>啣<EFBFBD>憪见<E686AA>
|
||||
usleep(500000); // 500ms
|
||||
if (!init()) {
|
||||
LOG_ERROR("[AudioCapture] Failed to reinitialize device");
|
||||
return -1; // 敶餃<E695B6>憭梯揖
|
||||
}
|
||||
|
||||
LOG_INFO("[AudioCapture] Device reinitialized successfully");
|
||||
return 0; // <20>祆活霂餃<E99C82>憭梯揖嚗䔶<E59A97>霈曉<E99C88>撌脫<E6928C>憭?}
|
||||
```
|
||||
|
||||
**main.cpp - 蝐餃<E89D90>摰匧<E691B0><E58CA7>屸<EFBFBD>霂舀<E99C82><E88880>?*嚗?```cpp
|
||||
// 靽桀<E99DBD>蝐餃<E89D90><E9A483>躰秤
|
||||
snd_pcm_sframes_t frames_read = mic.read(buffer, 160); // 雿輻鍂<E8BCBB>厩泵<E58EA9>瑞掩<E7919E>?
|
||||
// 瘛餃<E7989B><E9A483>躰秤璉<E7A7A4><E79289>?if (frames_read < 0) {
|
||||
LOG_ERROR("[AUD-CAP] Fatal error, exiting thread");
|
||||
break; // 摰匧<E691B0><E58CA7><EFBFBD><EFBFBD>箇瑪蝔页<E89D94>銝滚蔣<E6BB9A>滚<EFBFBD>隞𡝗芋<F0A19D97>?}
|
||||
|
||||
if (frames_read > 0) {
|
||||
ws_aud.send_binary((uint8_t*)buffer, frames_read * 2);
|
||||
}
|
||||
```
|
||||
|
||||
#### 撉諹<E69289>蝏𤘪<E89D8F>
|
||||
- <20>?蝔见<E89D94>蝔喳<E89D94>餈鞱<E9A488> 36+ 蝘𡜐<E89D98>靽桀<E99DBD><E6A180>㵪<EFBFBD>10 蝘鍦援皞<E68FB4><E79A9E>
|
||||
- <20>?<3F>芸𢆡<E88AB8>W<EFBFBD><EFBCB7>箏<EFBFBD><E7AE8F><EFBFBD><EFBFBD>嚗<EFBFBD><E59A97>霂蓥葉<E893A5>𣂼<EFBFBD><F0A382BC>W<EFBFBD> 3 甈?I/O <20>躰秤嚗?- <20>?<3F>喲<EFBFBD>蝥輻<E89DA5>憭梯揖銝滚蔣<E6BB9A>?IMU <20>峕<EFBFBD><E5B395>誩仍蝥輻<E89DA5>
|
||||
|
||||
---
|
||||
|
||||
### 9. WebSocket 撏拇<E6928F>靽桀<E99DBD> (Day 11) <20>?
|
||||
#### <20>桅<EFBFBD>
|
||||
蝔见<EFBFBD><EFBFBD>?WebSocket 餈墧𦻖憭梯揖嚗?03 Forbidden嚗匧<E59A97>撏拇<E6928F>嚗?- <20>躰秤靽⊥<E99DBD>嚗䫤terminate called without an active exception`
|
||||
- <20>笔<EFBFBD> 1嚗䫤perform_handshake()` 憭梯揖<E6A2AF>?`m_recv_thread` <20>芸鍳<E88AB8>?- <20>笔<EFBFBD> 2嚗䫤disconnect()` 隞滚<E99A9E>霂?`join()` <20>芸鍳<E88AB8>函<EFBFBD>蝥輻<E89DA5>
|
||||
- <20>笔<EFBFBD> 3嚗𡁏𦻖<F0A1818F>嗥瑪蝔钅獈憛𧼮銁 `recv()` <20>?`disconnect()` <20>湔𦻖<E6B994>喲𡡒 socket
|
||||
|
||||
#### 閫<><E996AB><EFBFBD>寞<EFBFBD>
|
||||
|
||||
**ws_client.cpp - 甇<>&<EFBFBD><EFBC86>鱏撘<E9B18F>餈墧𦻖瘚<F0A6BB96><E7989A>**嚗?```cpp
|
||||
void WSClient::disconnect() {
|
||||
// 1. 霈曄蔭<E69B84><E894AD><EFBFBD>雿? m_running = false;
|
||||
m_connected = false;
|
||||
|
||||
// 2. <20>煾<EFBFBD><E785BE><EFBFBD><EFBFBD>剖葷
|
||||
if (m_sockfd >= 0) {
|
||||
uint8_t close_frame[] = {0x88, 0x00};
|
||||
send(m_sockfd, close_frame, sizeof(close_frame), 0);
|
||||
## 🏗️ 系统总体架构
|
||||
|
||||
NaviGlass 采用 **端云协同** 架构,将即时性要求高的采集/播放任务放在前端眼镜终端,将算力密集的 AI 推理与大模型交互放在高性能 GPU 服务器端。
|
||||
|
||||
```mermaid
|
||||
graph TD
|
||||
subgraph Client [Avaota F1 终端]
|
||||
Cam[摄像头 GC2083] -->|JPEG 流| WS_Cam[WebSocket /ws/camera]
|
||||
Mic[麦克风] -->|PCM 流| WS_Audio[WebSocket /ws_audio]
|
||||
IMU[传感器 ICM-42688] -->|UDP JSON| UDP_IMU[UDP :12345]
|
||||
WS_Audio -->|TTS PCM| Spk[扬声器]
|
||||
end
|
||||
|
||||
subgraph Server [RTX 3090 服务器]
|
||||
WS_Cam --> |TurboJPEG| Visual[视觉感知中心]
|
||||
WS_Audio --> |SenseVoice| ASR[语音识别]
|
||||
UDP_IMU --> |Fusion| NavLogic[导航逻辑]
|
||||
|
||||
// 潃?<3F>喲睸靽桀<E99DBD>嚗𡁜<E59A97> shutdown 銝剜鱏<E5899C>餃<EFBFBD><E9A483>?recv()
|
||||
shutdown(m_sockfd, SHUT_RDWR);
|
||||
}
|
||||
|
||||
// 3. <20>?join 摰鮋<E691B0>餈鞱<E9A488><E99EB1><EFBFBD>瑪蝔? if (m_recv_thread.joinable()) {
|
||||
m_recv_thread.join();
|
||||
}
|
||||
|
||||
// 4. <20>喲𡡒 socket
|
||||
if (m_sockfd >= 0) {
|
||||
close(m_sockfd);
|
||||
m_sockfd = -1;
|
||||
}
|
||||
}
|
||||
```
|
||||
Visual --> |YOLOv11-seg| Lane[盲道检测]
|
||||
Visual --> |YOLOv11-seg| Indoor[室内分割]
|
||||
Visual --> |YOLOv11-det| Light[红绿灯检测]
|
||||
|
||||
ASR --> |Text| LLM[GLM-4V-Flash]
|
||||
LLM --> |Response| EdgeTTS[语音合成]
|
||||
EdgeTTS --> |PCM| WS_Audio
|
||||
|
||||
Lane --> |Guidance| NavMaster[导航总控]
|
||||
Indoor --> |Guidance| NavMaster
|
||||
NavMaster --> |Prompt| EdgeTTS
|
||||
end
|
||||
|
||||
**<2A>喲睸蝏誯<E89D8F>**嚗?- `shutdown()` <20>臭誑銝剜鱏<E5899C>餃<EFBFBD><E9A483>?`recv()` 靚<>鍂
|
||||
- 敹<>◆<EFBFBD>?`join()` <20>滩<EFBFBD><E6BBA9>?`shutdown()`
|
||||
- 雿輻鍂 `joinable()` 璉<><E79289>亦瑪蝔𧢲糓<F0A7A2B2>血虾 join
|
||||
Client <--> |FRP Tunnel| Server
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### 10. <20>喲<EFBFBD><E596B2><EFBFBD><EFBFBD><EFBFBD>帋縑摰䂿緵 (Day 11) <20>?
|
||||
#### <20><>瘙?- <20>唳<EFBFBD>蝟餌<E89D9F><E9A48C>芣𣈲<E88AA3><F0A388B2>𨺗憸煾<E686B8><E785BE><EFBFBD><EFBFBD>銝𠹺<E98A9D>嚗?- <20><>閬<EFBFBD><E996AC><EFBFBD>圈𨺗憸烐偘<E78390>橘<EFBFBD>TTS 霂剝𨺗<E5899D><F0A8BA97><EFBFBD>嚗?- <20>滚𦛚<E6BB9A>券<EFBFBD>朞<EFBFBD> WebSocket 餈𥪜<E9A488>鈭諹<E988AD><E8ABB9>?PCM <20>喲<EFBFBD><E596B2>唳旿
|
||||
## 🧠 核心模块实现详情
|
||||
|
||||
#### 閫<><E996AB><EFBFBD>寞<EFBFBD>
|
||||
### 1. 视觉感知中心 (Visual Perception)
|
||||
**文件**: `NaviGlassServer/navigation_master.py`, `workflow_*.py`
|
||||
|
||||
**甇仿炊 1: <20>拙<EFBFBD> WSClient <20>舀<EFBFBD>鈭諹<E988AD><E8ABB9>嗆㺭<E59786>桅<EFBFBD><E6A185>?*
|
||||
* **多模态导航状态机**:
|
||||
* `IDLE`: 待机/闲聊模式
|
||||
* `BLINDPATH_NAV`: 盲道导航 (Outdoor)
|
||||
* `INDOOR_NAV`: 室内导盲 (Indoor)
|
||||
* `TRAFFIC_LIGHT`: 红绿灯检测
|
||||
* `ITEM_SEARCH`: 物品寻找
|
||||
* **室内导航 (Indoor) [Day 28 New]**:
|
||||
* **模型**: `yolo11l-seg-indoor14.engine` (TensorRT FP16)
|
||||
* **策略**: 语义分割 (Seg) 优先。提取 `floor` (CLSID 0) 生成可行走区域掩膜 (Mask),计算中心点偏移量进行方向引导。
|
||||
* **避障**: 检测 `chair`, `table`, `cabinet` 等障碍物,结合深度/位置给出避让提示。
|
||||
* **盲道导航 (Outdoor)**:
|
||||
* **模型**: `yolo11l-seg-blind.engine`
|
||||
* **算法**: `FindBestPath` 算法,计算盲道分割掩膜的主方向,引导用户居中行走。
|
||||
|
||||
```cpp
|
||||
// ws_client.h - 瘛餃<E7989B>鈭諹<E988AD><E8ABB9>嗆㺭<E59786>桅<EFBFBD><E6A185>?class WSClient {
|
||||
public:
|
||||
void poll_binary_messages(std::function<void(const uint8_t*, size_t)> callback);
|
||||
|
||||
private:
|
||||
std::queue<std::vector<uint8_t>> m_binary_queue;
|
||||
std::mutex m_binary_mutex;
|
||||
};
|
||||
### 2. AI 交互管道 (Voice Interaction)
|
||||
**文件**: `NaviGlassServer/app_main.py`
|
||||
|
||||
// ws_client.cpp - recv_loop 靽嘥<E99DBD>鈭諹<E988AD><E8ABB9>嗆㺭<E59786>?if (opcode == OP_BINARY) {
|
||||
std::lock_guard<std::mutex> lock(m_binary_mutex);
|
||||
m_binary_queue.push(payload);
|
||||
LOG_DEBUG("[WS] Received binary: %zu bytes", payload.size());
|
||||
}
|
||||
```
|
||||
* **ASR (语音识别)**: FunASR/SenseVoiceSmall (本地运行,强制中文 `zh`)。
|
||||
* **LLM (大模型)**: ZhipuAI GLM-4.6v-Flash (通过 API)。支持视觉理解,用于 "这是什么"、"前方的文字" 等通用问答。
|
||||
* **TTS (语音合成)**: EdgeTTS (本地/在线)。生成 16kHz PCM 音频流,并通过 WebSocket 实时推送到客户端播放。
|
||||
|
||||
**甇仿炊 2: 銝餃儐<E9A483>舫<EFBFBD><E888AB>鞾𨺗憸烐偘<E78390>?*
|
||||
|
||||
```cpp
|
||||
// main.cpp - <20>嘥<EFBFBD><E598A5>𡝗碸憯啣膥嚗áw:1,0 - I2S <20>亙藁嚗?AudioPlayer speaker("hw:1,0", 16000, 1); // 16kHz, mono
|
||||
bool speaker_enabled = speaker.init();
|
||||
|
||||
// 銝餃儐<E9A483>臭葉<E887AD>交𤣰撟嗆偘<E59786>暸𨺗憸?if (speaker_enabled) {
|
||||
ws_aud.poll_binary_messages([&](const uint8_t* data, size_t size) {
|
||||
if (size % 2 == 0) { // S16_LE <20>唳旿
|
||||
size_t frames = size / 2;
|
||||
speaker.write((const int16_t*)data, frames);
|
||||
}
|
||||
});
|
||||
}
|
||||
```
|
||||
|
||||
**<2A>喲睸蝏誯<E89D8F>**嚗?- WebSocket <20>?`recv_loop` 撌脩<E6928C><E884A9>函𡠺蝡讠瑪蝔衤葉餈鞱<E9A488>
|
||||
- 鈭諹<E988AD><E8ABB9>嗆㺭<E59786>桅<EFBFBD>朞<EFBFBD><E69C9E>笔<EFBFBD>隡𣳇<E99AA1>鍦<EFBFBD>銝餌瑪蝔?- 雿輻鍂 `hw:1,0`嚗㇆2S嚗㕑<E59A97>屸<EFBFBD> `hw:0,0`嚗㇁udio Codec嚗?- <20>喲<EFBFBD><E596B2>澆<EFBFBD>敹<EFBFBD>◆<EFBFBD>寥<EFBFBD>嚗?6kHz, mono, S16_LE
|
||||
|
||||
#### 撉諹<E69289>蝏𤘪<E89D8F>
|
||||
- <20>?<3F>砍ㄟ<E7A08D>典<EFBFBD>憪见<E686AA><E8A781>𣂼<EFBFBD>嚗áw:1,0嚗?- <20>?<3F>喲<EFBFBD><E596B2><EFBFBD><EFBFBD>甇<EFBFBD>虜嚗<E8999C>歇<EFBFBD>煾<EFBFBD>?2000+ <20>唳旿<E594B3><E697BF><EFBFBD>
|
||||
- <20>?TTS <20>喲<EFBFBD><E596B2>剜𦆮撌脣<E6928C><E884A3>堆<EFBFBD>Day 13 靽桀<E99DBD>嚗?
|
||||
---
|
||||
|
||||
### 11. TTS 鈭衤辣敺芰㴓<E88AB0>餃<EFBFBD>靽桀<E99DBD> (Day 13) <20>?
|
||||
#### <20>桅<EFBFBD>
|
||||
TTS <20>喲<EFBFBD><E596B2>剜𦆮<E5899C>剜鱏蝏剔賒嚗峕<E59A97><E5B395>?5-15 蝘埝<E89D98><E59F9D>剜𦆮<E5899C>牐葵摮梹<E691AE>
|
||||
- 摰X<E691B0>蝡舀𠯫敹梹<E695B9>`[AudioPlayer] Underrun occurred, recovering...`
|
||||
- <20>滚𦛚<E6BB9A>冽𠯫敹梹<E695B9>TTS <20>烾𡢿<E783BE>?5-15 蝘?
|
||||
#### <20>孵<EFBFBD><E5ADB5><EFBFBD><EFBFBD>
|
||||
`omni_client.py` 銝凋蝙<E5878B>典<EFBFBD>甇亥翮隞<E7BFAE>膥憭<E886A5><E686AD> Omni API <20>滚<EFBFBD>嚗屸獈憛硺<E6869B><E7A1BA>港葵 asyncio 鈭衤辣敺芰㴓嚗?```python
|
||||
# <20>桅<EFBFBD>隞<EFBFBD><E99A9E>
|
||||
async def stream_chat(...):
|
||||
completion = client.chat.completions.create(stream=True, ...)
|
||||
for chunk in completion: # <20>?<3F>峕郊餈凋誨嚗屸獈憛硺<E6869B>隞嗅儐<E59785>荔<EFBFBD>
|
||||
yield OmniStreamPiece(...)
|
||||
```
|
||||
|
||||
#### 閫<><E996AB><EFBFBD>寞<EFBFBD>
|
||||
|
||||
**甇仿炊 1: Omni 摰X<E691B0>蝡臬<E89DA1>甇亙<E79487>**
|
||||
|
||||
雿輻鍂 `threading.Thread` + `asyncio.Queue` 閫<><E996AB>血<EFBFBD>甇?API 靚<>鍂嚗?
|
||||
```python
|
||||
async def stream_chat(...):
|
||||
queue = asyncio.Queue()
|
||||
loop = asyncio.get_running_loop()
|
||||
|
||||
def _sync_stream():
|
||||
"""<22>函𡠺蝡讠瑪蝔衤葉餈鞱<E9A488><E99EB1>峕郊 API 靚<>鍂"""
|
||||
completion = client.chat.completions.create(stream=True, ...)
|
||||
for chunk in completion:
|
||||
piece = OmniStreamPiece(...)
|
||||
loop.call_soon_threadsafe(queue.put_nowait, piece)
|
||||
loop.call_soon_threadsafe(queue.put_nowait, None) # 蝏𤘪<E89D8F><F0A498AA><EFBFBD>扇
|
||||
|
||||
thread = threading.Thread(target=_sync_stream, daemon=True)
|
||||
thread.start()
|
||||
|
||||
while True:
|
||||
item = await queue.get() # <20>鮋獈憛䂿<E6869B>敺? if item is None:
|
||||
break
|
||||
yield item
|
||||
```
|
||||
|
||||
**甇仿炊 2: 摰X<E691B0>蝡?TTS 憸<><E686B8><EFBFBD>脫㦤<E884AB>?*
|
||||
|
||||
```cpp
|
||||
// main.cpp - TTS 憸<><E686B8><EFBFBD>脣<EFBFBD><E884A3>?static std::vector<int16_t> tts_buffer;
|
||||
static const size_t PRE_BUFFER_FRAMES = 16000; // 1蝘㘾<E89D98>蝻枏<E89DBB>
|
||||
static const size_t MIN_PLAY_FRAMES = 8000; // 0.5蝘埝<EFBFBD>撠𤩺偘<EFBFBD>暸<EFBFBD><EFBFBD>?static bool is_buffering = true;
|
||||
|
||||
// <20>交𤣰<E4BAA4>嗉蕭<E59789>惩<EFBFBD>蝻枏<E89DBB><E69E8F>?ws_aud.poll_binary_messages([&](const uint8_t* data, size_t size) {
|
||||
const int16_t* samples = (const int16_t*)data;
|
||||
size_t frames = size / 2;
|
||||
tts_buffer.insert(tts_buffer.end(), samples, samples + frames);
|
||||
|
||||
// 蝘舐敞頞喳<E9A09E><E596B3>滚<EFBFBD>憪𧢲偘<F0A7A2B2>? if (is_buffering && tts_buffer.size() >= PRE_BUFFER_FRAMES) {
|
||||
is_buffering = false;
|
||||
}
|
||||
});
|
||||
|
||||
// <20>剜𦆮<E5899C>嗡蝙<E597A1>刻<EFBFBD>撠𤩺鸌甈? if (!is_buffering && tts_buffer.size() >= MIN_PLAY_FRAMES) {
|
||||
size_t play_frames = std::min(tts_buffer.size(), (size_t)1600); // 100ms
|
||||
speaker.write(tts_buffer.data(), play_frames);
|
||||
tts_buffer.erase(tts_buffer.begin(), tts_buffer.begin() + play_frames);
|
||||
}
|
||||
```
|
||||
|
||||
**甇仿炊 3: <20>滚𦛚<E6BB9A>券<EFBFBD><E588B8><EFBFBD>箔耨憭?*
|
||||
|
||||
```python
|
||||
# app_main.py - lifespan <20>閗繮 CancelledError
|
||||
@asynccontextmanager
|
||||
async def lifespan(app):
|
||||
# ... <20>臬𢆡<E887AC>餉<EFBFBD> ...
|
||||
try:
|
||||
yield
|
||||
except asyncio.CancelledError:
|
||||
pass # Ctrl+C 甇<>虜銵䔶蛹
|
||||
finally:
|
||||
print("[LIFESPAN] 摨𠉛鍂<F0A0899B>喲𡡒摰峕<E691B0>")
|
||||
# 撘箏<E69298><E7AE8F><EFBFBD><EFBFBD>箇瑪蝔? def _force_exit():
|
||||
time.sleep(0.5)
|
||||
os._exit(0)
|
||||
threading.Thread(target=_force_exit, daemon=True).start()
|
||||
```
|
||||
|
||||
#### 撉諹<E69289>蝏𤘪<E89D8F>
|
||||
| <20><><EFBFBD> | 靽桀<E99DBD><E6A180>?| 靽桀<E99DBD><E6A180>?|
|
||||
|------|--------|--------|
|
||||
| TTS <20>烾𡢿<E783BE>?| 5-15 蝘?| ~2 蝘?|
|
||||
| 擐𡝗活<F0A19D97>剜𦆮撱嗉<E692B1> | 蝡见朖 | ~1 蝘𡜐<E89D98>憸<EFBFBD><E686B8><EFBFBD>莎<EFBFBD> |
|
||||
| Underrun 憸𤑳<E686B8> | 瘥誩<E798A5>銝<EFBFBD>甈?| <20>嗅<EFBFBD> |
|
||||
| <20>滚𦛚<E6BB9A>券<EFBFBD><E588B8><EFBFBD>?| <20><>絲 | 甇<>虜餈𥪜<E9A488> |
|
||||
|
||||
#### <20>㛖<EFBFBD><E39B96>桅<EFBFBD>
|
||||
- TTS 隞齿<E99A9E>頧餃凝<E9A483>剔賒嚗㇉mni API <20>滚<EFBFBD><E6BB9A>g<EFBFBD>
|
||||
- 撱箄悅<E7AE84><E68285>揢<EFBFBD>?HTTP `/stream.wav` 璅∪<E79285>嚗<EFBFBD><E59A97><EFBFBD>?ESP32S3 摰䂿緵嚗?
|
||||
---
|
||||
|
||||
## <20><> <20>喲睸<E596B2>𣂼停
|
||||
|
||||
### <20><><EFBFBD>舐<EFBFBD><E88890>?1. <20>?隞𡡞妟摰䂿緵 GPIO 璅⊥<E79285> SPI 撽勗𢆡
|
||||
2. <20>?閫<><E996AB> I2S Device Tree <20>滨蔭<E6BBA8>暸<EFBFBD>
|
||||
3. <20>?ICM-42688 SPI 撽勗𢆡摰峕㟲摰䂿緵
|
||||
4. <20>?MPP 獢<>沲摰峕㟲<E5B395><E39FB2><EFBFBD>嚗ĀI<C480>𥕜SP<53>栐ENC嚗?5. <20>?GC2083 100% <20>閗繮<E99697>𣂼<EFBFBD><F0A382BC>?6. <20>?<3F>蹱<EFBFBD><E8B9B1>曎<EFBFBD>?60+ 摨㮖<E691A8>韏𡝗<E99F8F><F0A19D97>?7. <20>?撌亙<E6928C><E4BA99>暸<EFBFBD>蝵桃<E89DB5>銝<EFBFBD><E98A9D>嗆<EFBFBD>
|
||||
8. <20>?3.9MB 銝餌<E98A9D>摨讐<E691A8>霂烐<E99C82><E78390>?9. <20>?<3F>喲<EFBFBD><E596B2>躰秤<E8BAB0>芸𢆡<E88AB8>W<EFBFBD><EFBCB7>箏<EFBFBD>
|
||||
10. <20>?**WebSocket 撏拇<E6928F><E68B87>桅<EFBFBD>靽桀<E99DBD>**
|
||||
11. <20>?**<2A>喲<EFBFBD><E596B2><EFBFBD><EFBFBD><EFBFBD>帋縑摰䂿緵**
|
||||
12. <20>?**TTS 鈭衤辣敺芰㴓<E88AB0>餃<EFBFBD>靽桀<E99DBD>**嚗㇉mni 摰X<E691B0>蝡臬<E89DA1>甇亙<E79487>嚗?13. <20>?**摰X<E691B0>蝡?TTS 憸<><E686B8><EFBFBD>脫㦤<E884AB>?*
|
||||
14. <20>?**憿寧𤌍隞<F0A48C8D><E99A9E><EFBFBD>西澈** (蝘駁膄 19 銝芸<E98A9D>雿蹱<E99BBF>隞?
|
||||
15. <20>?**<2A>鮋獈憛噼歲撣扳㦤<E689B3>?* (撖潸⏛ FPS 0.5 -> 10.0)
|
||||
|
||||
### <20>扯<EFBFBD><E689AF><EFBFBD><EFBFBD>
|
||||
- <20>喲<EFBFBD><E596B2><EFBFBD>甅<EFBFBD><E79485><EFBFBD>16kHz <20>?- <20><><EFBFBD>憭游葷<E6B8B8><E891B7><EFBFBD>20fps <20>?- JPEG <20>讠憬韐券<E99F90>嚗?0 <20>?- SPI <20>帋縑<E5B88B>笔漲嚗鰺500kHz <20>?- IMU <20>唳旿<E594B3><E697BF><EFBFBD>1kHz <20>?- 蝻𤥁<E89DBB>鈭抒<E988AD>憭批<E686AD>嚗?.9MB <20>?- **<2A>蹂<EFBFBD>餈鞱<E9A488>撉諹<E69289>嚗𡁏<E59A97><F0A1818F>㗇芋<E39787>?00%<25>朞<EFBFBD>** <20>?- **TTS <20>烾𡢿<E783BE>䈑<EFBFBD>~2 蝘?*嚗<>耨憭滚<E686AD> 5-15 蝘𡜐<E89D98><F0A19C90>?
|
||||
---
|
||||
|
||||
## <20><> 蝏誯<E89D8F>蝘舐敞
|
||||
|
||||
### Device Tree <20>滨蔭
|
||||
- `pinctrl` <20>箏<EFBFBD>嚗𡁏<E59A97>銝芸<E98A9D><E88AB8>𡁶𡠺蝡贝<E89DA1><E8B49D>?- `function` vs `allwinner,function`嚗帋<E59A97><E5B88B><EFBFBD>蝙<EFBFBD>冽<EFBFBD><E586BD><EFBFBD><EFBFBD><EFBFBD>?- 撘閗<E69298>憭滨鍂<E6BBA8>脩<EFBFBD>嚗𡁻<E59A97>朞<EFBFBD> `/sys/kernel/debug/pinctrl` <20>埝䰻
|
||||
|
||||
### GPIO <20>批<EFBFBD>
|
||||
- `/sys/class/gpio` <20>亙藁雿輻鍂
|
||||
- 撖澆枂 <20>?霈曄蔭<E69B84>孵<EFBFBD> <20>?霂餃<E99C82><E9A483>?- GPIO 蝻硋噡霈∠<E99C88>嚗䫤(bank - 'A') * 32 + offset`
|
||||
|
||||
### SPI <20>讛悅
|
||||
- Mode 0嚗鋴POL=0, CPHA=0
|
||||
- <20>嗅<EFBFBD>嚗鐝OSI<53><49><EFBFBD> <20>?SCLK銝𠰴<E98A9D>瘝?<3F>?MISO<53><4F>甅
|
||||
- ICM-42688 霂餃<E99C82>嚗𡁻<E59A97>摮𡑒<E691AE> 0x80|撖<><E69296><EFBFBD>典𧑐<E585B8><F0A79190>
|
||||
|
||||
### ALSA <20>喲<EFBFBD>
|
||||
- 餈鞱<E9A488><E99EB1>園<EFBFBD>蝵?vs 蝻𤥁<E89DBB><F0A4A581>園<EFBFBD>蝵?- `amixer` 靚<><E99D9A><EFBFBD><EFBFBD>撌?- MIC Gain 靚<><E99D9A><EFBFBD>寞<EFBFBD>
|
||||
|
||||
### MPP 獢<>沲
|
||||
- <20>嘥<EFBFBD><E598A5>㚚◇摨𧶏<E691A8>VI <20>?ISP <20>?VENC
|
||||
- 蝻枏<E89DBB><E69E8F>箔<EFBFBD><E7AE94>𣇉<EFBFBD><F0A38789>?- JPEG 蝻𣇉<E89DBB>韐券<E99F90>銝擧<E98A9D>隞嗅之撠𤩺<E692A0>銵?
|
||||
### 鈭文<E988AD>蝻𤥁<E89DBB>
|
||||
- <20>蹱<EFBFBD><E8B9B1><EFBFBD><EFBFBD>暹𦻖憿箏<E686BF>閫<EFBFBD><E996AB>
|
||||
- Cedar 摨㮖<E691A8>韏硋<E99F8F>蝟?- 撌亙<E6928C><E4BA99>曇楝敺<E6A59D><E695BA>霂<EFBFBD>䲮瘜?
|
||||
---
|
||||
|
||||
## <20>𩤃<EFBFBD> <20>㛖<EFBFBD><E39B96>桅<EFBFBD>銝𦒘<E98A9D><F0A69298>𡝗䲮<F0A19D97>?
|
||||
### <20><><EFBFBD>舀䔝蝝X䲮<EFBCB8>?1. **蝖砌辣 SPI**
|
||||
- 敶枏<E695B6>嚗鎭PIO 璅⊥<E79285> (~500kHz)
|
||||
- <20>臭<EFBFBD><E887AD>吔<EFBFBD>雿輻鍂蝖砌辣 SPI (<28>衽Hz)
|
||||
- <20>嗥<EFBFBD>嚗𡁻<E59A97>雿?CPU <20>删鍂
|
||||
|
||||
2. **I2C <20>寞<EFBFBD>**
|
||||
- 敶枏<E695B6>嚗𡁜歇<F0A1819C>曉<EFBFBD>
|
||||
- <20>舀䔝蝝g<E89D9D><EFBD87>寞<EFBFBD><E5AF9E>嘥<EFBFBD><E598A5>硋<EFBFBD><E7A18B>? - <20>嗥<EFBFBD>嚗𡁏凒<F0A1818F><E58792><EFBFBD><EFBFBD><EFBFBD><EFBFBD><EFBFBD>?
|
||||
3. **musl 撌亙<E6928C><E4BA99>?*
|
||||
- 敶枏<E695B6>嚗帋蝙<E5B88B>?glibc
|
||||
- <20>航<EFBFBD>蝘鳴<E89D98>閫<EFBFBD><E996AB> musl 撌亙<E6928C><E4BA99>? - <20>嗥<EFBFBD>嚗𡁜<E59A97>撠誩虾<E8AAA9>扯<EFBFBD><E689AF><EFBFBD>辣雿梶妖
|
||||
|
||||
### <20>扯<EFBFBD>隡睃<E99AA1>
|
||||
- GPIO 璅⊥<E79285> SPI <20>笔漲隡睃<E99AA1>嚗<EFBFBD>𤌍<EFBFBD>?1-2MHz嚗?- <20><><EFBFBD>雿輻鍂隡睃<E99AA1>
|
||||
- 憭𡁶瑪蝔贝<E89D94>頧賢<E9A0A7>銵?
|
||||
### 隞<><E99A9E>韐券<E99F90>
|
||||
- 蝏煺<E89D8F>瘜券<E7989C>憌擧聢
|
||||
- 瘛餃<E7989B><E9A483>訫<EFBFBD>瘚贝<E7989A>
|
||||
- <20>躰秤憭<E7A7A4><E686AD>摰<EFBFBD><E691B0>
|
||||
### 3. 服务器性能优化 (Performance)
|
||||
**关键技术**:
|
||||
* **TensorRT 加速**: 所有 YOLO 模型导出为 `.engine`,启用 FP16 半精度。
|
||||
* **TurboJPEG**: 替代 OpenCV 进行 JPEG 编解码,速度提升 3 倍。
|
||||
* **零拷贝直传**: 非导航模式下,直接转发 JPEG 数据流给前端网页,不进行解码,CPU 占用降低 90%。
|
||||
* **GPU 线程池**: `ThreadPoolExecutor` 管理 GPU 推理任务,避免 GIL 锁阻塞 Websocket 事件循环。
|
||||
|
||||
---
|
||||
|
||||
---
|
||||
## 🕶️ 终端硬件与嵌入式实现
|
||||
|
||||
### 12. 憿寧𤌍皜<F0A48C8D><E79A9C>銝?TTS 瘚<><E7989A> (Day 14) <20>?
|
||||
#### 隡睃<E99AA1><E79D83><EFBFBD>捆
|
||||
1. **HTTP <20><><EFBFBD><EFBFBD>餃<EFBFBD>靽桀<E99DBD>**嚗𡁜<E59A97><F0A1819C>?HTTP 20ms pacing 敺芰㴓<E88AB0>餃<EFBFBD>鈭?WebSocket <20>煾<EFBFBD><E785BE><EFBFBD>撖潸稲 TTS <20>⊿▼<E28ABF><E296BC>耨憭齿䲮獢<E4B2AE><E78DA2>撠?HTTP 撟踵偘蝘餉秐<E9A489>𤾸蝱隞餃𦛚 `asyncio.create_task()`<60>?2. **憿寧𤌍<E5AFA7>西澈**嚗𡁶宏<F0A181B6>?HTTP TTS <20>詨<EFBFBD>隞<EFBFBD><E99A9E><EFBFBD>𦠜<EFBFBD>霂閗<E99C82><E99697>穿<EFBFBD><E7A9BF><EFBFBD>辣<EFBFBD>誩<EFBFBD> 19 銝芥<E98A9D>?3. **<EFBFBD><EFBFBD><EFBFBD>**嚗関TS <20>剜𦆮摰<F0A686AE><E691B0>瘚<EFBFBD><E7989A>嚗峕<E59A97><E5B395>剔賒<E58994>?
|
||||
### 13. 撖潸⏛璅∪<EFBFBD><EFBFBD>⊿▼靽桀<EFBFBD> (Day 15) <20>?
|
||||
#### <20>桅<EFBFBD><E6A185>啗情
|
||||
撖潸⏛璅∪<EFBFBD>撘<EFBFBD><EFBFBD>臬<EFBFBD>嚗𤅎PS 隞?10.0 <20>渲<EFBFBD><E6B8B2>?0.5-1.5嚗𣬚𤫇<EFBFBD>W㨃憿踴<EFBFBD>?
|
||||
#### <20><><EFBFBD>舀䲮獢<E4B2AE><E78DA2><EFBFBD>鮋獈憛噼歲撣扳㦤<E689B3>?**<2A>煺誨<E785BA><E8AAA8>䔮憸?*嚗䫤await loop.run_in_executor` <20>賜<EFBFBD><E8B39C>函瑪蝔𧢲<E89D94><F0A7A2B2>扯<EFBFBD>嚗䔶<E59A97>銝餃儐<E9A483>臭<EFBFBD>蝑匧<E89D91>蝏𤘪<E89D8F>餈𥪜<E9A488>嚗<EFBFBD>紡<EFBFBD>港葡銵屸獈憛𠺶<E6869B>?
|
||||
**靽桀<E99DBD><E6A180>寞<EFBFBD>**嚗?1. **Fire-and-Forget 璅∪<E79285>**嚗帋蜓敺芰㴓<E88AB0>𣂷漱隞餃𦛚<E9A483>𡒊<EFBFBD><F0A1928A>喟誧蝏哨<E89D8F>銝滨<E98A9D>敺?`await`<60>?2. **<EFBFBD><EFBFBD><EFBFBD>啣葷隡睃<EFBFBD>**嚗𡁜<E59A97><F0A1819C>𨀣<EFBFBD><F0A880A3>啣葷銝𥪯<E98A9D>銝<EFBFBD>撣扳𧊋憭<F0A78A8B><E686AD>摰䕘<E691B0><E49598>躰<EFBFBD><E8BAB0>?Pending 撣改<E692A3>蝖桐<E89D96>憭<EFBFBD><E686AD><EFBFBD><EFBFBD><EFBFBD>唳㺭<E594B3>柴<EFBFBD>?3. **蝏𤘪<E89D8F>憭滨鍂**嚗𡁜銁<F0A1819C>𤾸蝱憭<E89DB1><E686AD>摰峕<E691B0><E5B395>㵪<EFBFBD>撟踵偘銝𠹺<E98A9D>撣抒<E692A3>璉<EFBFBD>瘚讠<E7989A><E8AEA0>頣<EFBFBD>靽脲<E99DBD> UI <20>瑟鰵<E7919F><E9B0B5>蛹 10FPS<50>?
|
||||
**<EFBFBD><EFBFBD><EFBFBD>**嚗?- 摰X<E691B0>蝡舫<E89DA1><E888AB><EFBFBD><EFBFBD>10 FPS (1280x720)
|
||||
**平台**: Avaota F1 (全志 V821 / RISC-V 架构)
|
||||
**系统**: Tina Linux (OpenWrt base, musl libc)
|
||||
|
||||
### 1. 传感器驱动
|
||||
* **摄像头**:通过 MPP (Media Process Platform) 框架直接采集 ISP 处理后的图像,硬件编码为 JPEG。
|
||||
* **IMU**: 使用 GPIO 模拟 SPI 驱动 ICM-42688,实现 10Hz 数据采集。
|
||||
* **音频**: ALSA 架构。
|
||||
* 播放: I2S 接口 (hw:1,0) -> MAX98357A 功放。
|
||||
* 录音: Audio Codec (hw:0,0) -> 驻极体麦克风。
|
||||
|
||||
### 2. 客户端程序 (`NaviGlassClient`)
|
||||
**语言**: C++
|
||||
* **多线程架构**: `CameraThread`, `AudioCaptureThread`, `IMUThread` 独立运行。
|
||||
* **断网重连**:
|
||||
* WebSocket 连接具备指数退避 (Exponential Backoff) 重连机制。
|
||||
* *注: 底层 WiFi 接口需依赖 OS 脚本保活。*
|
||||
* **音频流**: 20ms 分包 (320 samples @ 16kHz),确保低延迟传输。
|
||||
|
||||
---
|
||||
|
||||
### 14. <20>箔辣<E7AE94>滚<EFBFBD>銝舘䌊<E88898>臬𢆡<E887AC>滨蔭 (Day 16) <20>?
|
||||
#### <20>桅<EFBFBD> 1嚗𡁜𢆡<F0A1819C><F0A286A1>曎<EFBFBD>亙膥頝臬<E9A09D>
|
||||
蝔见<EFBFBD>蝻𤥁<EFBFBD><EFBFBD>嗡蝙<EFBFBD>?`/lib32/ld.so.1`嚗䔶<EFBFBD>撘<EFBFBD><EFBFBD>烐踎摰鮋<EFBFBD>頝臬<EFBFBD>銝?`/lib/ld-musl-riscv32.so.1`嚗<EFBFBD>紡<EFBFBD>?`not found` <20>躰秤<E8BAB0>?
|
||||
**閫<><E996AB><EFBFBD>寞<EFBFBD>**嚗?```makefile
|
||||
# src/Makefile 瘛餃<E7989B>
|
||||
LDFLAGS += -Wl,--dynamic-linker=/lib/ld-musl-riscv32.so.1
|
||||
```
|
||||
|
||||
#### <20>桅<EFBFBD> 2嚗朞䌊<E69C9E>臬𢆡<E887AC>寞<EFBFBD><E5AF9E>㗇𥋘
|
||||
|
||||
| <20>寞<EFBFBD> | 蝏𤘪<E89D8F> | <20>桅<EFBFBD> |
|
||||
|------|------|------|
|
||||
| `/etc/init.d/rc.final` | <20>?憭梯揖 | 撖潸稲 SD <20>⊥<EFBFBD>瘚见仃韐伐<E99F90><E4BC90><EFBFBD><EFBFBD>滚<EFBFBD><E6BB9A>箔辣 |
|
||||
| `crontab @reboot` | <20>?憭梯揖 | BusyBox crond 銝齿𣈲<E9BDBF><F0A388B2>迨霂剜<E99C82> |
|
||||
| `/etc/init.d/avaota` + `load_script.conf` | <20>?<3F>𣂼<EFBFBD> | 蝔喳<E89D94><E596B3>舫<EFBFBD> |
|
||||
|
||||
#### <20>𣂼<EFBFBD><F0A382BC>寞<EFBFBD>霂西圾
|
||||
|
||||
**1. <20>𥕦遣 init <20>𡁏𧋦**嚗?```bash
|
||||
cat > /etc/init.d/avaota << 'EOF'
|
||||
#!/bin/sh /etc/rc.common
|
||||
START=99
|
||||
start() {
|
||||
sleep 15
|
||||
ulimit -c 0
|
||||
/mnt/UDISK/app/avaota_client > /tmp/avaota.log 2>&1 &
|
||||
}
|
||||
stop() {
|
||||
killall avaota_client
|
||||
}
|
||||
EOF
|
||||
chmod +x /etc/init.d/avaota
|
||||
```
|
||||
|
||||
**2. 瘛餃<E7989B><E9A483>啣鍳<E595A3>典<EFBFBD>銵?*嚗?```bash
|
||||
echo "avaota" >> /etc/init.d/load_script.conf
|
||||
```
|
||||
|
||||
#### <20>喲睸蝏誯<E89D8F>
|
||||
- **<2A>臬𢆡撱嗉<E692B1>**嚗𡁻<E59A97>閬?15 蝘垍<E89D98>敺<EFBFBD>𨺗憸𤑳頂蝏笔<E89D8F>憪见<E686AA>嚗? 蝘雴<E89D98>憭<EFBFBD><E686AD>
|
||||
- **蝳<>鍂 core dump**嚗䫤ulimit -c 0` <20>脫迫蝔见<E89D94>撏拇<E6928F><E68B87>烐說 overlay <20><>躹
|
||||
- **WiFi <20>芸𢆡餈墧𦻖**嚗𡁶頂蝏煺<E89D8F>靽嘥<E99DBD><E598A5>滨蔭<E6BBA8>?`/etc/wifi/wifimg.config`嚗峕<EFBFBD><EFBFBD><EFBFBD><EFBFBD>刻<EFBFBD><EFBFBD>砌葉<EFBFBD>滨蔭
|
||||
- **<2A>踹<EFBFBD> rc.final**嚗帋<E59A97>撟脫贋 SD <20>?mdev <20>嘥<EFBFBD><E598A5>𡝗<EFBFBD>蝔?
|
||||
#### <20>祉<EFBFBD><E7A589>滚𦛚<E6BB9A>券<EFBFBD>蝵?
|
||||
靽格㺿 `main.cpp` <20>舀<EFBFBD> frp <20><><EFBFBD>蝛輸<E89D9B>𧶏<EFBFBD>
|
||||
```cpp
|
||||
// 靽格㺿<E6A0BC>?const char* SERVER_HOST = "192.168.110.188";
|
||||
// 靽格㺿<E6A0BC>?const char* SERVER_HOST = "8.148.25.142";
|
||||
```
|
||||
|
||||
**frpc.toml <20>滨蔭**嚗?```toml
|
||||
serverAddr = "8.148.25.142"
|
||||
serverPort = 7000
|
||||
auth.token = "your_token"
|
||||
|
||||
[[proxies]]
|
||||
name = "avaota_server"
|
||||
type = "tcp"
|
||||
localIP = "127.0.0.1"
|
||||
localPort = 8081
|
||||
remotePort = 8081
|
||||
|
||||
[[proxies]]
|
||||
name = "avaota_imu_udp"
|
||||
type = "udp"
|
||||
localIP = "127.0.0.1"
|
||||
localPort = 12345
|
||||
remotePort = 12345
|
||||
```
|
||||
|
||||
> **Day 17 銵亙<E98AB5>**嚗𡁜<E59A97>憿餅溶<E9A485>?UDP 12345 隞<><E99A9E>嚗<EFBFBD>炏<EFBFBD>?IMU <20>唳旿<E594B3>䭾<EFBFBD><E4ADBE>朞<EFBFBD><E69C9E>祉<EFBFBD>隡㰘<E99AA1><E3B098><EFBFBD><EFBFBD><EFBFBD>園<EFBFBD><E59C92>典<EFBFBD>蝵烐<E89DB5><E78390>∪膥<E288AA>脩<EFBFBD>憓坔<E68693><E59D94>?UDP 12345 蝡臬藁<E887AC>?- <20>滚𦛚<E6BB9A>典<EFBFBD><E585B8><EFBFBD><EFBFBD>~15 FPS (<28>㗇<EFBFBD><E39787>拍鍂)
|
||||
- 撖潸⏛雿㯄<E99BBF>嚗𡁏<E59A97><F0A1818F><EFBFBD><EFBFBD><EFBFBD>⊿▼
|
||||
|
||||
---
|
||||
|
||||
## <20><> <20><><EFBFBD><EFBFBD><EFBFBD>皞?
|
||||
### 蝖砌辣<E7A08C><E8BEA3>﹝
|
||||
- [<5B>典<EFBFBD> V821 Datasheet](https://www.aw-ol.com)
|
||||
- [ICM-42688 Datasheet](https://invensense.tdk.com)
|
||||
- [MAX98357A Datasheet](https://www.maximintegrated.com)
|
||||
- [GC2083 Datasheet](http://www.galaxycore.com.cn)
|
||||
|
||||
### 頧臭辣<E887AD><E8BEA3>﹝
|
||||
- [ALSA Documentation](https://www.alsa-project.org)
|
||||
- [MPP Framework Guide](https://github.com/allwinner)
|
||||
- [Linux GPIO Subsystem](https://www.kernel.org/doc/html/latest/driver-api/gpio/)
|
||||
|
||||
### 撘<><E69298>烐𠯫敹?- [x] [Day1.md](file:///d:/CodingProjects/Antigravity/NaviGlass/Docs/DevLogs/Day1.md) - SDK蝻𤥁<E89DBB><F0A4A581>臬<EFBFBD><E887AC>剖遣
|
||||
- [x] [Day2.md](file:///d:/CodingProjects/Antigravity/NaviGlass/Docs/DevLogs/Day2.md) - 蝵𤑳<E89DB5>摨梶<E691A8>霂?- [x] [Day3.md](file:///d:/CodingProjects/Antigravity/NaviGlass/Docs/DevLogs/Day3.md) - I2S<32>喲<EFBFBD>颲枏枂
|
||||
- [x] [Day4.md](file:///d:/CodingProjects/Antigravity/NaviGlass/Docs/DevLogs/Day4.md) - 暻血<E69ABB>憌?IMU
|
||||
- [x] [Day5.md](file:///d:/CodingProjects/Antigravity/NaviGlass/Docs/DevLogs/Day5.md) - GC2083<38><33><EFBFBD>憭?- [x] [Day6.md](file:///d:/CodingProjects/Antigravity/NaviGlass/Docs/DevLogs/Day6.md) - 蝖砌辣撉諹<E69289>
|
||||
- [x] [Day7.md](file:///d:/CodingProjects/Antigravity/NaviGlass/Docs/DevLogs/Day7.md) - 撌亙<E6928C><E4BA99>暸<EFBFBD>蝵?- [x] [Day8.md](file:///d:/CodingProjects/Antigravity/NaviGlass/Docs/DevLogs/Day8.md) - <20>港<EFBFBD>蝻𤥁<E89DBB><F0A4A581>𣂼<EFBFBD>
|
||||
- [x] [Day9.md](file:///d:/CodingProjects/Antigravity/NaviGlass/Docs/DevLogs/Day9.md) - **musl 撌亙<E6928C><E4BA99>曆耨憭?+ <20>蹂<EFBFBD>瘚贝<E7989A><E8B49D>朞<EFBFBD>** <20>?- [x] [Day10.md](file:///d:/CodingProjects/Antigravity/NaviGlass/Docs/DevLogs/Day10.md) - **<2A>喲<EFBFBD> I/O <20>躰秤靽桀<E99DBD> + 蝔见<E89D94>蝔喳<E89D94><E596B3>扳<EFBFBD><E689B3>?* <20>?- [x] [Day11.md](file:///d:/CodingProjects/Antigravity/NaviGlass/Docs/DevLogs/Day11.md) - **WebSocket 撏拇<E6928F>靽桀<E99DBD> + <20>喲<EFBFBD><E596B2>剜𦆮摰䂿緵** <20>?- [x] [Day12.md](file:///d:/CodingProjects/Antigravity/NaviGlass/Docs/DevLogs/Day12.md) - **<2A>喲<EFBFBD><E596B2>䂿㴓<E482BF><E3B493><EFBFBD> + <20>砍ㄟ<E7A08D>冽毽<E586BD>喳膥靽桀<E99DBD>** <20>?- [x] [Day13.md](file:///d:/CodingProjects/Antigravity/NaviGlass/Docs/DevLogs/Day13.md) - **TTS 鈭衤辣敺芰㴓<E88AB0>餃<EFBFBD>靽桀<E99DBD> + 摰X<E691B0>蝡舫<E89DA1>蝻枏<E89DBB>** <20>?- [x] [Day14.md](file:///d:/CodingProjects/Antigravity/NaviGlass/Docs/DevLogs/Day14.md) - **憿寧𤌍皜<F0A48C8D><E79A9C> + TTS 瘚<><E7989A><EFBFBD>剜𦆮** <20>?- [x] [Day15.md](file:///d:/CodingProjects/Antigravity/NaviGlass/Docs/DevLogs/Day15.md) - **撖潸⏛璅∪<E79285><E288AA>扯<EFBFBD>隡睃<E99AA1> (頝喳葷<E596B3>箏<EFBFBD>)** <20>?- [x] [Day16.md](file:///d:/CodingProjects/Antigravity/NaviGlass/Docs/DevLogs/Day16.md) - **<2A>箔辣<E7AE94>滚<EFBFBD>銝舘䌊<E88898>臬𢆡<E887AC>滨蔭** <20>?- [x] [Day17.md](file:///d:/CodingProjects/Antigravity/NaviGlass/Docs/DevLogs/Day17.md) - **摰文<E691B0>瘚贝<E7989A>+<2B>脤<EFBFBD>霂剝𨺗靽桀<E99DBD>+IMU UDP隞<50><E99A9E>** <20>?- [ ] [Day19.md](file:///d:/CodingProjects/Antigravity/NaviGlass/Docs/DevLogs/Day19.md) - **隡㰘<E99AA1>隡睃<E99AA1> + Python GIL <20>園<EFBFBD>霂𦠜鱏** <20>𩤃<EFBFBD>
|
||||
|
||||
---
|
||||
|
||||
### 15. Python GIL <20>扯<EFBFBD><E689AF>園<EFBFBD>霂𦠜鱏 (Day 19) <20>𩤃<EFBFBD>
|
||||
|
||||
#### <20>桅<EFBFBD><E6A185>𤑳緵
|
||||
|
||||
<EFBFBD>滚𦛚<EFBFBD>冽<EFBFBD>霂閧𤩅<EFBFBD>枏紡<EFBFBD>芣𧒄<EFBFBD>𤑳緵<EFBFBD>扯<EFBFBD><EFBFBD>桅<EFBFBD>嚗?- **GPU <20>拍鍂<E68B8D><E98D82><EFBFBD> 7%** - 餈靝<E9A488>鈭𡡞<E988AD><F0A1A19E>?- **Python 餈𤤿<E9A488><F0A4A4BF>删鍂 120% CPU** - 雿<><E99BBF><EFBFBD>∪膥<E288AA>?CPU 隞?3%嚗?6 <20>詨⏚<E8A9A8>函<EFBFBD><E587BD><EFBFBD><EFBFBD>嚗?- **撣批<E692A3><E689B9>?FPS 隞?3-4 撣?* - <20>駁𢒰<E9A781>⊿▼銝仿<E98A9D>
|
||||
|
||||
#### <20>孵<EFBFBD><E5ADB5><EFBFBD><EFBFBD>
|
||||
|
||||
```python
|
||||
# app_main.py 蝚?237 銵?frame_processing_executor = ThreadPoolExecutor(max_workers=3, ...)
|
||||
```
|
||||
|
||||
`ThreadPoolExecutor` <20>?Python GIL (<28>典<EFBFBD>閫<EFBFBD><E996AB><EFBFBD>券<EFBFBD>) <20>𣂼<EFBFBD>嚗峕<E59A97>瘜閧<E7989C>甇<EFBFBD>僎銵<E5838E><E98AB5> CPU 撖<><E69296><EFBFBD>衤遙<E8A1A4>∴<EFBFBD>JPEG 蝻𤥁圾<F0A4A581><E59CBE><EFBFBD><EFBFBD><EFBFBD><EFBFBD>暹葡<E69AB9>橒<EFBFBD><E6A992>?
|
||||
#### 閫<><E996AB><EFBFBD>寞<EFBFBD>
|
||||
|
||||
雿輻鍂 **PyNvJpeg** 撠?JPEG 蝻𤥁圾<F0A4A581><E59CBE>宏<EFBFBD>?GPU嚗?
|
||||
```bash
|
||||
pip install pynvjpeg
|
||||
```
|
||||
|
||||
```python
|
||||
import pynvjpeg as nj
|
||||
_nvjpeg = nj.NvJpeg()
|
||||
|
||||
def gpu_decode(jpeg_bytes):
|
||||
return _nvjpeg.decode(jpeg_bytes)
|
||||
|
||||
def gpu_encode(image, quality=85):
|
||||
return _nvjpeg.encode(image, quality)
|
||||
```
|
||||
|
||||
**<EFBFBD>嗆<EFBFBD>?*嚗尠<E59A97> <20>芸<EFBFBD><E88AB8>踝<EFBFBD>敺<EFBFBD><E695BA><EFBFBD>亙<EFBFBD><E4BA99>?
|
||||
---
|
||||
|
||||
**<EFBFBD><EFBFBD><EFBFBD>擧凒<EFBFBD>?*嚗?025-12-23 (Day 19 - 隡㰘<E99AA1>隡睃<E99AA1> + GIL <20>園<EFBFBD>霂𦠜鱏)
|
||||
**憿寧𤌍<E5AFA7>嗆<EFBFBD>?*嚗尠<E59A97>儭?**99% 摰峕<E691B0>嚗<EFBFBD>紡<EFBFBD>芣<EFBFBD>扯<EFBFBD>敺<EFBFBD><E695BA><EFBFBD>?(Python GIL <20>園<EFBFBD>)**
|
||||
**Day 19 <20>湔鰵**嚗関urboJPEG 隡睃<E99AA1><E79D83><EFBFBD>葉<EFBFBD><E89189><EFBFBD>雿㮖耨憭溻<E686AD><E6BABB>IL <20>園<EFBFBD>霂𦠜鱏
|
||||
**敺<><E695BA><EFBFBD>?*嚗䥪yNvJpeg GPU JPEG <20>𣳇<EFBFBD>麄<EFBFBD><E9BA84>TS <20>剛<EFBFBD>靽桀<E99DBD>
|
||||
## 🛠️ 部署与运维
|
||||
|
||||
* **内网穿透**: FRP (Fast Reverse Proxy)。将本地端口 8081 (WS) 和 12345 (UDP) 暴露到公网服务器。
|
||||
* **进程管理**: PM2 (Server), Systemd/Init.d (Client)。
|
||||
* **可视化监控**: Web Dashboard (`/static/index.html`) 提供实时第一视角画面、IMU 姿态、AI 对话日志和系统状态监控。
|
||||
|
||||
117
task_complete.md
117
task_complete.md
@@ -2,21 +2,22 @@
|
||||
|
||||
**项目**:NaviGlass项目开发
|
||||
**平台**:全志 V821 (32-bit RISC-V)
|
||||
**更新时间**:2026-01-05
|
||||
**整体进度**:100%(室内导盲模型训练优化完成)
|
||||
**更新时间**:2026-01-06
|
||||
**整体进度**:100%(室内导盲系统优化交付)
|
||||
|
||||
## 📖 快速导航
|
||||
|
||||
| 章节 | 说明 |
|
||||
|------|------|
|
||||
| [已完成任务](#-已完成任务) | Day 1-27 完成的功能 |
|
||||
| [待验证问题](#️-待验证问题-day-20-优化后) | 当前需要验证的修复 |
|
||||
| [已完成任务](#-已完成任务) | Day 1-28 完成的功能 |
|
||||
| [后续规划](#-后续规划与遗留问题) | 待办 / 债务 / 未来展望 |
|
||||
| [进度统计](#-进度统计) | 各模块完成度 |
|
||||
| [时间线](#-时间线) | 开发时间线 |
|
||||
| [里程碑](#-里程碑) | 关键节点回顾 |
|
||||
| [时间线](#-时间线) | 开发流水账 |
|
||||
|
||||
**相关文档**:
|
||||
- [实现计划详情](file:///d:/CodingProjects/Antigravity/NaviGlass/Docs/implementation_plan_complete.md)
|
||||
- [Day 日志](file:///d:/CodingProjects/Antigravity/NaviGlass/Docs/DevLogs/) (Day1-Day27)
|
||||
- [Day 日志](file:///d:/CodingProjects/Antigravity/NaviGlass/Docs/DevLogs/) (Day1-Day28)
|
||||
|
||||
---
|
||||
|
||||
@@ -413,37 +414,39 @@
|
||||
- [x] **跨平台路径修复**: 解决 Windows 生成 `data.yaml` 在 Linux 报错问题
|
||||
- [x] **文档同步**: 更新《训练指南》至 V2 (22类 + 路径注意)
|
||||
|
||||
---
|
||||
|
||||
## ⏳ 待完成任务
|
||||
|
||||
|
||||
### 性能评估
|
||||
- [ ] CPU 使用率监控(目标: <80%)
|
||||
- [ ] 内存使用监控(目标: <100MB)
|
||||
- [x] 摄像头实际帧率: 客户端 10 FPS, 服务器处理 ~15 FPS
|
||||
- [x] IMU 采样率: 10Hz
|
||||
|
||||
### 稳定性验证
|
||||
- [ ] 长时间运行测试(5分钟基础 / 1小时稳定性 / 内存泄漏检查)
|
||||
- [ ] 异常恢复测试(断网重连 / 设备异常恢复)
|
||||
- [ ] 压力测试(高负载 / 边界条件)
|
||||
### 室内导航优化与 Bug 修复 (Day 28)
|
||||
- [x] **模型精准化**
|
||||
- [x] 切换至 `yolo11l-seg-indoor14.engine` (14类)
|
||||
- [x] 更新 `workflow_indoor.py` 类别映射
|
||||
- [x] 地面检测率提升,解决无路误报
|
||||
- [x] **语音播报修复**
|
||||
- [x] **强制首播**: 解决初始无路提示被节流的问题
|
||||
- [x] **即时反馈**: 启动命令后立即播报“正在寻找可行走区域”
|
||||
- [x] **音频 Pop 消除**: 增加静音填充至 200ms
|
||||
- [x] **崩溃修复**: 移除 undefined class 引用
|
||||
- [x] **终端 WiFi 诊断**
|
||||
- [x] 确认断连不可恢复为 OS 层面问题
|
||||
- [x] 建议部署 `wifi_check.sh` 守护脚本
|
||||
|
||||
---
|
||||
|
||||
## 🔮 未来优化方向
|
||||
## 🛤️ 后续规划与遗留问题
|
||||
|
||||
### 性能优化
|
||||
- [ ] 内存使用优化
|
||||
- [ ] CPU 负载平衡
|
||||
- [ ] 网络带宽管理
|
||||
- [ ] 功耗优化
|
||||
### 🔴 优先待办 (Priority)
|
||||
- [ ] **稳定性验证**: 长时间运行测试(5分钟基础 / 1小时稳定性 / 内存泄漏检查)
|
||||
- [ ] **异常恢复**: 模拟断网重连、设备异常恢复等边界条件
|
||||
- [ ] **WiFi 保活**: 部署终端脚本 `wifi_check.sh` 解决断网不重连问题
|
||||
|
||||
### 硬件升级探索
|
||||
- [ ] 硬件 SPI 替代 GPIO 模拟(提升IMU速度)
|
||||
- [ ] I2C 通信方案重新探索
|
||||
### 🟠 技术债务 (Debt)
|
||||
1. **音频底噪**: 需探索软件降噪 (AEC/ANS) 以消除回环底噪
|
||||
2. **硬件屏蔽**: 扬声器电流声需硬件改进(磁环/布线)
|
||||
3. **代码规范**: 补充单元测试,统一注释风格
|
||||
|
||||
### 🔵 长期探索 (Future)
|
||||
- [ ] **硬件升级**: 尝试硬件 SPI 替代 GPIO 模拟(提升 IMU 采样率)
|
||||
- [ ] **I2C 攻坚**: 重新探索 ICM-42688 的 I2C 通信模式
|
||||
- [ ] **能效优化**: 内存占用优化与 CPU 负载平衡
|
||||
|
||||
---
|
||||
|
||||
## 📊 进度统计
|
||||
|
||||
@@ -474,7 +477,7 @@
|
||||
| 导航模式 | 100% | ✅ Day 15 修复 |
|
||||
| 服务器优化 | 100% | ✅ Day 18-19 完成 |
|
||||
| AI 交互管道 | 100% | ✅ Day 21-22 修复 (GLM-4V/SenseVoice) |
|
||||
| 室内导盲模型 | 100% | ✅ Day 25 训练完成 |
|
||||
| 室内导盲模型 | 100% | ✅ Day 28 优化交付 (14类模型) |
|
||||
|
||||
---
|
||||
|
||||
@@ -595,41 +598,16 @@
|
||||
- **稳定性修复**: 彻底解决 FP16 训练 NaN 问题 (AMP=False)
|
||||
- **完整性验证**: 数据集健康检查 100% 通过
|
||||
|
||||
---
|
||||
|
||||
## 📝 技术债务
|
||||
|
||||
### 需要优化的地方
|
||||
1. **GPIO 模拟性能**: 当前 SPI 速度 ~500kHz,可优化到 1-2MHz
|
||||
2. **错误处理**: 网络断开重连、设备异常恢复、日志完善
|
||||
3. **代码规范**: 需要统一注释风格、需要单元测试
|
||||
|
||||
### 遗留问题
|
||||
1. **音频底噪**: 开启 Playback Switch 可能引入回环底噪,需探索软件降噪 (AEC/ANS)
|
||||
2. **扬声器杂音** (Day 17): 确认为硬件问题,需加磁环或重新布线
|
||||
3. **I2C 问题**: ICM-42688 I2C 模式未解决,保留为技术探索方向
|
||||
4. **硬件 SPI**: 未尝试使用硬件 SPI,可作为性能优化方向
|
||||
### Milestone 24: 室内导盲系统优化交付 ✅
|
||||
**时间**: Day 28
|
||||
**成果**:
|
||||
- **高精度感知**: 切换 14 类模型,地面检测率/避障精准度显著提升
|
||||
- **体验升级**: 修复无路提醒节流 Bug,消除音频 Pop 噪声,交互响应即时化
|
||||
- **系统诊断**: 明确终端 WiFi 断连根因并给出自动化运维方案
|
||||
|
||||
---
|
||||
|
||||
## 🛠️ 开发工具与资源
|
||||
|
||||
### 必备工具
|
||||
- [x] WinSCP - 文件传输
|
||||
- [x] PuTTY - SSH 终端
|
||||
- [x] PhoenixSuit - 固件烧录
|
||||
- [x] VS Code - 代码编辑
|
||||
|
||||
### 参考文档
|
||||
- [x] [全志 V821 Datasheet](https://www.aw-ol.com)
|
||||
- [x] [ICM-42688 Datasheet](https://invensense.tdk.com)
|
||||
- [x] [MAX98357A Datasheet](https://www.maximintegrated.com)
|
||||
- [x] [ALSA Documentation](https://www.alsa-project.org)
|
||||
|
||||
### 开发笔记
|
||||
- [Day1.md](file:///d:/CodingProjects/Antigravity/NaviGlass/Docs/DevLogs/Day1.md) ~ [Day23.md](file:///d:/CodingProjects/Antigravity/NaviGlass/Docs/DevLogs/Day23.md)
|
||||
|
||||
---
|
||||
|
||||
## 📅 时间线
|
||||
|
||||
@@ -660,15 +638,4 @@ Day 24: YOLOE模型训练 ✅ 完成 (问题诊断+数据集筛选)
|
||||
Day 25: 室内导盲模型完成 ✅ 完成 (训练+验证通过)
|
||||
Day 26: 室内导盲服务器集成 ✅ 完成 (IndoorNavigator工作流)
|
||||
Day 27: 模型训练优化与完善 ✅ 完成 (NaN修复+22类适配+路径修复)
|
||||
```
|
||||
|
||||
|
||||
---
|
||||
|
||||
**🎯 当前状态:100% 完成!室内导盲系统已集成并可使用**
|
||||
|
||||
|
||||
|
||||
## ⚠️ 待验证问题
|
||||
|
||||
*(当前无遗留问题,所有已知 BUG 均已修复)*
|
||||
Day 28: 室内导航优化与修复 ✅ 完成 (14类模型+语音修复+音频Pop修复)
|
||||
|
||||
@@ -10,7 +10,7 @@
|
||||
WebSocket: 用于音频/视频流的实时双向传输 (websockets 库)
|
||||
HTTP: 用于控制指令和状态查询
|
||||
|
||||
- 新一代 AI 交互管道 (Day 21+ 架构)
|
||||
- 新一代 AI 交互管道 (Day 28+ 架构)
|
||||
大语言模型 (LLM): GLM-4.6v-Flash (通过 zai-sdk 调用智谱 AI,支持多模态视觉理解)
|
||||
语音识别 (ASR): SenseVoiceSmall (通过 funasr 本地部署,高精度中文识别)
|
||||
语音活动检测 (VAD): Silero VAD (PyTorch 实现,含 300ms 环形缓冲,抗噪能力强)
|
||||
@@ -18,6 +18,10 @@
|
||||
|
||||
- 计算机视觉 (CV)
|
||||
核心引擎: Ultralytics YOLO (YOLO11 / YOLOv8)
|
||||
模型矩阵:
|
||||
- 盲道导航: yolo11l-seg-blind.engine
|
||||
- 室内导盲: yolo11l-seg-indoor14.engine (14类)
|
||||
- 红绿灯: yolov8n.engine
|
||||
任务类型: 目标检测 (Detection) + 语义分割 (Segmentation)
|
||||
推理加速: TensorRT (NVIDIA 深度学习推理引擎,FP16半精度 + .engine 模型文件)
|
||||
图像处理:
|
||||
|
||||
Reference in New Issue
Block a user