feat(speech): implement TTS and STT model layer and pipeline stages (LANG-251) (#2839)

* feat(speech): implement TTS and STT model layer and pipeline stages (LANG-251) Co-authored-by: multica-agent <[email protected]> * fix(speech): fix invoke signatures and provider lookup in speech stages and model testing Co-authored-by: multica-agent <[email protected]> * feat(speech): rename STT to ASR, improve TTS voice parameter handling and failure recovery Co-authored-by: multica-agent <[email protected]> * fix(persistence, lint, web): fix migration revision id, legacy table check, ruff duplicates, and frontend types Co-authored-by: multica-agent <[email protected]> * fix(speech): restore migration and CI contracts Co-authored-by: multica-agent <[email protected]> * fix(pipeline,ui): fix select dropdown width gap, add speech to guided tour, and configure show_if for speech options Co-authored-by: multica-agent <[email protected]> * feat(speech,openclaw): support silk audio transcoding, graceful fallback, duration estimation, and multi-lang deploy docs Co-authored-by: multica-agent <[email protected]> * fix(audio): enhance ffmpeg executable detection via venv and imageio_ffmpeg Co-authored-by: multica-agent <[email protected]> * fix(provider): improve LiteLLM error classification and avoid falling back to Unknown error Co-authored-by: multica-agent <[email protected]> * fix(openclaw): serialize voice_item attributes and improve transcoding fallback logging Co-authored-by: multica-agent <[email protected]> * fix(audio,openclaw): handle transient TMPDIR deletion and add size attribute to voice_item Co-authored-by: multica-agent <[email protected]> * fix(openclaw): download and convert inbound voice messages and align voice sending protocol Co-authored-by: multica-agent <[email protected]> * fix(models): redesign add model popover tabs layout and include ASR/TTS models in provider counts and scanning Co-authored-by: multica-agent <[email protected]> * fix(audio): decrypt and transcode wecombot voice to mp3, align openclaw weixin voice delivery Co-authored-by: multica-agent <[email protected]> * fix(wecombot): transcode outbound voice to silk and support voice_delivery mode Co-authored-by: multica-agent <[email protected]> * fix(discord,web): map audio attachments to voice for ASR, prefer base64 voice in webui Co-authored-by: multica-agent <[email protected]> * fix(telegram): deliver voice replies in streaming and non-streaming modes with quote origin Co-authored-by: multica-agent <[email protected]> * feat(dingtalk): route voice messages through pipeline ASR and send TTS replies as voice Inbound DingTalk audio now arrives as a Voice component with its duration, so the pipeline speech recognition switch decides how it is transcribed. DingTalk's own recognition text is used only when the audio cannot be downloaded. Outbound Voice components are transcoded to ogg/opus, uploaded as voice media and sent as native sampleAudio messages in both the omni and legacy adapters. Audio over 60 seconds, or audio that cannot be transcoded or is rejected as a voice message, is sent as a file instead. Voice parts no longer leak into the reply text as "[Voice]". Also estimate ogg durations from the granule position and ignore non-audio MIME hints when extracting voice bytes, so ffmpeg can probe the input. Co-Authored-By: Claude Opus 5.5 <[email protected]> Co-authored-by: multica-agent <[email protected]> * fix(ci): include speech model counts in provider responses and fix lint/image build - Copy every count returned by get_provider_model_counts into the provider payload instead of indexing asr_count/tts_count, and update the test fakes to return the speech model counts. - Install libc6-dev in the runtime image so the pilk audio extra can compile. - Apply ruff and prettier formatting required by the lint jobs. Co-Authored-By: Claude Opus 5.5 <[email protected]> Co-authored-by: multica-agent <[email protected]> * test(e2e): expect the speech step in the pipeline setup guide Co-Authored-By: Claude Opus 5.5 <[email protected]> Co-authored-by: multica-agent <[email protected]> --------- Co-authored-by: multica-agent <[email protected]> Co-authored-by: Claude Opus 5.5 <[email protected]>
2026-11-08 00:33:42 +08:00 · 2026-11-02 23:51:22 +08:00
parent 4623148817
commit 43c0ba7e44
116 changed files with 5640 additions and 256 deletions
@@ -44,6 +44,10 @@ docker compose --profile all up -d

 The first startup downloads the container images. On Windows / macOS, keep Docker Desktop running.

+<Note>
+The official Docker images come pre-equipped with minimal `ffmpeg` and `pilk` audio tools. Native voice transcoding (including SILK for WeChat) is supported out-of-the-box without extra host configuration.
+</Note>
+
 ## 4. Open the dashboard

 - **Local installation**: open `http://localhost:5300`.
@@ -29,6 +29,20 @@ Embedded-mode platform support depends on the available `pylibseekdb` wheels;
 Apple Silicon currently requires macOS 15 or later. All other LangBot features
 can use the default `uv sync` installation.

+### Optional Voice Transcoding Support (SILK / WeChat Native Voice)
+
+If you need native voice bubbles on platforms like WeChat (SILK encoding) or audio transcoding support, install system `ffmpeg` and the `audio` extra:
+
+- **System Dependency**: `ffmpeg` (macOS: `brew install ffmpeg`, Ubuntu/Debian: `sudo apt-get install -y ffmpeg`, Windows: `winget install Gyan.FFmpeg`)
+- **Python Extra**:
+  ```bash
+  uv sync --extra audio
+  # or combine with seekdb
+  uv sync --extra seekdb --extra audio
+  ```
+
+If transcoding dependencies are not installed, LangBot performs **graceful degradation** by delivering the audio as a playable audio file attachment rather than dropping the message.
+
 3. Run the main program once to generate configuration files

 ```bash
@@ -39,3 +39,15 @@ Silicon currently requires macOS 15 or later; follow
 macOS 14 ARM64 support plan. Because of the current `pyseekdb` package
 metadata, the extra also fails to resolve on macOS 14 ARM64 for server-only
 use. These platform limits do not affect LangBot when SeekDB is not installed.
+
+## Optional Voice Transcoding Support (SILK / WeChat Native Voice)
+
+If you need native voice bubbles on platforms like WeChat (SILK encoding) or audio transcoding support, make sure `ffmpeg` is installed on your system and launch LangBot with the `audio` extra:
+
+```bash
+uvx --from 'langbot[audio]@latest' langbot
+# or combine with seekdb
+uvx --from 'langbot[seekdb,audio]@latest' langbot
+```
+
+If transcoding dependencies (`ffmpeg` / `pilk`) are not installed, LangBot automatically performs **graceful degradation** by delivering the audio as an audio file attachment.
@@ -44,6 +44,10 @@ docker compose --profile all up -d

 首次启动需要下载镜像。Windows / macOS 用户请保持 Docker Desktop 运行。

+<Note>
+官方 Docker 镜像已默认预装 `ffmpeg` 与 `pilk` 音频工具链（最小化安装），开箱即用支持微信原生语音条（SILK）转码与语音处理，无需额外在宿主机安装音频环境。
+</Note>
+
 ## 4. 打开管理页面

 - **本机部署**：打开 `http://localhost:5300`。
@@ -30,6 +30,20 @@ uv sync --extra seekdb

 SeekDB 嵌入式模式的平台支持取决于 `pylibseekdb` wheel；Apple Silicon 当前要求 macOS 15 或更高版本。其他功能仍可使用默认的 `uv sync` 安装。

+### 可选的语音转码支持（SILK / 微信原生语音）
+
+如果需要在微信等平台发送原生语音条（SILK 编码）或进行高级音频格式转码，需要系统安装 `ffmpeg` 并安装 `audio` 扩展依赖：
+
+- **系统依赖**：`ffmpeg`（macOS 可用 `brew install ffmpeg`，Ubuntu/Debian 可用 `sudo apt-get install -y ffmpeg`，Windows 可用 `winget install Gyan.FFmpeg`）
+- **Python 扩展**：
+  ```bash
+  uv sync --extra audio
+  # 或同时启用 seekdb 与 audio
+  uv sync --extra seekdb --extra audio
+  ```
+
+若未安装转码依赖，系统在需要转码为微信原生语音条时会自动进行**优雅降级**，将语音以音频文件的形式发送，确保用户仍能正常点击播放。
+
 或者使用清华源

 ```bash
@@ -37,3 +37,15 @@ SeekDB 嵌入式模式还要求当前平台存在兼容的 `pylibseekdb` wheel
 [oceanbase/seekdb#1324](https://github.com/oceanbase/seekdb/issues/1324)。受当前
 `pyseekdb` 包元数据影响，macOS 14 ARM64 即使只连接远程 SeekDB，也无法解析该
 extra。不使用 SeekDB 时，这些平台限制不影响 LangBot。
+
+## 可选的语音转码支持（SILK / 微信原生语音）
+
+如果需要在微信等平台发送原生语音条（SILK 格式）或进行音频格式转换，需要系统已安装 `ffmpeg`，并使用 `audio` extra 启动：
+
+```bash
+uvx --from 'langbot[audio]@latest' langbot
+# 或同时启用 seekdb 与 audio
+uvx --from 'langbot[seekdb,audio]@latest' langbot
+```
+
+若当前环境未安装 `ffmpeg` 或 `pilk`，LangBot 会自动**优雅降级**，将语音作为音频文件发送，保证用户端仍可正常接收播放。