feat(voice): 语音silent降级信号(C2) — provider仅在结果为空时出现
- voice/Chain+Service: STTWithProvider/TTSWithProvider(真实档位cloud/local/edge-tts/espeak/silent), 旧方法改薄包装
- gateway: transcribe空文本带{text,provider} / voice_output空音频带provider / speech空音频tts_empty+X-Voice-Provider头
- middleware: CORS Access-Control-Expose-Headers: X-Voice-Provider
- 可选接口 voiceProviderReporter(type断言透传), 自定义实现回退silent
- 新增测试10个(provider_signal/voice_signal), build/vet/test全绿+交叉编译
- Docs/api-client v0.23 + api-reference CORS 同步
This commit is contained in:
AskaEth committed
1 parent
00f784b55f
commit
99f222761e
9 files changed
+345
-25
No files matched your search
+4
-3
@@ -30,8 +30,8 @@
|
||||
| 7 | `/api/v1/sessions/{id}` | DELETE | JWT(owner/admin) | ✅ | **删除次级会话及其消息** → `{data:{deleted:<id>}}`；主会话不可删回 `403 forbidden`；未注入 HistoryStore 回 `501` |
|
||||
| 8 | `/api/v1/sessions/{id}` | PATCH | JWT(owner/admin) | ✅ | **会话改名**（A4）：body `{title}`（trim、≤80 rune，空/超长回 `400`）→ `{data:{id,title}}`；不存在回 `404`；主会话(`id="web"`)可改名但不可删；未注入 HistoryStore 回 `501`。另：会话**首条用户消息到达时自动命名** title（取前 20 rune 折叠空白；纯空白回落 `主会话`/`次级会话`），列表不再空名 |
|
||||
| 9 | `/api/v1/personas` | GET | JWT | ✅ | `{data:[persona...]}` |
|
||||
| 10 | `/api/v1/audio/transcribe` | POST | JWT(owner) | ✅ | **语音→文本**：body `{audio:"<base64>",format?,deviceId?}` → `{data:{text}}`（走后端 STT，百炼 `qwen-audio-3.0-asr-flash`） |
|
||||
| 11 | `/api/v1/audio/speech` | POST | JWT(owner) | ✅ | **文本→音频**：body `{text,voice?,format?,instruction?}` → `200` 音频字节（`Content-Type: audio/<format>`，非 JSON envelope，走后端 TTS，百炼 `cosyvoice-v3-flash`，复刻音色可带 `instruction` 情绪指令） |
|
||||
| 10 | `/api/v1/audio/transcribe` | POST | JWT(owner) | ✅ | **语音→文本**：body `{audio:"<base64>",format?,deviceId?}` → `{data:{text, provider?}}`（走后端 STT，百炼 `qwen-audio-3.0-asr-flash`；**当结果为空文本/silent 降级时响应体带 `provider`**（如 `"silent"`，非空文本省略），客户端据此提示「没识别到声音」） |
|
||||
| 11 | `/api/v1/audio/speech` | POST | JWT(owner) | ✅ | **文本→音频**：body `{text,voice?,format?,instruction?}` → `200` 音频字节（`Content-Type: audio/<format>`，非 JSON envelope，走后端 TTS，百炼 `cosyvoice-v3-flash`，复刻音色可带 `instruction` 情绪指令；**静默降级空音频时仍 `500 tts_empty`，并附 `X-Voice-Provider: silent` 响应头**，客户端据此提示「没生成声音」） |
|
||||
| 12 | `/api/v1/audio/voice-clone` | POST | JWT(owner) | ✅ | **声音复刻建音色**：body `{url:"<公网参考音频URL>",prefix?}` → `{data:{voiceId}}`（走百炼 `voice-enrollment` create_voice） |
|
||||
|
||||
> **语音 REST 源（`/api/v1/audio/*`，均 owner-only + 审计）**：原生/移动端用它上传/获取音频，**绕开 WS 4KB 帧限**。
|
||||
@@ -136,7 +136,7 @@
|
||||
- **上行（客户端→服务端）**: `mode_switch` / `chat`（跑 loop 并广播流式） / `voice_input` / `subagent_ask` / `task_output_query` / `interrupt`。
|
||||
- **`interrupt`**（上行，客户端→服务端）：`{sessionId}` → 会话编排器中断（design/15 §3）——取消该会话当前在跑的任务（context cancel）、清空其待处理队列、回 `session_interrupted`；用户说「停下/算了/换个话题」时发。命中空闲会话为 no-op。
|
||||
- **`voice_input`**（上行，客户端→服务端）：`_{audio?, format?, text?, sessionId?, deviceId?}`。`audio` 为 base64（或 data: URI）音频，`format` 为其编码（`wav`/`mp3`/`ogg`）；`text` 为已转写文本时跳过 STT；`deviceId` 为**客户端具体端标识**（design/16 §4.5，如 `voiceassist:myapp:v1`，可选），随该语音轮次落库、聊天气泡可标注来源。服务端走 STT→agent 聊天→TTS（design/06 §1），回复经 `voice_output` 定向回该连接。**真 STT/TTS 已接通阿里百炼**（`DASHSCOPE_API_KEY` 必填：STT=`qwen-audio-3.0-asr-flash`、TTS=`cosyvoice-v3-flash`+`longxiaochun_v3`）；无凭据时后端走降级链（cloud → edge-tts/espeak（未实现）→ `silent` 空音频）。无注入语音服务时回 `error` 帧（`voice unavailable`）。
|
||||
- **`voice_output`**（下行，服务端→客户端）：`{audio, format, text, reply}`。`audio` 为 base64 合成音频，`format` 为其编码，`text` 为识别文本，`reply` 为助手回复文本。
|
||||
- **`voice_output`**（下行，服务端→客户端）：`{audio, format, text, reply, provider?}`。`audio` 为 base64 合成音频，`format` 为其编码，`text` 为识别文本，`reply` 为助手回复文本；**当合成音频为空（silent 降级）时带 `provider`**（如 `"silent"`，非空音频省略）——客户端据此提示「没生成声音」，但该帧**仍照常发出、不算业务错误**。
|
||||
- **向后兼容**：未映射/无法确定的事件仍走 `stream_chunk` + `stream_end`（非流式回退路径）。
|
||||
|
||||
---
|
||||
@@ -211,6 +211,7 @@
|
||||
- 语音：上行 `voice_input`（base64 音频或已转写 text），下行 `voice_output`（合成音频 base64 + text）。
|
||||
|
||||
## 5. 变更记录
|
||||
- `v0.23`（**C2 语音 silent 降级信号**）：三处统一用 **`provider`** 只在「结果为空」时出现（非空/成功时才省略）——`POST /api/v1/audio/transcribe` 空文本响应体带 `{text:"", provider:"silent"}`；WS `voice_output` 空音频 payload 带 `provider`；`POST /api/v1/audio/speech` 空音频仍 `500 tts_empty` 并加 `X-Voice-Provider: silent` 响应头（CORS `Access-Control-Expose-Headers` 已放开）。`voice.Chain/Service` 新增 `STTWithProvider/TTSWithProvider`（返回真实档位 cloud/local/edge-tts/espeak/silent），网关用可选接口 `voiceProviderReporter` 透传，自定义实现回退 `silent`。**含 `X-Voice-Provider`（`apperr` 包装、`providerSignal`）；`DASHSCOPE_API_KEY` 无/降级时信号为 `silent`。**（`go test -race` 在本机受内核 TSAN VMA 限制失败，非本代码问题。）
|
||||
- `v0.22`（**B3 会话列表 lastMessage 预览**）：`GET /api/v1/sessions` 与 `/api/v1/admin/sessions` 每个会话新增可选 **`lastMessage`** `{content,role,createdAt}`（content 折叠空白截前 60 rune + `…`；空会话省略），**单条 SQL**（LEFT JOIN 取每会话 `ORDER BY created_at DESC,rowid DESC LIMIT 1`，无 N+1，走 `idx_chat_msgs_session_created` 覆盖索引）。`unreadCount` 需已读游标设计，后置。
|
||||
- `v0.21`（**A4 会话 title**）：`PATCH /api/v1/sessions/{id}`（**新增**，body `{title}` ≤80 rune → `{data:{id,title}}`；空/超长 400、不存在 404、主会话可改名）；会话**首条用户消息到达时自动命名**（`history.append` 前 `ensureTitle`：折叠空白取前 20 rune，纯空白回落 `主会话`/`次级会话`），仅首次命名、不覆盖。`HistoryStore` 新增 `RenameSession`；隔离会话（`sess-xxx`）不在 `chat_sessions` 有行、不自动命名（其本不在 `/sessions` 列表）。**边界**：前端 `createSubSession(title?)` 的 POST body 含 `title` 但后端暂忽略（会被首条消息自动命名覆盖），未处理。
|
||||
- `v0.20`（**A3 消息 mode 落库**）：`POST /api/v1/chat` 与 `ws.chat` 上行新增**可选 `mode`**（行为模式，design/01 §3 / design/16 §5.1）。后端把每条持久化消息的 `Mode` **真实写入**（IM 路径恒 `chat`；Web 路径经 ctx 透传本轮 `mode`，缺省回落 `chat`；恢复/续接路径沿用发起轮次的 `mode`），消息 DTO、`message_persisted`、`channel_persisted` 的 `mode` 不再恒空，供前端按模式渲染气泡（**仅 `chat` 解析 `<p>` 拆气泡**，其它模式原样整条）。
|
||||
|
||||
@@ -15,7 +15,7 @@
|
||||
- 成功：`{"data": <payload>}`（HTTP 200 / 201 / 202 / 204）。
|
||||
- 失败：`{"error": {"code": <string>, "message": <string>}}`（HTTP 400/401/403/404/409/500/501 等）。
|
||||
- **错误码**：业务错误带稳定字符串 `code`（如 `unauthorized` / `not_found` / `bad_request` / `chat_busy` / `not_implemented`），便于客户端分支；`message` 面向人读。
|
||||
- **CORS**：`Access-Control-Allow-Origin` 为配置的 `ALLOWED_ORIGINS` 白名单（`*` 时放开）；`Allow-Methods: GET, POST, PUT, DELETE, OPTIONS`；`Allow-Headers: Authorization, Content-Type`；`Max-Age: 600`。`OPTIONS` 预检直接回 `204`。同源/无 Origin 请求放行。**注意：CORS 只允许 `Authorization, Content-Type` 头——若客户端要额外发送 `X-API-Version`，需自行在网关侧放开或仅作为文档惯例（当前服务端**不校验**该头）。**
|
||||
- **CORS**：`Access-Control-Allow-Origin` 为配置的 `ALLOWED_ORIGINS` 白名单（`*` 时放开）；`Allow-Methods: GET, POST, PUT, DELETE, OPTIONS`；`Allow-Headers: Authorization, Content-Type`；`Access-Control-Expose-Headers: X-Voice-Provider`（让跨域浏览器能读到该语音降级信号头，非跨域场景不受影响）；`Max-Age: 600`。`OPTIONS` 预检直接回 `204`。同源/无 Origin 请求放行。**注意：CORS 只允许 `Authorization, Content-Type` 头——若客户端要额外发送 `X-API-Version`，需自行在网关侧放开或仅作为文档惯例（当前服务端**不校验**该头）。**
|
||||
- **请求日志**：每个请求由 `logging` 中间件记录一行（method/path/status/duration_ms），并计入可观测性指标。panic 由 `recover` 中间件兜底为 `500 {error:{code:internal,...}}`，不崩溃进程。
|
||||
- **超时**：WS 连接 ping 周期 54s、pong 等待 60s、写超时 10s；读上限 4096 字节/帧。
|
||||
|
||||
|
||||
@@ -25,8 +25,16 @@ func NewChain(steps ...Provider) *Chain {
|
||||
|
||||
// STT walks the chain for a transcription.
|
||||
func (c *Chain) STT(ctx context.Context, audio Audio) (string, error) {
|
||||
text, _, err := c.STTWithProvider(ctx, audio)
|
||||
return text, err
|
||||
}
|
||||
|
||||
// STTWithProvider is STT but also reports the name of the provider rung that
|
||||
// produced the transcription (cloud / local / edge-tts / silent), so a caller can
|
||||
// surface which tier actually served the request.
|
||||
func (c *Chain) STTWithProvider(ctx context.Context, audio Audio) (string, string, error) {
|
||||
if len(c.steps) == 0 {
|
||||
return "", ErrProviderFailed
|
||||
return "", "", ErrProviderFailed
|
||||
}
|
||||
var lastErr error
|
||||
for _, p := range c.steps {
|
||||
@@ -36,11 +44,11 @@ func (c *Chain) STT(ctx context.Context, audio Audio) (string, error) {
|
||||
}
|
||||
text, err := stt.STT(ctx, audio)
|
||||
if err == nil {
|
||||
return text, nil
|
||||
return text, p.Name(), nil
|
||||
}
|
||||
lastErr = err
|
||||
}
|
||||
return "", fmt.Errorf("voice: %w: %v", ErrProviderFailed, lastErr)
|
||||
return "", "", fmt.Errorf("voice: %w: %v", ErrProviderFailed, lastErr)
|
||||
}
|
||||
|
||||
// CloneVoice creates a custom voice via the cloud provider's voice-enrollment
|
||||
@@ -55,8 +63,15 @@ func (c *Chain) CloneVoice(ctx context.Context, referenceURL, prefix string) (st
|
||||
|
||||
// TTS walks the chain for synthesized audio.
|
||||
func (c *Chain) TTS(ctx context.Context, text string) (Audio, error) {
|
||||
audio, _, err := c.TTSWithProvider(ctx, text)
|
||||
return audio, err
|
||||
}
|
||||
|
||||
// TTSWithProvider is TTS but also reports the name of the provider rung that
|
||||
// produced the audio.
|
||||
func (c *Chain) TTSWithProvider(ctx context.Context, text string) (Audio, string, error) {
|
||||
if len(c.steps) == 0 {
|
||||
return Audio{}, ErrProviderFailed
|
||||
return Audio{}, "", ErrProviderFailed
|
||||
}
|
||||
var lastErr error
|
||||
for _, p := range c.steps {
|
||||
@@ -66,11 +81,11 @@ func (c *Chain) TTS(ctx context.Context, text string) (Audio, error) {
|
||||
}
|
||||
audio, err := tts.TTS(ctx, text)
|
||||
if err == nil {
|
||||
return audio, nil
|
||||
return audio, p.Name(), nil
|
||||
}
|
||||
lastErr = err
|
||||
}
|
||||
return Audio{}, fmt.Errorf("voice: %w: %v", ErrProviderFailed, lastErr)
|
||||
return Audio{}, "", fmt.Errorf("voice: %w: %v", ErrProviderFailed, lastErr)
|
||||
}
|
||||
|
||||
// Option customizes a chain built by New.
|
||||
|
||||
@@ -0,0 +1,50 @@
|
||||
package voice
|
||||
|
||||
import (
|
||||
"context"
|
||||
"testing"
|
||||
)
|
||||
|
||||
func TestChain_STTWithProvider(t *testing.T) {
|
||||
t.Parallel()
|
||||
steps := []Provider{
|
||||
&stubProvider{name: "cloud", stt: &stubSTT{err: ErrNotConfigured}},
|
||||
&stubProvider{name: "local", stt: &stubSTT{text: "hi"}},
|
||||
}
|
||||
text, rung, err := NewChain(steps...).STTWithProvider(context.Background(), Audio{Format: "wav", Data: []byte("x")})
|
||||
if err != nil {
|
||||
t.Fatalf("STTWithProvider error: %v", err)
|
||||
}
|
||||
if text != "hi" || rung != "local" {
|
||||
t.Errorf("text/rung = %q/%q, want hi/local", text, rung)
|
||||
}
|
||||
}
|
||||
|
||||
func TestChain_TTSWithProvider_Silent(t *testing.T) {
|
||||
t.Parallel()
|
||||
steps := []Provider{
|
||||
&stubProvider{name: "cloud", tts: &stubTTS{err: ErrNotConfigured}},
|
||||
SilentProvider(),
|
||||
}
|
||||
audio, rung, err := NewChain(steps...).TTSWithProvider(context.Background(), "hi")
|
||||
if err != nil {
|
||||
t.Fatalf("TTSWithProvider error: %v", err)
|
||||
}
|
||||
if len(audio.Data) != 0 || rung != "silent" {
|
||||
t.Errorf("audio/rung = %+v/%q, want empty/silent", audio, rung)
|
||||
}
|
||||
}
|
||||
|
||||
func TestService_TTSWithProvider(t *testing.T) {
|
||||
t.Parallel()
|
||||
svc := NewService(NewChain(NewProvider("edge-tts", nil, &stubTTS{audio: Audio{Format: "mp3", Data: []byte("a")}})))
|
||||
defer svc.Close()
|
||||
|
||||
audio, rung, err := svc.TTSWithProvider(context.Background(), "hi")
|
||||
if err != nil {
|
||||
t.Fatalf("TTSWithProvider error: %v", err)
|
||||
}
|
||||
if string(audio.Data) != "a" || rung != "edge-tts" {
|
||||
t.Errorf("audio/rung = %+v/%q, want a/edge-tts", audio, rung)
|
||||
}
|
||||
}
|
||||
@@ -53,9 +53,10 @@ type job struct {
|
||||
|
||||
// jobResult carries the outcome written back to the caller.
|
||||
type jobResult struct {
|
||||
text string
|
||||
audio Audio
|
||||
err error
|
||||
text string
|
||||
audio Audio
|
||||
provider string
|
||||
err error
|
||||
}
|
||||
|
||||
// NewService builds a Service backed by chain and starts the worker pool.
|
||||
@@ -99,9 +100,9 @@ func (s *Service) run(j job) {
|
||||
res := jobResult{}
|
||||
switch j.kind {
|
||||
case jobSTT:
|
||||
res.text, res.err = s.chain.STT(s.bgCtx, j.audio)
|
||||
res.text, res.provider, res.err = s.chain.STTWithProvider(s.bgCtx, j.audio)
|
||||
case jobTTS:
|
||||
res.audio, res.err = s.chain.TTS(s.bgCtx, j.text)
|
||||
res.audio, res.provider, res.err = s.chain.TTSWithProvider(s.bgCtx, j.text)
|
||||
}
|
||||
j.res <- res
|
||||
}
|
||||
@@ -109,21 +110,35 @@ func (s *Service) run(j job) {
|
||||
// STT enqueues a transcription request and waits for the result. ctx cancels
|
||||
// the wait and the underlying provider call.
|
||||
func (s *Service) STT(ctx context.Context, audio Audio) (string, error) {
|
||||
text, _, err := s.STTWithProvider(ctx, audio)
|
||||
return text, err
|
||||
}
|
||||
|
||||
// STTWithProvider is STT but also reports the provider rung that produced the
|
||||
// transcription.
|
||||
func (s *Service) STTWithProvider(ctx context.Context, audio Audio) (string, string, error) {
|
||||
res, err := s.submit(ctx, job{kind: jobSTT, audio: audio, res: make(chan jobResult, 1)})
|
||||
if err != nil {
|
||||
return "", err
|
||||
return "", "", err
|
||||
}
|
||||
return res.text, res.err
|
||||
return res.text, res.provider, res.err
|
||||
}
|
||||
|
||||
// TTS enqueues a synthesis request and waits for the result. ctx cancels the
|
||||
// wait and the underlying provider call.
|
||||
func (s *Service) TTS(ctx context.Context, text string) (Audio, error) {
|
||||
audio, _, err := s.TTSWithProvider(ctx, text)
|
||||
return audio, err
|
||||
}
|
||||
|
||||
// TTSWithProvider is TTS but also reports the provider rung that produced the
|
||||
// audio.
|
||||
func (s *Service) TTSWithProvider(ctx context.Context, text string) (Audio, string, error) {
|
||||
res, err := s.submit(ctx, job{kind: jobTTS, text: text, res: make(chan jobResult, 1)})
|
||||
if err != nil {
|
||||
return Audio{}, err
|
||||
return Audio{}, "", err
|
||||
}
|
||||
return res.audio, res.err
|
||||
return res.audio, res.provider, res.err
|
||||
}
|
||||
|
||||
// CloneVoice creates a custom voice, delegating to the underlying chain. It is a
|
||||
|
||||
@@ -61,7 +61,7 @@ func (s *Server) handleTranscribe(w http.ResponseWriter, r *http.Request) {
|
||||
writeError(w, apperr.Wrap(apperr.ErrBadRequest, "bad_request", "audio is required"))
|
||||
return
|
||||
}
|
||||
text, err := s.voice.STT(r.Context(), audio)
|
||||
text, rung, err := sttWithProvider(r.Context(), s.voice, audio)
|
||||
if err != nil {
|
||||
s.auditClaims(r, "audio.transcribe", "error")
|
||||
s.log.Warn("voice transcribe", "err", err)
|
||||
@@ -69,7 +69,11 @@ func (s *Server) handleTranscribe(w http.ResponseWriter, r *http.Request) {
|
||||
return
|
||||
}
|
||||
s.auditClaims(r, "audio.transcribe", "ok")
|
||||
writeJSON(w, http.StatusOK, map[string]any{"text": text})
|
||||
out := map[string]any{"text": text}
|
||||
if sig := providerSignal(rung, strings.TrimSpace(text) == ""); sig != "" {
|
||||
out["provider"] = sig
|
||||
}
|
||||
writeJSON(w, http.StatusOK, out)
|
||||
}
|
||||
|
||||
// handleSpeech synthesizes speech for the given text and writes the raw audio
|
||||
@@ -96,7 +100,7 @@ func (s *Server) handleSpeech(w http.ResponseWriter, r *http.Request) {
|
||||
if req.Instruction != "" {
|
||||
ctx = voice.WithTTSInstruction(ctx, req.Instruction)
|
||||
}
|
||||
audi, err := s.voice.TTS(ctx, req.Text)
|
||||
audi, rung, err := ttsWithProvider(ctx, s.voice, req.Text)
|
||||
if err != nil {
|
||||
s.auditClaims(r, "audio.speech", "error")
|
||||
s.log.Warn("voice speech", "err", err)
|
||||
@@ -105,6 +109,7 @@ func (s *Server) handleSpeech(w http.ResponseWriter, r *http.Request) {
|
||||
}
|
||||
if len(audi.Data) == 0 {
|
||||
s.auditClaims(r, "audio.speech", "error")
|
||||
w.Header().Set(voiceProviderHeader, providerSignal(rung, true))
|
||||
writeError(w, apperr.Wrap(apperr.ErrInternal, "tts_empty", "no audio synthesized"))
|
||||
return
|
||||
}
|
||||
|
||||
@@ -25,6 +25,7 @@ func (s *Server) cors(next http.Handler) http.Handler {
|
||||
w.Header().Set("Vary", "Origin")
|
||||
w.Header().Set("Access-Control-Allow-Methods", "GET, POST, PUT, DELETE, OPTIONS")
|
||||
w.Header().Set("Access-Control-Allow-Headers", "Authorization, Content-Type")
|
||||
w.Header().Set("Access-Control-Expose-Headers", voiceProviderHeader)
|
||||
w.Header().Set("Access-Control-Max-Age", "600")
|
||||
}
|
||||
if r.Method == http.MethodOptions {
|
||||
|
||||
@@ -28,6 +28,54 @@ type VoiceService interface {
|
||||
CloneVoice(ctx context.Context, referenceURL, prefix string) (string, error)
|
||||
}
|
||||
|
||||
// voiceProviderHeader is the response header carrying the provider rung on the
|
||||
// binary /api/v1/audio/speech response, where a JSON signal field cannot be added.
|
||||
const voiceProviderHeader = "X-Voice-Provider"
|
||||
|
||||
// voiceProviderReporter is the optional extension of VoiceService: a service that
|
||||
// can report which provider rung (cloud / local / edge-tts / espeak / silent)
|
||||
// produced a result. *voice.Chain and *voice.Service implement it; a custom
|
||||
// VoiceService that does not is still supported and falls back to the plain
|
||||
// STT/TTS methods with no reported rung.
|
||||
type voiceProviderReporter interface {
|
||||
STTWithProvider(ctx context.Context, audio voice.Audio) (string, string, error)
|
||||
TTSWithProvider(ctx context.Context, text string) (voice.Audio, string, error)
|
||||
}
|
||||
|
||||
// sttWithProvider runs STT and returns the transcript plus the provider rung that
|
||||
// served it ("" when the injected service cannot report one).
|
||||
func sttWithProvider(ctx context.Context, v VoiceService, audio voice.Audio) (string, string, error) {
|
||||
if r, ok := v.(voiceProviderReporter); ok {
|
||||
return r.STTWithProvider(ctx, audio)
|
||||
}
|
||||
text, err := v.STT(ctx, audio)
|
||||
return text, "", err
|
||||
}
|
||||
|
||||
// ttsWithProvider runs TTS and returns the audio plus the provider rung that
|
||||
// served it.
|
||||
func ttsWithProvider(ctx context.Context, v VoiceService, text string) (voice.Audio, string, error) {
|
||||
if r, ok := v.(voiceProviderReporter); ok {
|
||||
return r.TTSWithProvider(ctx, text)
|
||||
}
|
||||
audio, err := v.TTS(ctx, text)
|
||||
return audio, "", err
|
||||
}
|
||||
|
||||
// providerSignal returns the "provider" degraded signal for a voice result: the
|
||||
// actual provider rung when the result is empty, "silent" when the rung is
|
||||
// unknown, and "" (omit the field) when the result carries real text/audio. It
|
||||
// only flags "no real speech/audio"; it is never a business error.
|
||||
func providerSignal(rung string, empty bool) string {
|
||||
if !empty {
|
||||
return ""
|
||||
}
|
||||
if rung == "" {
|
||||
return "silent"
|
||||
}
|
||||
return rung
|
||||
}
|
||||
|
||||
// voiceInputRequest is the decoded payload of a voice_input frame. Audio is a
|
||||
// base64 (or data: URI) audio blob; Format is its encoding ("wav"/"mp3"/"ogg");
|
||||
// Text overrides STT when present (a client that already transcribed can skip
|
||||
@@ -119,18 +167,22 @@ func (h *hub) runVoiceTurn(c *wsConn, req voiceInputRequest) {
|
||||
if strings.TrimSpace(reply) == "" {
|
||||
reply = res.StopReason
|
||||
}
|
||||
audi, err := h.voice.TTS(ctx, reply)
|
||||
audi, rung, err := ttsWithProvider(ctx, h.voice, reply)
|
||||
if err != nil {
|
||||
h.log.Warn("voice tts", "err", err)
|
||||
h.voiceError(c, "voice synthesis failed")
|
||||
return
|
||||
}
|
||||
c.enqueueJSON(wsEvent{Type: wsVoiceOutput, Data: map[string]any{
|
||||
data := map[string]any{
|
||||
"audio": base64.StdEncoding.EncodeToString(audi.Data),
|
||||
"format": audi.Format,
|
||||
"text": transcript,
|
||||
"reply": reply,
|
||||
}})
|
||||
}
|
||||
if sig := providerSignal(rung, len(audi.Data) == 0); sig != "" {
|
||||
data["provider"] = sig
|
||||
}
|
||||
c.enqueueJSON(wsEvent{Type: wsVoiceOutput, Data: data})
|
||||
}
|
||||
|
||||
// voiceError replies with a typed error frame for a failed voice turn.
|
||||
|
||||
@@ -0,0 +1,181 @@
|
||||
package gateway
|
||||
|
||||
import (
|
||||
"context"
|
||||
"encoding/base64"
|
||||
"encoding/json"
|
||||
"net/http"
|
||||
"testing"
|
||||
|
||||
"git.yeij.top/AskaEth/Cyrene_v2/internal/auth"
|
||||
"git.yeij.top/AskaEth/Cyrene_v2/internal/runtime/loop"
|
||||
"git.yeij.top/AskaEth/Cyrene_v2/internal/runtime/multimodal/voice"
|
||||
)
|
||||
|
||||
// signalVoice is a VoiceService that also implements voiceProviderReporter, so a
|
||||
// test can control both the STT/TTS result and the reported provider rung.
|
||||
type signalVoice struct {
|
||||
text string
|
||||
audio voice.Audio
|
||||
rung string
|
||||
sttErr error
|
||||
ttsErr error
|
||||
}
|
||||
|
||||
func (s *signalVoice) STT(_ context.Context, _ voice.Audio) (string, error) {
|
||||
return s.text, s.sttErr
|
||||
}
|
||||
|
||||
func (s *signalVoice) TTS(_ context.Context, _ string) (voice.Audio, error) {
|
||||
return s.audio, s.ttsErr
|
||||
}
|
||||
|
||||
func (s *signalVoice) CloneVoice(_ context.Context, _, _ string) (string, error) { return "", nil }
|
||||
|
||||
func (s *signalVoice) STTWithProvider(_ context.Context, _ voice.Audio) (string, string, error) {
|
||||
return s.text, s.rung, s.sttErr
|
||||
}
|
||||
|
||||
func (s *signalVoice) TTSWithProvider(_ context.Context, _ string) (voice.Audio, string, error) {
|
||||
return s.audio, s.rung, s.ttsErr
|
||||
}
|
||||
|
||||
func TestHandleTranscribe_SilentSignal(t *testing.T) {
|
||||
env := newEnv(t, nil, false)
|
||||
env.srv.SetVoice(&signalVoice{rung: "silent"})
|
||||
|
||||
token := env.token(t, auth.RoleOwner)
|
||||
resp, body := doReq(t, http.MethodPost, env.ts.URL+"/api/v1/audio/transcribe", map[string]any{
|
||||
"audio": base64.StdEncoding.EncodeToString([]byte("raw")),
|
||||
}, token)
|
||||
if resp.StatusCode != http.StatusOK {
|
||||
t.Fatalf("status = %d, body = %s", resp.StatusCode, body)
|
||||
}
|
||||
var data struct {
|
||||
Text string `json:"text"`
|
||||
Provider string `json:"provider"`
|
||||
}
|
||||
if err := json.Unmarshal(decodeBody(t, body)["data"], &data); err != nil {
|
||||
t.Fatalf("decode data: %v (body=%s)", err, body)
|
||||
}
|
||||
if data.Text != "" || data.Provider != "silent" {
|
||||
t.Errorf("data = %+v, want empty text + provider silent", data)
|
||||
}
|
||||
}
|
||||
|
||||
func TestHandleTranscribe_SilentFallbackWithoutReporter(t *testing.T) {
|
||||
env := newEnv(t, nil, false)
|
||||
// audioStubVoice does not implement voiceProviderReporter, so the handler
|
||||
// must still flag an empty transcription as silent.
|
||||
env.srv.SetVoice(&audioStubVoice{})
|
||||
|
||||
token := env.token(t, auth.RoleOwner)
|
||||
resp, body := doReq(t, http.MethodPost, env.ts.URL+"/api/v1/audio/transcribe", map[string]any{
|
||||
"audio": base64.StdEncoding.EncodeToString([]byte("raw")),
|
||||
}, token)
|
||||
if resp.StatusCode != http.StatusOK {
|
||||
t.Fatalf("status = %d, body = %s", resp.StatusCode, body)
|
||||
}
|
||||
var data struct {
|
||||
Provider string `json:"provider"`
|
||||
}
|
||||
if err := json.Unmarshal(decodeBody(t, body)["data"], &data); err != nil {
|
||||
t.Fatalf("decode data: %v (body=%s)", err, body)
|
||||
}
|
||||
if data.Provider != "silent" {
|
||||
t.Errorf("provider = %q, want silent (body=%s)", data.Provider, body)
|
||||
}
|
||||
}
|
||||
|
||||
func TestHandleTranscribe_SuccessOmitsProvider(t *testing.T) {
|
||||
env := newEnv(t, nil, false)
|
||||
env.srv.SetVoice(&signalVoice{text: "你好", rung: "cloud"})
|
||||
|
||||
token := env.token(t, auth.RoleOwner)
|
||||
resp, body := doReq(t, http.MethodPost, env.ts.URL+"/api/v1/audio/transcribe", map[string]any{
|
||||
"audio": base64.StdEncoding.EncodeToString([]byte("raw")),
|
||||
}, token)
|
||||
if resp.StatusCode != http.StatusOK {
|
||||
t.Fatalf("status = %d, body = %s", resp.StatusCode, body)
|
||||
}
|
||||
var raw map[string]json.RawMessage
|
||||
if err := json.Unmarshal(decodeBody(t, body)["data"], &raw); err != nil {
|
||||
t.Fatalf("decode data: %v (body=%s)", err, body)
|
||||
}
|
||||
if _, present := raw["provider"]; present {
|
||||
t.Errorf("provider must be omitted on a non-empty transcription: %s", body)
|
||||
}
|
||||
}
|
||||
|
||||
func TestHandleSpeech_EmptyAudioSignal(t *testing.T) {
|
||||
env := newEnv(t, nil, false)
|
||||
env.srv.SetVoice(&signalVoice{rung: "silent"})
|
||||
|
||||
token := env.token(t, auth.RoleOwner)
|
||||
resp, body := doReq(t, http.MethodPost, env.ts.URL+"/api/v1/audio/speech", map[string]any{
|
||||
"text": "你好",
|
||||
}, token)
|
||||
if resp.StatusCode != http.StatusInternalServerError {
|
||||
t.Fatalf("status = %d, want 500 (body=%s)", resp.StatusCode, body)
|
||||
}
|
||||
if got := resp.Header.Get(voiceProviderHeader); got != "silent" {
|
||||
t.Errorf("%s = %q, want silent", voiceProviderHeader, got)
|
||||
}
|
||||
var env2 struct {
|
||||
Error struct {
|
||||
Code string `json:"code"`
|
||||
} `json:"error"`
|
||||
}
|
||||
if err := json.Unmarshal(body, &env2); err != nil {
|
||||
t.Fatalf("decode error: %v (body=%s)", err, body)
|
||||
}
|
||||
if env2.Error.Code != "tts_empty" {
|
||||
t.Errorf("error code = %q, want tts_empty (body=%s)", env2.Error.Code, body)
|
||||
}
|
||||
}
|
||||
|
||||
func TestHubVoiceOutput_SilentSignal(t *testing.T) {
|
||||
t.Parallel()
|
||||
hb := newTestHub(&stubRunner{res: &loop.Result{Text: "你好", Steps: 1, StopReason: "stop"}}, nil)
|
||||
hb.voice = &signalVoice{text: "你好", rung: "silent"}
|
||||
c := newFakeConn(hb, 32)
|
||||
hb.register(c)
|
||||
|
||||
c.route(wsInbound{Type: "voice_input", Data: json.RawMessage(`{"audio":"aGVsbG8=","format":"wav"}`)})
|
||||
ev := readFakeFrame(t, c)
|
||||
if ev.Type != wsVoiceOutput {
|
||||
t.Fatalf("frame type = %q, want voice_output (frame=%+v)", ev.Type, ev)
|
||||
}
|
||||
r, ok := ev.Data.(map[string]any)
|
||||
if !ok {
|
||||
t.Fatalf("data = %T, want map[string]any", ev.Data)
|
||||
}
|
||||
if r["provider"] != "silent" {
|
||||
t.Errorf("provider = %v, want silent (frame=%+v)", r["provider"], ev.Data)
|
||||
}
|
||||
}
|
||||
|
||||
func TestHubVoiceOutput_SuccessOmitsProvider(t *testing.T) {
|
||||
t.Parallel()
|
||||
hb := newTestHub(&stubRunner{res: &loop.Result{Text: "你好", Steps: 1, StopReason: "stop"}}, nil)
|
||||
hb.voice = &signalVoice{
|
||||
text: "你好",
|
||||
audio: voice.Audio{Format: "wav", Data: []byte("audio-bytes")},
|
||||
rung: "edge-tts",
|
||||
}
|
||||
c := newFakeConn(hb, 32)
|
||||
hb.register(c)
|
||||
|
||||
c.route(wsInbound{Type: "voice_input", Data: json.RawMessage(`{"audio":"aGVsbG8=","format":"wav"}`)})
|
||||
ev := readFakeFrame(t, c)
|
||||
if ev.Type != wsVoiceOutput {
|
||||
t.Fatalf("frame type = %q, want voice_output (frame=%+v)", ev.Type, ev)
|
||||
}
|
||||
r, ok := ev.Data.(map[string]any)
|
||||
if !ok {
|
||||
t.Fatalf("data = %T, want map[string]any", ev.Data)
|
||||
}
|
||||
if _, present := r["provider"]; present {
|
||||
t.Errorf("provider must be omitted for non-empty audio: %+v", ev.Data)
|
||||
}
|
||||
}
|
||||
Reference in new issue
Block a user