feat(model): LLM token 预算可配置化 — models.json每模型maxTokens + LLM_FALLBACK_MAX_TOKENS回退兜底
- config.go: LLMConfig.FallbackMaxTokens(LLM_FALLBACK_MAX_TOKENS, 默认8192) - runtimeconfig/models.go: Group.MaxTokens(models.json每模型独立预算, 负值拒绝) - model/fallback.go: Chain.WithTokenBudget + applyBudget; runtime.go buildSnapshot接入(active.MaxTokens, env.FallbackMaxTokens) - 优先级: 显式Request.MaxTokens > 组maxTokens > 仅多步链末端的LLM_FALLBACK_MAX_TOKENS > 适配器默认8192; 单步链不动env兜底(避免OpenAI主模型行为回归) - gateway admin: AdminLlmProvider.MaxTokens 透传/回显/负值400 - .env.example + .env(LLM_FALLBACK_MAX_TOKENS=8192); Docs/api-client v0.27 + api-reference + design/04 - 新增测试(config/runtimeconfig/budget_test/admin_write), build/vet/test绿+3平台交叉编译
This commit is contained in:
1 parent
f347681034
commit
53002f1e11
13 files changed
+384
-15
No files matched your search
@@ -46,6 +46,10 @@ LLM_BASE_URL=https://api.openai.com/v1
|
||||
LLM_API_KEY=sk-xxxxx
|
||||
LLM_MODEL=gpt-4o
|
||||
LLM_FALLBACK_MODEL=gpt-4o-mini
|
||||
# 最后一道回退模型(fallback 链末端)的 token 预算上限。仅当链确有回退模型、
|
||||
# 且该模型组未在 models.json 设 maxTokens、请求也未显式给 MaxTokens 时生效;
|
||||
# 0 = 关闭该兜底(改用适配器默认,如 anthropic 8192)。每模型组预算见 models.json 的 maxTokens。
|
||||
LLM_FALLBACK_MAX_TOKENS=8192
|
||||
|
||||
# ---- LLM 视觉模型(多模态路由:含图片的请求路由到此模型) ----
|
||||
# 不配置则图片路由回主模型;仅当启用多模态能力时设置。
|
||||
|
||||
+3
-2
@@ -57,7 +57,7 @@
|
||||
| 6 | `/api/v1/admin/channels` | GET | ✅ | 渠道列表/健康 + 每平台已配账号(脱敏 token 掩码,design/04 §3.1);已连接渠道的账号附 **bot 身份**(`botId`/`botUsername`/`botNickname`,如 KOOK `GET /api/v3/user/me` 自动获取) |
|
||||
| 7 | `/api/v1/admin/iot` | GET | ✅ | IoT 设备列表 |
|
||||
| 8 | `/api/v1/admin/users` | GET/POST | ✅(GET) | 用户/角色/API key(GET 已实现;POST 创建用户/签发 key 随 M10 落地) |
|
||||
| 9 | `/api/v1/admin/llm` | GET/POST | ✅ | LLM 多厂商多模型(models.json):GET 列表(APIKey 脱敏,含 `provider` 协议);POST 保存组→落盘 `models.json`→**热更新**运行中的 LLM stack(design/04 §3.1,无需重启) |
|
||||
| 9 | `/api/v1/admin/llm` | GET/POST | ✅ | LLM 多厂商多模型(models.json):GET 列表(APIKey 脱敏,含 `provider` 协议与 `maxTokens` 预算);POST 保存组→落盘 `models.json`→**热更新**运行中的 LLM stack(design/04 §3.1,无需重启) |
|
||||
| 10 | `/api/v1/admin/automations` | GET/POST | ✅(GET) | 自动化与提醒(GET 已实现;POST 创建 M10) |
|
||||
| 11 | `/api/v1/admin/backthink` | GET | ✅ | 后台思考记录 |
|
||||
| 12 | `/api/v1/admin/rag` | GET | ✅ | 知识库文档列表(分块/嵌入状态);带 `?query=` 时并入检索结果 |
|
||||
@@ -86,7 +86,7 @@
|
||||
- `POST /admin/automations` body `{name,kind:"cron|event",expression}` → `201 {data:{id,name,kind}}`(真实注册规则);无效 cron/规则回 `400 invalid_cron|invalid_rule`。
|
||||
- `POST /admin/automations/{id}/run` → `200 {data:{id,output}}`(手动触发);`404`。
|
||||
- `DELETE /admin/automations/{id}` → `200 {data:{id,deleted:true}}`;`404`。
|
||||
- `POST /admin/llm` body `{id?,provider?,baseUrl,apiKey?,model,multimodal?,fallback?}` → `201 {data:{provider:{id,name,provider,baseUrl,model,multimodal,enabled,fallback?}, persisted:true, path}}`。**真实持久化**:读-改-写 `models.json`(本地、gitignore),保存组提升为**生效组**(index 0),并**热更新**运行中的 LLM provider(`path` 为落盘文件路径);API key 写入文件但**永不回显/不落日志**。`provider` 为空默认 `openai`;缺 `baseUrl`/`model` 回 `400`;未注入 LLM stack 回 `500 llm_unavailable`、写盘失败 `500 llm_persist_failed`、热重载失败 `500 llm_reload_failed`。`.env` 为兜底。
|
||||
- `POST /admin/llm` body `{id?,provider?,baseUrl,apiKey?,model,multimodal?,fallback?,maxTokens?}` → `201 {data:{provider:{id,name,provider,baseUrl,model,multimodal,enabled,fallback?,maxTokens?}, persisted:true, path}}`。**真实持久化**:读-改-写 `models.json`(本地、gitignore),保存组提升为**生效组**(index 0),并**热更新**运行中的 LLM provider(`path` 为落盘文件路径);API key 写入文件但**永不回显/不落日志**。`provider` 为空默认 `openai`;缺 `baseUrl`/`model` 回 `400`;`maxTokens` 为负回 `400`;未注入 LLM stack 回 `500 llm_unavailable`、写盘失败 `500 llm_persist_failed`、热重载失败 `500 llm_reload_failed`。`maxTokens` 缺省/`0` = 未设(回落 `LLM_FALLBACK_MAX_TOKENS` / 适配器默认)。`.env` 为兜底。
|
||||
- `POST /admin/llm/test` body `{provider?,baseUrl,model,apiKey?,prompt?}` → `200 {data:{ok,provider,model,reply,tokens}}`(**真实模型调用**);无效/失败回 `400 invalid_provider` / `500 llm_test_failed`。API key 仅本次请求内使用,不落库、不进 wire。
|
||||
- `POST /admin/channels/open`、`POST /admin/channels/close` body `{name}` → `200 {data:{name,connected}}`(真实 Connect/Disconnect);`404`(未注册渠道)/`500`(凭据缺失等连接失败)。
|
||||
- `POST /admin/channels/test` body `{platform?,accountId,isGroup}` → `200 {data:{role,owner,isGroup,env,verdict}}`。**结构性判定**:channel 层 actor 元数据 × `configs/users/owner.yaml` 白名单,绝不靠提示词。
|
||||
@@ -215,6 +215,7 @@
|
||||
- 语音:上行 `voice_input`(base64 音频或已转写 text),下行 `voice_output`(合成音频 base64 + text)。
|
||||
|
||||
## 5. 变更记录
|
||||
- `v0.27`(**LLM token 预算可配置化**):`models.json` 模型组新增可选 **`maxTokens`**(`int`,缺省/`0` = 未设),作为该组主/备链的 token 预算;`GET /api/v1/admin/llm` 的 `AdminLlmProvider` 与 `POST /api/v1/admin/llm` body/回显同步该字段(`omitempty`,未设为 0 时不出现)。`.env` 新增 **`LLM_FALLBACK_MAX_TOKENS`**(`int`,默认 `8192`,`0` = 关闭兜底),只作用于**确有回退模型**时链末端的回退步。**优先级**:显式 `Request.MaxTokens`(审查/判定类小预算)> 生效组 `maxTokens` > `LLM_FALLBACK_MAX_TOKENS`(仅链末端、且组未设时)> 适配器默认(anthropic 8192)。单步链(无 fallback)不套用 env 兜底。`maxTokens` 为负回 `400 bad_request`。**契约版本 `v0.27`**。
|
||||
- `v0.26`(**A5 消息正向增量**):`GET /api/v1/sessions/{id}/messages` 新增 **`after`** 查询参数(`<createdAt 毫秒>`,升序返回 `created_at > after` 的消息,默认 `limit=50`/上限 200)——解决「WS 重连只回放主会话最近 50 条、断线超 50 条永久丢」的缺口;`before` 现行为不变;**`after` 与 `before` 互斥**(同传 `400`);负数/非数字 `400`。`HistoryStore` 新增 `ListAfter`(走 `idx_chat_msgs_session_created`,`rowid` 兜底同毫秒并列)。**游标契约**:客户端用「本页最后一条 `createdAt`」作下一页 `after`;同毫秒消息 > limit 时纯 `createdAt` 游标会跨页重读(超罕见,客户端按 `id` 去重兜底,与现有绕行一致)。
|
||||
- `v0.25`(**B2 refresh token**):`POST /api/v1/auth/refresh`(**新增**,`{refreshToken}` → 新 access + 新 refresh,默认单次使用旋转,并发重放 fail-closed `invalid_grant`)+ `POST /api/v1/auth/logout`(**新增**,撤单个或全设备)。`POST /api/v1/auth/login` 在 `{token,expiresAt,role}` 之外**新增 `refreshToken`**(不透明 256-bit 随机串,**仅 SHA-256 哈希落库**,`AUTH_REFRESH_TTL` 默认 `720h`,`AUTH_REFRESH_ROTATE_DISABLED` 关旋转)。新表 `refresh_tokens`(迁移 0008,id/user_id/role/token_hash UNIQUE/expires_at/revoked_at;含过期/轮换原子事务);`rotate-jwt` 成功即删除全部 refresh(返回 `refreshTokensRevoked`),`revoke-keys` 不动 refresh。**密码/secret 记录见 `internal/auth`;refresh 全文绝不落日志/审计。**
|
||||
- `v0.24`(**B1 图片上传端点**):`POST /api/v1/upload`(**新增**,multipart `file`,owner/admin+审计 → `201 {data:{url,size,mime,name}}`)+ `GET /api/v1/files/{name}`(**新增**,同端口静态服务)。安全:`MaxBytesReader`(5MB)、`DetectContentType` 嗅探(不信客户端 Content-Type)、白名单 raster image(拒 SVG/HTML)、`crypto/rand` 128-bit 随机名(丢弃原始文件名)、非可执行上传目录(0700)、`nosniff`;路径穿越全形态拒绝(含编码 `%2f`/`%2e%2e`)。错误码 `400/413/415`(`internal/errors` 增 `ErrPayloadTooLarge`/`ErrUnsupportedMediaType`)。**设计取舍**:GET 不鉴权(能力 URL,`chat.image`/`<img>` 无法带 Bearer),POST 仍 owner/admin。`url` 为绝对地址(`X-Forwarded-Proto`/TLS + `r.Host`,因 `ParseImageRef` 只认 `http(s)://`)。`Deps.DataDir` 由 wire 注入(`resolveDataDir`)。
|
||||
|
||||
@@ -445,9 +445,10 @@ curl -s http://<host>:8080/api/v1/admin/iot -H "Authorization: Bearer $TOKEN" -H
|
||||
|
||||
### 4.9 GET /api/v1/admin/llm
|
||||
- **响应**:`200 {"data":{"providers":[<AdminLlmProvider>...]}}`(`AdminLlmConfig`)。
|
||||
- `AdminLlmProvider`:`{id, name, provider, baseUrl, model, multimodal, enabled, fallback?}`。
|
||||
- `AdminLlmProvider`:`{id, name, provider, baseUrl, model, multimodal, enabled, fallback?, maxTokens?}`。
|
||||
- `provider` 协议 ∈ `openai|anthropic`;`name` 为友好标签(OpenAI/Anthropic 或组 id)。
|
||||
- `enabled`:index 0 的组为生效组 = `true`。
|
||||
- `maxTokens`:该组的 token 预算(`omitempty`,缺省/0 = 未设);缺省时链末端回退步回落 `LLM_FALLBACK_MAX_TOKENS`(`.env`,默认 8192),再回落适配器默认(anthropic 8192)。
|
||||
- **API key 永不返回**(wire 无 key 字段)。
|
||||
- 数据源:`models.json` 各组(index 0 = 生效组);无 `models.json` 时按 `.env` 合成单组。curl:
|
||||
|
||||
@@ -676,7 +677,7 @@ curl -s -X DELETE http://<host>:8080/api/v1/admin/automations/<id> -H "Authoriza
|
||||
```
|
||||
|
||||
### 5.13 POST /api/v1/admin/llm
|
||||
- **Body**(`llmProviderRequest`):`{"id?","name?","provider?","baseUrl","apiKey?","model","multimodal?","fallback?"}`。`baseUrl`+`model` 必填;`provider` 空默认 `openai`(仅允许 `openai|anthropic`)。
|
||||
- **Body**(`llmProviderRequest`):`{"id?","name?","provider?","baseUrl","apiKey?","model","multimodal?","fallback?","maxTokens?"}`。`baseUrl`+`model` 必填;`provider` 空默认 `openai`(仅允许 `openai|anthropic`);`maxTokens` 缺省/`0` = 未设,负数回 `400`。
|
||||
- **响应**:`201 {"data":{"provider":<AdminLlmProvider>,"persisted":true,"path":"<models.json 绝对路径>"}}`。**真实持久化**:读-改-写 `models.json`,保存组**提升为生效组**(index 0),并**热重载**运行中的 LLM stack(无需重启)。
|
||||
- **错误**:`400`(缺 baseUrl/model / invalid provider);`500`(LLM 服务未注入 / 写盘失败 / 热重载失败)。**API key 写入文件但永不回显、不落日志**。
|
||||
- **审计**:`llm.addProvider`。curl:
|
||||
|
||||
@@ -90,6 +90,13 @@ type LLMConfig struct {
|
||||
// set, the model router routes image content to it (see model.Router.WithVisionModel).
|
||||
VisionModel string `env:"LLM_VISION_MODEL"`
|
||||
|
||||
// FallbackMaxTokens caps the last-resort fallback model in a group's chain
|
||||
// (primary → fallback). It is the final safety budget: it only applies when
|
||||
// the chain actually has a fallback step, that group sets no per-group
|
||||
// MaxTokens (models.json maxTokens) and the request carries no explicit
|
||||
// budget. Zero disables the cap (the adapter default then applies).
|
||||
FallbackMaxTokens int `env:"LLM_FALLBACK_MAX_TOKENS" default:"8192"`
|
||||
|
||||
// Anthropic native provider, used when Provider=anthropic. Independent
|
||||
// config block so each vendor is a self-contained config.
|
||||
AnthropicBaseURL string `env:"ANTHROPIC_BASE_URL"`
|
||||
|
||||
@@ -91,6 +91,55 @@ IOT_PROVIDER=none
|
||||
}
|
||||
}
|
||||
|
||||
// unsetEnvForTest clears key for the test and restores the original value (or
|
||||
// absence) on cleanup, so a .env file is not shadowed by an ambient process env
|
||||
// variable. config tests are not parallel, so this is safe.
|
||||
func unsetEnvForTest(t *testing.T, key string) {
|
||||
t.Helper()
|
||||
if v, ok := os.LookupEnv(key); ok {
|
||||
t.Cleanup(func() { _ = os.Setenv(key, v) })
|
||||
} else {
|
||||
t.Cleanup(func() { _ = os.Unsetenv(key) })
|
||||
}
|
||||
if err := os.Unsetenv(key); err != nil {
|
||||
t.Fatalf("unset %s: %v", key, err)
|
||||
}
|
||||
}
|
||||
|
||||
func TestLoad_LlmFallbackMaxTokens(t *testing.T) {
|
||||
t.Setenv("JWT_SECRET", strings.Repeat("s", 40))
|
||||
unsetEnvForTest(t, "LLM_FALLBACK_MAX_TOKENS")
|
||||
|
||||
// Default: the last-resort fallback budget is a meaningful 8192.
|
||||
cfg, err := Load("")
|
||||
if err != nil {
|
||||
t.Fatalf("Load() error: %v", err)
|
||||
}
|
||||
if cfg.LLM.FallbackMaxTokens != 8192 {
|
||||
t.Errorf("LLM.FallbackMaxTokens = %d, want default 8192", cfg.LLM.FallbackMaxTokens)
|
||||
}
|
||||
|
||||
// A .env file is parsed into the field.
|
||||
path := writeEnvFile(t, "LLM_FALLBACK_MAX_TOKENS=4096\n")
|
||||
cfg, err = Load(path)
|
||||
if err != nil {
|
||||
t.Fatalf("Load(%q) error: %v", path, err)
|
||||
}
|
||||
if cfg.LLM.FallbackMaxTokens != 4096 {
|
||||
t.Errorf("LLM.FallbackMaxTokens = %d, want 4096 from .env", cfg.LLM.FallbackMaxTokens)
|
||||
}
|
||||
|
||||
// Process env wins over .env.
|
||||
t.Setenv("LLM_FALLBACK_MAX_TOKENS", "2048")
|
||||
cfg, err = Load(path)
|
||||
if err != nil {
|
||||
t.Fatalf("Load(%q) error: %v", path, err)
|
||||
}
|
||||
if cfg.LLM.FallbackMaxTokens != 2048 {
|
||||
t.Errorf("LLM.FallbackMaxTokens = %d, want 2048 (process env wins)", cfg.LLM.FallbackMaxTokens)
|
||||
}
|
||||
}
|
||||
|
||||
func TestLoad_MissingRequiredFails(t *testing.T) {
|
||||
// Do not set JWT_SECRET.
|
||||
if _, err := Load(""); err == nil {
|
||||
|
||||
@@ -42,6 +42,11 @@ type Group struct {
|
||||
Model string `json:"model"`
|
||||
Fallback string `json:"fallback,omitempty"`
|
||||
Multimodal bool `json:"multimodal,omitempty"`
|
||||
// MaxTokens is the group's own token budget, applied to every model in the
|
||||
// group's fallback chain (primary and fallback). Zero means "unset": the
|
||||
// last-resort fallback step then falls back to LLM_FALLBACK_MAX_TOKENS, and
|
||||
// beyond that the adapter default applies.
|
||||
MaxTokens int `json:"maxTokens,omitempty"`
|
||||
}
|
||||
|
||||
// ErrInvalidGroup is returned when a group fails schema validation.
|
||||
@@ -67,6 +72,9 @@ func (g Group) Validate() error {
|
||||
if strings.TrimSpace(g.Model) == "" {
|
||||
return fmt.Errorf("model is required")
|
||||
}
|
||||
if g.MaxTokens < 0 {
|
||||
return fmt.Errorf("maxTokens must be >= 0, got %d", g.MaxTokens)
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
|
||||
@@ -164,6 +164,59 @@ func TestSaveLoadRoundTrip(t *testing.T) {
|
||||
}
|
||||
}
|
||||
|
||||
func TestGroupMaxTokensRoundTrip(t *testing.T) {
|
||||
// maxTokens is decoded from a hand-written models.json.
|
||||
path := writeModelsFile(t, `[
|
||||
{"id":"openai","provider":"openai","baseUrl":"https://a","apiKey":"sk-1","model":"gpt-4o","maxTokens":4096}
|
||||
]`)
|
||||
groups, warn := LoadModels(path, envCfg())
|
||||
if warn != nil {
|
||||
t.Fatalf("warn = %v, want nil", warn)
|
||||
}
|
||||
if len(groups) != 1 || groups[0].MaxTokens != 4096 {
|
||||
t.Fatalf("groups = %+v, want one group with maxTokens 4096", groups)
|
||||
}
|
||||
|
||||
// SaveModels writes it back, and LoadModels reads it again.
|
||||
out := filepath.Join(t.TempDir(), "models.json")
|
||||
in := []Group{
|
||||
{ID: "openai", Provider: ProviderOpenAI, BaseURL: "https://a", APIKey: "sk-1", Model: "gpt-4o", MaxTokens: 2048},
|
||||
{ID: "claude", Provider: ProviderAnthropic, BaseURL: "https://b", APIKey: "sk-2", Model: "claude"},
|
||||
}
|
||||
if err := SaveModels(out, in); err != nil {
|
||||
t.Fatalf("SaveModels: %v", err)
|
||||
}
|
||||
raw, err := os.ReadFile(out)
|
||||
if err != nil {
|
||||
t.Fatalf("read saved file: %v", err)
|
||||
}
|
||||
if !strings.Contains(string(raw), `"maxTokens": 2048`) {
|
||||
t.Errorf("saved file missing maxTokens: %s", raw)
|
||||
}
|
||||
round, warn := LoadModels(out, envCfg())
|
||||
if warn != nil {
|
||||
t.Fatalf("load warn = %v", warn)
|
||||
}
|
||||
if round[0].MaxTokens != 2048 || round[1].MaxTokens != 0 {
|
||||
t.Fatalf("round trip maxTokens = (%d,%d), want (2048,0)", round[0].MaxTokens, round[1].MaxTokens)
|
||||
}
|
||||
|
||||
// Upsert and Activate preserve the field.
|
||||
upserted := Upsert(round, Group{ID: "openai", Provider: ProviderOpenAI, BaseURL: "https://a2", Model: "gpt-4o", MaxTokens: 3000})
|
||||
if upserted[0].MaxTokens != 3000 {
|
||||
t.Errorf("Upsert maxTokens = %d, want 3000", upserted[0].MaxTokens)
|
||||
}
|
||||
activated := Activate(round, Group{ID: "claude", Provider: ProviderAnthropic, BaseURL: "https://b", Model: "claude", MaxTokens: 512})
|
||||
if activated[0].MaxTokens != 512 {
|
||||
t.Errorf("Activate maxTokens = %d, want 512", activated[0].MaxTokens)
|
||||
}
|
||||
|
||||
// A negative budget is rejected by validation (never silently persisted).
|
||||
if err := (Group{ID: "x", Provider: ProviderOpenAI, BaseURL: "https://x", Model: "m", MaxTokens: -1}).Validate(); err == nil {
|
||||
t.Fatal("expected Validate to reject a negative maxTokens")
|
||||
}
|
||||
}
|
||||
|
||||
func TestSaveModels_RejectsInvalid(t *testing.T) {
|
||||
path := filepath.Join(t.TempDir(), "models.json")
|
||||
if err := SaveModels(path, []Group{{ID: "", Provider: "x", BaseURL: "https://a", Model: "m"}}); err == nil {
|
||||
|
||||
@@ -0,0 +1,185 @@
|
||||
package model
|
||||
|
||||
import (
|
||||
"context"
|
||||
"encoding/json"
|
||||
"io"
|
||||
"net/http"
|
||||
"net/http/httptest"
|
||||
"path/filepath"
|
||||
"testing"
|
||||
|
||||
"git.yeij.top/AskaEth/Cyrene_v2/internal/config"
|
||||
"git.yeij.top/AskaEth/Cyrene_v2/internal/config/runtimeconfig"
|
||||
)
|
||||
|
||||
// newFallbackDispatchServer serves two chain steps from one endpoint: the
|
||||
// "main" model fails so the chain moves on, every other model succeeds.
|
||||
func newFallbackDispatchServer(t *testing.T, ch chan<- capturedRequest) *httptest.Server {
|
||||
t.Helper()
|
||||
srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
|
||||
var body map[string]any
|
||||
_ = json.NewDecoder(r.Body).Decode(&body)
|
||||
ch <- capturedRequest{method: r.Method, path: r.URL.Path, body: body}
|
||||
if body["model"] == "main" {
|
||||
w.WriteHeader(http.StatusInternalServerError)
|
||||
_, _ = io.WriteString(w, `{"error":"down"}`)
|
||||
return
|
||||
}
|
||||
w.Header().Set("Content-Type", "application/json")
|
||||
w.WriteHeader(http.StatusOK)
|
||||
_, _ = io.WriteString(w, okCompletion)
|
||||
}))
|
||||
t.Cleanup(srv.Close)
|
||||
return srv
|
||||
}
|
||||
|
||||
func chatOnce(t *testing.T, chain *Chain) {
|
||||
t.Helper()
|
||||
if _, err := chain.Chat(context.Background(),
|
||||
Request{Messages: []Message{TextMessage(RoleUser, "hi")}}, Options{}); err != nil {
|
||||
t.Fatalf("Chain.Chat error: %v", err)
|
||||
}
|
||||
}
|
||||
|
||||
// TestChain_Chat_GroupBudgetAppliesToPrimary asserts the active group's own
|
||||
// maxTokens is applied to a request that carries no explicit budget.
|
||||
func TestChain_Chat_GroupBudgetAppliesToPrimary(t *testing.T) {
|
||||
t.Parallel()
|
||||
ch := make(chan capturedRequest, 1)
|
||||
srv := newCompletionServer(t, http.StatusOK, "application/json", okCompletion, ch)
|
||||
chain := NewChain(Step{Model: "m", Adapter: NewOpenAIChat(config.LLMConfig{BaseURL: srv.URL, Model: "m"})}).
|
||||
WithTokenBudget(4096, 8192)
|
||||
|
||||
chatOnce(t, chain)
|
||||
if got := (<-ch).body["max_tokens"]; got != float64(4096) {
|
||||
t.Errorf("body.max_tokens = %v, want the group budget 4096", got)
|
||||
}
|
||||
}
|
||||
|
||||
// TestChain_Chat_ExplicitRequestWinsOverGroupBudget asserts an explicit
|
||||
// Request.MaxTokens is never overridden (the small review/judge budgets).
|
||||
func TestChain_Chat_ExplicitRequestWinsOverGroupBudget(t *testing.T) {
|
||||
t.Parallel()
|
||||
ch := make(chan capturedRequest, 1)
|
||||
srv := newCompletionServer(t, http.StatusOK, "application/json", okCompletion, ch)
|
||||
chain := NewChain(Step{Model: "m", Adapter: NewOpenAIChat(config.LLMConfig{BaseURL: srv.URL, Model: "m"})}).
|
||||
WithTokenBudget(4096, 8192)
|
||||
|
||||
if _, err := chain.Chat(context.Background(),
|
||||
Request{Messages: []Message{TextMessage(RoleUser, "hi")}, MaxTokens: 64}, Options{}); err != nil {
|
||||
t.Fatalf("Chain.Chat error: %v", err)
|
||||
}
|
||||
if got := (<-ch).body["max_tokens"]; got != float64(64) {
|
||||
t.Errorf("body.max_tokens = %v, want the explicit 64", got)
|
||||
}
|
||||
}
|
||||
|
||||
// TestChain_Chat_FallbackEndAppliesEnvBudget asserts the last-resort budget only
|
||||
// reaches the final fallback step, never the primary.
|
||||
func TestChain_Chat_FallbackEndAppliesEnvBudget(t *testing.T) {
|
||||
t.Parallel()
|
||||
primaryCh := make(chan capturedRequest, 1)
|
||||
primary := newCompletionServer(t, http.StatusInternalServerError, "application/json", `{"error":"down"}`, primaryCh)
|
||||
backupCh := make(chan capturedRequest, 1)
|
||||
backup := newCompletionServer(t, http.StatusOK, "application/json", okCompletion, backupCh)
|
||||
chain := NewChain(
|
||||
Step{Model: "main", Adapter: NewOpenAIChat(config.LLMConfig{BaseURL: primary.URL, Model: "main"})},
|
||||
Step{Model: "backup", Adapter: NewOpenAIChat(config.LLMConfig{BaseURL: backup.URL, Model: "backup"})},
|
||||
).WithTokenBudget(0, 3000)
|
||||
|
||||
chatOnce(t, chain)
|
||||
if got := (<-primaryCh).body["max_tokens"]; got != nil {
|
||||
t.Errorf("primary body.max_tokens = %v, want omitted (no budget on the primary)", got)
|
||||
}
|
||||
if got := (<-backupCh).body["max_tokens"]; got != float64(3000) {
|
||||
t.Errorf("fallback body.max_tokens = %v, want the env fallback budget 3000", got)
|
||||
}
|
||||
}
|
||||
|
||||
// TestChain_Chat_GroupBudgetWinsOverFallbackEnv asserts a group maxTokens beats
|
||||
// the last-resort env budget even on the final fallback step.
|
||||
func TestChain_Chat_GroupBudgetWinsOverFallbackEnv(t *testing.T) {
|
||||
t.Parallel()
|
||||
primaryCh := make(chan capturedRequest, 1)
|
||||
primary := newCompletionServer(t, http.StatusInternalServerError, "application/json", `{"error":"down"}`, primaryCh)
|
||||
backupCh := make(chan capturedRequest, 1)
|
||||
backup := newCompletionServer(t, http.StatusOK, "application/json", okCompletion, backupCh)
|
||||
chain := NewChain(
|
||||
Step{Model: "main", Adapter: NewOpenAIChat(config.LLMConfig{BaseURL: primary.URL, Model: "main"})},
|
||||
Step{Model: "backup", Adapter: NewOpenAIChat(config.LLMConfig{BaseURL: backup.URL, Model: "backup"})},
|
||||
).WithTokenBudget(2000, 999)
|
||||
|
||||
chatOnce(t, chain)
|
||||
if got := (<-backupCh).body["max_tokens"]; got != float64(2000) {
|
||||
t.Errorf("fallback body.max_tokens = %v, want the group budget 2000", got)
|
||||
}
|
||||
}
|
||||
|
||||
// TestChain_Chat_SingleStepIgnoresFallbackEnv asserts a chain with no fallback
|
||||
// model (one step = the primary) never gets the last-resort budget, so the
|
||||
// adapter default is preserved.
|
||||
func TestChain_Chat_SingleStepIgnoresFallbackEnv(t *testing.T) {
|
||||
t.Parallel()
|
||||
ch := make(chan capturedRequest, 1)
|
||||
srv := newCompletionServer(t, http.StatusOK, "application/json", okCompletion, ch)
|
||||
chain := NewChain(Step{Model: "m", Adapter: NewOpenAIChat(config.LLMConfig{BaseURL: srv.URL, Model: "m"})}).
|
||||
WithTokenBudget(0, 3000)
|
||||
|
||||
chatOnce(t, chain)
|
||||
if got := (<-ch).body["max_tokens"]; got != nil {
|
||||
t.Errorf("body.max_tokens = %v, want omitted (single-step chain is not a fallback)", got)
|
||||
}
|
||||
}
|
||||
|
||||
// TestRuntimeLLM_GroupMaxTokensEndToEnd proves the models.json maxTokens field
|
||||
// reaches the provider request body through the real runtime stack.
|
||||
func TestRuntimeLLM_GroupMaxTokensEndToEnd(t *testing.T) {
|
||||
ch := make(chan capturedRequest, 1)
|
||||
srv := newCompletionServer(t, http.StatusOK, "application/json", okCompletion, ch)
|
||||
path := filepath.Join(t.TempDir(), "models.json")
|
||||
if err := runtimeconfig.SaveModels(path, []runtimeconfig.Group{
|
||||
{ID: "openai", Provider: ProviderOpenAI, BaseURL: srv.URL, APIKey: "sk-1", Model: "m", MaxTokens: 1234},
|
||||
}); err != nil {
|
||||
t.Fatalf("SaveModels: %v", err)
|
||||
}
|
||||
m := NewRuntimeLLM(path, rtEnv(), nil)
|
||||
if got := m.Groups()[0].MaxTokens; got != 1234 {
|
||||
t.Fatalf("Groups()[0].MaxTokens = %d, want 1234", got)
|
||||
}
|
||||
if _, err := m.Chat(context.Background(),
|
||||
Request{Messages: []Message{TextMessage(RoleUser, "hi")}}, Options{}); err != nil {
|
||||
t.Fatalf("RuntimeLLM.Chat error: %v", err)
|
||||
}
|
||||
if got := (<-ch).body["max_tokens"]; got != float64(1234) {
|
||||
t.Errorf("body.max_tokens = %v, want the group budget 1234", got)
|
||||
}
|
||||
}
|
||||
|
||||
// TestRuntimeLLM_FallbackEndAppliesEnvBudget proves LLM_FALLBACK_MAX_TOKENS is
|
||||
// wired from config through buildSnapshot into the final fallback step.
|
||||
func TestRuntimeLLM_FallbackEndAppliesEnvBudget(t *testing.T) {
|
||||
ch := make(chan capturedRequest, 2)
|
||||
srv := newFallbackDispatchServer(t, ch)
|
||||
path := filepath.Join(t.TempDir(), "models.json")
|
||||
if err := runtimeconfig.SaveModels(path, []runtimeconfig.Group{
|
||||
{ID: "openai", Provider: ProviderOpenAI, BaseURL: srv.URL, APIKey: "sk-1", Model: "main", Fallback: "backup"},
|
||||
}); err != nil {
|
||||
t.Fatalf("SaveModels: %v", err)
|
||||
}
|
||||
env := rtEnv()
|
||||
env.FallbackMaxTokens = 2500
|
||||
m := NewRuntimeLLM(path, env, nil)
|
||||
if _, err := m.Chat(context.Background(),
|
||||
Request{Messages: []Message{TextMessage(RoleUser, "hi")}}, Options{}); err != nil {
|
||||
t.Fatalf("RuntimeLLM.Chat error: %v", err)
|
||||
}
|
||||
primary := <-ch
|
||||
backup := <-ch
|
||||
if got := primary.body["max_tokens"]; got != nil {
|
||||
t.Errorf("primary body.max_tokens = %v, want omitted", got)
|
||||
}
|
||||
if got := backup.body["max_tokens"]; got != float64(2500) {
|
||||
t.Errorf("fallback body.max_tokens = %v, want LLM_FALLBACK_MAX_TOKENS 2500", got)
|
||||
}
|
||||
}
|
||||
@@ -19,6 +19,13 @@ type Step struct {
|
||||
// single provider is expected.
|
||||
type Chain struct {
|
||||
steps []Step
|
||||
// groupMaxTokens is the active model group's own budget (models.json
|
||||
// maxTokens). When > 0 it applies to every step of the group's chain and
|
||||
// wins over fallbackMaxTokens.
|
||||
groupMaxTokens int
|
||||
// fallbackMaxTokens is the last-resort budget (LLM_FALLBACK_MAX_TOKENS). It
|
||||
// only applies to the final step of a chain that actually has a fallback.
|
||||
fallbackMaxTokens int
|
||||
}
|
||||
|
||||
// NewChain builds a fallback chain from the given steps.
|
||||
@@ -26,6 +33,37 @@ func NewChain(steps ...Step) *Chain {
|
||||
return &Chain{steps: steps}
|
||||
}
|
||||
|
||||
// WithTokenBudget sets the chain's token budgets: the active group's own budget
|
||||
// and the last-resort fallback budget. It must be called before the chain is
|
||||
// published to concurrent callers (the runtime builds a fresh chain per
|
||||
// snapshot and swaps it in atomically), so the fields are never mutated in
|
||||
// flight.
|
||||
func (c *Chain) WithTokenBudget(groupMaxTokens, fallbackMaxTokens int) *Chain {
|
||||
c.groupMaxTokens = groupMaxTokens
|
||||
c.fallbackMaxTokens = fallbackMaxTokens
|
||||
return c
|
||||
}
|
||||
|
||||
// applyBudget resolves the effective MaxTokens for step idx, honoring the
|
||||
// precedence: an explicit Request.MaxTokens > the group's MaxTokens > the
|
||||
// last-resort LLM_FALLBACK_MAX_TOKENS > the adapter default. The last-resort
|
||||
// budget only applies when the chain really has a fallback step (len > 1) and
|
||||
// only to that final step; a single-step chain is a primary model, not a
|
||||
// fallback, so its adapter default is left untouched.
|
||||
func (c *Chain) applyBudget(req Request, idx int) Request {
|
||||
if req.MaxTokens != 0 {
|
||||
return req
|
||||
}
|
||||
if c.groupMaxTokens > 0 {
|
||||
req.MaxTokens = c.groupMaxTokens
|
||||
return req
|
||||
}
|
||||
if c.fallbackMaxTokens > 0 && len(c.steps) > 1 && idx == len(c.steps)-1 {
|
||||
req.MaxTokens = c.fallbackMaxTokens
|
||||
}
|
||||
return req
|
||||
}
|
||||
|
||||
// NewOpenAIChatChain builds a fallback chain from a single supplier config: the
|
||||
// primary model first, then cfg.Fallback (when set and distinct), both served
|
||||
// by an OpenAIChatAdapter against the same OpenAI-compatible endpoint. It
|
||||
@@ -45,8 +83,8 @@ func (c *Chain) Chat(ctx context.Context, req Request, opts Options) (Response,
|
||||
return Response{}, ErrNoSteps
|
||||
}
|
||||
var lastErr error
|
||||
for _, s := range c.steps {
|
||||
stepReq := req
|
||||
for i, s := range c.steps {
|
||||
stepReq := c.applyBudget(req, i)
|
||||
if s.Model != "" {
|
||||
stepReq.Model = s.Model
|
||||
}
|
||||
@@ -68,8 +106,8 @@ func (c *Chain) ChatStream(ctx context.Context, req Request, onEvent func(Stream
|
||||
return Response{}, ErrNoSteps
|
||||
}
|
||||
var lastErr error
|
||||
for _, s := range c.steps {
|
||||
stepReq := req
|
||||
for i, s := range c.steps {
|
||||
stepReq := c.applyBudget(req, i)
|
||||
if s.Model != "" {
|
||||
stepReq.Model = s.Model
|
||||
}
|
||||
|
||||
@@ -169,6 +169,12 @@ func buildSnapshot(groups []runtimeconfig.Group, env config.LLMConfig) (*runtime
|
||||
if err != nil && !errors.Is(err, ErrNoModel) {
|
||||
return nil, err
|
||||
}
|
||||
if chain != nil {
|
||||
// Budgets are resolved inside the chain: the active group's own
|
||||
// maxTokens (models.json) wins, and env.FallbackMaxTokens only caps the
|
||||
// last step of a chain that really has a fallback (design/04 §3.1).
|
||||
chain = chain.WithTokenBudget(active.MaxTokens, env.FallbackMaxTokens)
|
||||
}
|
||||
router := NewRouter(DefaultModelForProvider(cfg),
|
||||
WithVisionModel(env.VisionModel),
|
||||
WithDefaultProvider(cfg.Provider),
|
||||
|
||||
@@ -58,6 +58,10 @@ type AdminLlmProvider struct {
|
||||
Multimodal bool `json:"multimodal"`
|
||||
Enabled bool `json:"enabled"`
|
||||
Fallback string `json:"fallback,omitempty"`
|
||||
// MaxTokens is the group's own token budget (0 = unset; the runtime then
|
||||
// falls back to LLM_FALLBACK_MAX_TOKENS on the last fallback step, then the
|
||||
// adapter default).
|
||||
MaxTokens int `json:"maxTokens,omitempty"`
|
||||
}
|
||||
|
||||
// AdminLlmConfig is the wire shape of GET /api/v1/admin/llm.
|
||||
@@ -360,6 +364,7 @@ func (s *Server) adminGetLlm(w http.ResponseWriter, r *http.Request) {
|
||||
Multimodal: g.Multimodal,
|
||||
Enabled: i == 0,
|
||||
Fallback: g.Fallback,
|
||||
MaxTokens: g.MaxTokens,
|
||||
})
|
||||
}
|
||||
writeJSON(w, http.StatusOK, AdminLlmConfig{Providers: providers})
|
||||
|
||||
@@ -37,7 +37,8 @@ const (
|
||||
|
||||
// llmProviderRequest is the POST /api/v1/admin/llm body for adding/updating a
|
||||
// provider group. Name is a client-side display label (not persisted — the
|
||||
// models.json schema carries id/provider/baseUrl/apiKey/model/fallback/multimodal).
|
||||
// models.json schema carries id/provider/baseUrl/apiKey/model/fallback/
|
||||
// multimodal/maxTokens).
|
||||
type llmProviderRequest struct {
|
||||
ID string `json:"id"`
|
||||
Name string `json:"name"`
|
||||
@@ -47,6 +48,10 @@ type llmProviderRequest struct {
|
||||
Model string `json:"model"`
|
||||
Multimodal bool `json:"multimodal"`
|
||||
Fallback string `json:"fallback"`
|
||||
// MaxTokens is the group's own token budget. Zero means unset (the runtime
|
||||
// falls back to LLM_FALLBACK_MAX_TOKENS on the last fallback step, then the
|
||||
// adapter default). Negative is rejected.
|
||||
MaxTokens int `json:"maxTokens"`
|
||||
}
|
||||
|
||||
// adminAddLlmProvider saves a provider group to models.json (read-modify-write,
|
||||
@@ -64,6 +69,11 @@ func (s *Server) adminAddLlmProvider(w http.ResponseWriter, r *http.Request) {
|
||||
writeError(w, apperr.Wrap(apperr.ErrBadRequest, "bad_request", "baseUrl and model are required"))
|
||||
return
|
||||
}
|
||||
if req.MaxTokens < 0 {
|
||||
s.auditClaims(r, "llm.addProvider", "error")
|
||||
writeError(w, apperr.Wrap(apperr.ErrBadRequest, "bad_request", "maxTokens must be >= 0"))
|
||||
return
|
||||
}
|
||||
provider := req.Provider
|
||||
if provider == "" {
|
||||
provider = model.ProviderOpenAI
|
||||
@@ -92,6 +102,7 @@ func (s *Server) adminAddLlmProvider(w http.ResponseWriter, r *http.Request) {
|
||||
Model: req.Model,
|
||||
Fallback: req.Fallback,
|
||||
Multimodal: req.Multimodal,
|
||||
MaxTokens: req.MaxTokens,
|
||||
})
|
||||
if err := runtimeconfig.SaveModels(s.llm.Path(), groups); err != nil {
|
||||
s.auditClaims(r, "llm.addProvider", "error")
|
||||
@@ -114,6 +125,7 @@ func (s *Server) adminAddLlmProvider(w http.ResponseWriter, r *http.Request) {
|
||||
Multimodal: req.Multimodal,
|
||||
Enabled: id == s.llm.Group().ID,
|
||||
Fallback: req.Fallback,
|
||||
MaxTokens: req.MaxTokens,
|
||||
}
|
||||
s.auditClaims(r, "llm.addProvider", "ok")
|
||||
writeJSON(w, http.StatusCreated, map[string]any{
|
||||
|
||||
@@ -231,7 +231,7 @@ func TestAdminAddLlmProviderPersistsAndReloads(t *testing.T) {
|
||||
tok := signAny(t, jwt, "acct", auth.RoleOwner)
|
||||
|
||||
resp, body := doReq(t, http.MethodPost, ts.URL+"/api/v1/admin/llm",
|
||||
map[string]any{"id": "deepseek", "provider": "openai", "baseUrl": "https://api.deepseek.com/v1", "apiKey": "sk-ds", "model": "deepseek-chat", "multimodal": true}, tok)
|
||||
map[string]any{"id": "deepseek", "provider": "openai", "baseUrl": "https://api.deepseek.com/v1", "apiKey": "sk-ds", "model": "deepseek-chat", "multimodal": true, "maxTokens": 1234}, tok)
|
||||
if resp.StatusCode != http.StatusCreated {
|
||||
t.Fatalf("status = %d, want 201 (body=%s)", resp.StatusCode, body)
|
||||
}
|
||||
@@ -243,7 +243,7 @@ func TestAdminAddLlmProviderPersistsAndReloads(t *testing.T) {
|
||||
if err := json.Unmarshal(decodeBody(t, body)["data"], &res); err != nil {
|
||||
t.Fatalf("decode add: %v", err)
|
||||
}
|
||||
if !res.Persisted || res.Path != modelsPath || res.Provider.Model != "deepseek-chat" {
|
||||
if !res.Persisted || res.Path != modelsPath || res.Provider.Model != "deepseek-chat" || res.Provider.MaxTokens != 1234 {
|
||||
t.Fatalf("add res = %+v", res)
|
||||
}
|
||||
|
||||
@@ -252,8 +252,8 @@ func TestAdminAddLlmProviderPersistsAndReloads(t *testing.T) {
|
||||
if warn != nil {
|
||||
t.Fatalf("load models warn = %v", warn)
|
||||
}
|
||||
if len(groups) == 0 || groups[0].ID != "deepseek" || groups[0].APIKey != "sk-ds" {
|
||||
t.Fatalf("file groups = %+v, want deepseek first with sk-ds", groups)
|
||||
if len(groups) == 0 || groups[0].ID != "deepseek" || groups[0].APIKey != "sk-ds" || groups[0].MaxTokens != 1234 {
|
||||
t.Fatalf("file groups = %+v, want deepseek first with sk-ds and maxTokens 1234", groups)
|
||||
}
|
||||
|
||||
// The running stack reflects the new active provider/model (hot reload).
|
||||
@@ -270,8 +270,8 @@ func TestAdminAddLlmProviderPersistsAndReloads(t *testing.T) {
|
||||
if err := json.Unmarshal(decodeBody(t, gBody)["data"], &list); err != nil {
|
||||
t.Fatalf("decode list: %v", err)
|
||||
}
|
||||
if len(list.Providers) == 0 || list.Providers[0].Model != "deepseek-chat" {
|
||||
t.Fatalf("list = %+v, want deepseek-chat first", list.Providers)
|
||||
if len(list.Providers) == 0 || list.Providers[0].Model != "deepseek-chat" || list.Providers[0].MaxTokens != 1234 {
|
||||
t.Fatalf("list = %+v, want deepseek-chat first with maxTokens 1234", list.Providers)
|
||||
}
|
||||
if strings.Contains(string(gBody), "sk-ds") || strings.Contains(string(gBody), "sk-env") {
|
||||
t.Fatalf("API key leaked in GET /admin/llm: %s", gBody)
|
||||
|
||||
Reference in new issue
Block a user