diff --git a/Docs/api-client.md b/Docs/api-client.md index 13bd6a1..0524bf6 100644 --- a/Docs/api-client.md +++ b/Docs/api-client.md @@ -33,6 +33,8 @@ | 10 | `/api/v1/audio/transcribe` | POST | JWT(owner) | ✅ | **语音→文本**：body `{audio:"",format?,deviceId?}` → `{data:{text, provider?}}`（走后端 STT，百炼 `qwen-audio-3.0-asr-flash`；**当结果为空文本/silent 降级时响应体带 `provider`**（如 `"silent"`，非空文本省略），客户端据此提示「没识别到声音」） | | 11 | `/api/v1/audio/speech` | POST | JWT(owner) | ✅ | **文本→音频**：body `{text,voice?,format?,instruction?}` → `200` 音频字节（`Content-Type: audio/`，非 JSON envelope，走后端 TTS，百炼 `cosyvoice-v3-flash`，复刻音色可带 `instruction` 情绪指令；**静默降级空音频时仍 `500 tts_empty`，并附 `X-Voice-Provider: silent` 响应头**，客户端据此提示「没生成声音」） | | 12 | `/api/v1/audio/voice-clone` | POST | JWT(owner) | ✅ | **声音复刻建音色**：body `{url:"<公网参考音频URL>",prefix?}` → `{data:{voiceId}}`（走百炼 `voice-enrollment` create_voice） | +| 13 | `/api/v1/upload` | POST | JWT(owner/admin) | ✅ | **图片上传**（B1）：multipart/form-data，字段 `file`（raster image，5MB 上限）→ `201 {data:{url,size,mime,name}}`；`url` 为**同源访问地址**（`/api/v1/files/`，可直接作 `chat.image`）。校验：`MaxBytesReader` 防超大 → `DetectContentType` 嗅探（**不信客户端 Content-Type**）→ 白名单 png/jpeg/gif/webp/bmp（**拒 SVG/HTML**）；文件名用 **crypto/rand 128-bit 随机名**（原始文件名丢弃），存 `/uploads/`（0700）。错误码：缺文件 `400 bad_request`、超限 `413 payload_too_large`、非白名单 `415 unsupported_media_type`、未授权 `401/403` | +| 14 | `/api/v1/files/{name}` | GET | 无需（能力URL） | ✅ | **上传文件访问**（B1）：从 `/uploads/` 读取，同名端口服务。`name` 仅接受「32 位小写 hex + 白名单扩展名」；**路径穿越全形态拒绝**（`..`/`%2f`/`%2e%2e` 等）。返回 `Content-Type`/`Content-Length`/`Cache-Control: private,max-age=86400,immutable`/`X-Content-Type-Options: nosniff`（支持 Range）。**设计取舍**：GET 不鉴权——该 URL 是**128-bit 随机能力地址**（`chat.image`/`` 无法带 Bearer），POST 仍 owner/admin+审计；任何持有该 URL 者都能取文件，勿外泄到他人 | > **语音 REST 源（`/api/v1/audio/*`，均 owner-only + 审计）**：原生/移动端用它上传/获取音频，**绕开 WS 4KB 帧限**。 > - `POST /api/v1/audio/transcribe`：body `{audio:"", format?:"wav", deviceId?}` → `200 {data:{text}}`。`audio` 为 base64（或 data: URI）；`format` 为编码（`wav`/`mp3`/`ogg`，缺省 `wav`）；空 audio 回 `400`；未注入语音服务 `500 voice_unavailable`、识别失败 `500 stt_failed`。 @@ -211,6 +213,7 @@ - 语音：上行 `voice_input`（base64 音频或已转写 text），下行 `voice_output`（合成音频 base64 + text）。 ## 5. 变更记录 +- `v0.24`（**B1 图片上传端点**）：`POST /api/v1/upload`（**新增**，multipart `file`，owner/admin+审计 → `201 {data:{url,size,mime,name}}`）+ `GET /api/v1/files/{name}`（**新增**，同端口静态服务）。安全：`MaxBytesReader`(5MB)、`DetectContentType` 嗅探（不信客户端 Content-Type）、白名单 raster image（拒 SVG/HTML）、`crypto/rand` 128-bit 随机名（丢弃原始文件名）、非可执行上传目录(0700)、`nosniff`；路径穿越全形态拒绝（含编码 `%2f`/`%2e%2e`）。错误码 `400/413/415`（`internal/errors` 增 `ErrPayloadTooLarge`/`ErrUnsupportedMediaType`）。**设计取舍**：GET 不鉴权（能力 URL，`chat.image`/`` 无法带 Bearer），POST 仍 owner/admin。`url` 为绝对地址（`X-Forwarded-Proto`/TLS + `r.Host`，因 `ParseImageRef` 只认 `http(s)://`）。`Deps.DataDir` 由 wire 注入（`resolveDataDir`）。 - `v0.23`（**C2 语音 silent 降级信号**）：三处统一用 **`provider`** 只在「结果为空」时出现（非空/成功时才省略）——`POST /api/v1/audio/transcribe` 空文本响应体带 `{text:"", provider:"silent"}`；WS `voice_output` 空音频 payload 带 `provider`；`POST /api/v1/audio/speech` 空音频仍 `500 tts_empty` 并加 `X-Voice-Provider: silent` 响应头（CORS `Access-Control-Expose-Headers` 已放开）。`voice.Chain/Service` 新增 `STTWithProvider/TTSWithProvider`（返回真实档位 cloud/local/edge-tts/espeak/silent），网关用可选接口 `voiceProviderReporter` 透传，自定义实现回退 `silent`。**含 `X-Voice-Provider`（`apperr` 包装、`providerSignal`）；`DASHSCOPE_API_KEY` 无/降级时信号为 `silent`。**（`go test -race` 在本机受内核 TSAN VMA 限制失败，非本代码问题。） - `v0.22`（**B3 会话列表 lastMessage 预览**）：`GET /api/v1/sessions` 与 `/api/v1/admin/sessions` 每个会话新增可选 **`lastMessage`** `{content,role,createdAt}`（content 折叠空白截前 60 rune + `…`；空会话省略），**单条 SQL**（LEFT JOIN 取每会话 `ORDER BY created_at DESC,rowid DESC LIMIT 1`，无 N+1，走 `idx_chat_msgs_session_created` 覆盖索引）。`unreadCount` 需已读游标设计，后置。 - `v0.21`（**A4 会话 title**）：`PATCH /api/v1/sessions/{id}`（**新增**，body `{title}` ≤80 rune → `{data:{id,title}}`；空/超长 400、不存在 404、主会话可改名）；会话**首条用户消息到达时自动命名**（`history.append` 前 `ensureTitle`：折叠空白取前 20 rune，纯空白回落 `主会话`/`次级会话`），仅首次命名、不覆盖。`HistoryStore` 新增 `RenameSession`；隔离会话（`sess-xxx`）不在 `chat_sessions` 有行、不自动命名（其本不在 `/sessions` 列表）。**边界**：前端 `createSubSession(title?)` 的 POST body 含 `title` 但后端暂忽略（会被首条消息自动命名覆盖），未处理。 diff --git a/Docs/api-reference.md b/Docs/api-reference.md index 1ed2d0c..218af42 100644 --- a/Docs/api-reference.md +++ b/Docs/api-reference.md @@ -38,9 +38,9 @@ ### 0.3 上传 / 媒体（重要确认） -- **无 `/upload` 端点**。图片走 `POST /api/v1/chat` 或 WS `chat` 的 `image` 字段（**base64 data: URI** 或 http(s) URL，入视觉真推理，见 §6 流式与直连）。 -- **无 REST 语音端点**。语音**只走 WS**：上行 `voice_input`（base64 音频或已转写 text），下行 `voice_output`（合成音频 base64）。见 §6.5。 -- 管理端导入知识文档（RAG）也走 JSON body 文本（`POST /api/v1/admin/rag`），非文件上传。 +- **`POST /api/v1/upload`**（B1）上传图片 → 返回**同源 URL**（`/api/v1/files/`），可直接作 `chat.image`；也可继续用 **base64 data: URI** 或 http(s) URL 走 `POST /api/v1/chat` / WS `chat` 的 `image` 字段（入视觉真推理，见 §6 流式与直连）。 +- **REST 语音端点已存在**：`POST /api/v1/audio/transcribe`（STT）、`POST /api/v1/audio/speech`（TTS）、`POST /api/v1/audio/voice-clone`（声音复刻），均 owner-only + 审计，绕开 WS 4KB 帧限。语音也可走 WS `voice_input`（上行）/`voice_output`（下行）。 +- 管理端导入知识文档（RAG）走 JSON body 文本（`POST /api/v1/admin/rag`），非文件上传。 --- diff --git a/cmd/cyrene/wire.go b/cmd/cyrene/wire.go index a2578ab..79cf4d5 100644 --- a/cmd/cyrene/wire.go +++ b/cmd/cyrene/wire.go @@ -684,6 +684,7 @@ func (a *app) wireFrontend() error { Frontend: resolveFrontendFS(a.log), Webhooks: a.channelBridge.WebhookHandlers(), DatabasePath: a.dbPath, + DataDir: resolveDataDir(a.cfg), }) if err != nil { return fmt.Errorf("gateway: %w", err) diff --git a/internal/errors/error.go b/internal/errors/error.go index adebe51..5b52c78 100644 --- a/internal/errors/error.go +++ b/internal/errors/error.go @@ -96,6 +96,13 @@ var ( // ErrBadRequest means the request is malformed (400). ErrBadRequest = newWithHTTP("bad_request", "bad request", http.StatusBadRequest) + // ErrPayloadTooLarge means the request body exceeds the allowed size (413). + ErrPayloadTooLarge = newWithHTTP("payload_too_large", "payload too large", http.StatusRequestEntityTooLarge) + + // ErrUnsupportedMediaType means the request carries a media type the endpoint + // does not accept (415). + ErrUnsupportedMediaType = newWithHTTP("unsupported_media_type", "unsupported media type", http.StatusUnsupportedMediaType) + // ErrRateLimited means the caller exceeded a rate limit (429). ErrRateLimited = newWithHTTP("rate_limited", "rate limited", http.StatusTooManyRequests) diff --git a/internal/services/gateway/server.go b/internal/services/gateway/server.go index 1b03a4c..4969515 100644 --- a/internal/services/gateway/server.go +++ b/internal/services/gateway/server.go @@ -276,6 +276,11 @@ type Deps struct { // It backs the admin backup/restore endpoints. Optional; when empty those // endpoints answer honestly with a clear not-configured error. DatabasePath string + // DataDir is the resolved data directory (the value resolveDataDir returns, + // which may differ from cfg.DataDir when the OS default applies). It backs + // the image-upload store (/uploads) and its static route. Optional; + // when empty the upload handlers fall back to cfg.DataDir. + DataDir string // Streaming marks the injected Loop as streaming-capable (its OnStream is // wired to publish ws.stream_chunk). When set the hub skips its own single // final stream_chunk emission, since deltas already stream through OnStream. @@ -316,6 +321,10 @@ type Server struct { streaming bool frontend fs.FS dbPath string + // dataDir is the resolved data directory backing the upload store + // (/uploads). Injected by the composition root; cfg.DataDir is the + // fallback. See Deps.DataDir. + dataDir string // llm is the hot-swappable provider stack (models.json). When nil the // admin /llm endpoints fall back to the env config. It is optional so tests // that do not wire it keep working. @@ -364,6 +373,7 @@ func New(cfg *config.Config, deps Deps) (*Server, error) { ownerRegistrar: deps.OwnerRegistrar, webhooks: deps.Webhooks, dbPath: deps.DatabasePath, + dataDir: deps.DataDir, log: sourcedLog(deps.Log), } s.hub = newHub(s) @@ -395,6 +405,14 @@ func (s *Server) Handler() http.Handler { mux.Handle("POST /api/v1/audio/speech", s.auth.Authorize(auth.RoleOwner)(http.HandlerFunc(s.handleSpeech))) mux.Handle("POST /api/v1/audio/voice-clone", s.auth.Authorize(auth.RoleOwner)(http.HandlerFunc(s.handleVoiceClone))) + // Image upload (B1): a multipart POST that stores a raster image under the + // resolved data dir and returns a same-origin URL usable directly as + // chat.image. Upload is owner/admin + audited; the static route below is a + // capability URL (unguessable name) because chat.image / cannot carry a + // bearer token. Both ride the one gateway port (single-binary red line). + mux.Handle("POST /api/v1/upload", s.auth.Authorize(auth.RoleOwner, auth.RoleAdmin)(http.HandlerFunc(s.handleUpload))) + mux.HandleFunc("GET /api/v1/files/{name}", s.handleUploadFile) + // Session/model reads and writes are the owner's private history; owner/admin only // (chat uses the same roles). Session metadata in GET /sessions stays readAuth for // any authenticated role, but the message content (and thinking) is owner/admin-only @@ -480,6 +498,7 @@ func (s *Server) Handler() http.Handler { } var h http.Handler = mux + h = guardUploadPaths(h) if s.frontend != nil { h = s.spa(h) } diff --git a/internal/services/gateway/upload.go b/internal/services/gateway/upload.go new file mode 100644 index 0000000..c769ad5 --- /dev/null +++ b/internal/services/gateway/upload.go @@ -0,0 +1,371 @@ +package gateway + +import ( + "crypto/rand" + "encoding/hex" + "errors" + "fmt" + "io" + "mime/multipart" + "net" + "net/http" + "os" + "path/filepath" + "strconv" + "strings" + + apperr "git.yeij.top/AskaEth/Cyrene_v2/internal/errors" +) + +// maxUploadBytes caps a single uploaded image at 5 MiB. The multipart envelope +// (boundaries, part headers) gets a little headroom on top of that; the decoded +// file itself is checked against this exact limit. +const ( + maxUploadBytes = 5 << 20 + uploadMultipartSlack = 1 << 20 + uploadNameBytes = 16 + uploadSniffBytes = 512 +) + +// uploadFilePrefix is the same-origin static route that serves stored uploads +// (registered in Handler). It lives under /api/v1/ so the gateway mux — not the +// SPA fallback — owns it, and so it is part of the versioned client contract. +const uploadFilePrefix = "/api/v1/files/" + +// uploadResponse is the POST /api/v1/upload success payload. URL is an absolute +// http(s) URL built from the request authority so it can be handed straight to +// chat.image: the vision layer classifies only http(s):// as a URL and would +// otherwise treat a bare path as base64. Name is the server-generated storage +// name; the client's original filename is never echoed back or persisted. +type uploadResponse struct { + URL string `json:"url"` + Size int64 `json:"size"` + MIME string `json:"mime"` + Name string `json:"name"` +} + +// handleUpload accepts a multipart/form-data request with a single `file` part, +// validates it is a raster image within the size limit, stores it under +// /uploads under a cryptographically random name, audits the write and +// returns the URL to attach as chat.image. It is owner/admin only (registered in +// Handler). The client's original filename is deliberately ignored: it is +// attacker-controlled and could carry a path separator or overwrite a file. +func (s *Server) handleUpload(w http.ResponseWriter, r *http.Request) { + r.Body = http.MaxBytesReader(w, r.Body, maxUploadBytes+uploadMultipartSlack) + if err := r.ParseMultipartForm(1 << 20); err != nil { + s.auditClaims(r, "upload.image", "error") + writeUploadBodyError(w, err) + return + } + file, header, err := r.FormFile("file") + if err != nil { + s.auditClaims(r, "upload.image", "error") + writeError(w, apperr.Wrap(apperr.ErrBadRequest, "bad_request", "file is required")) + return + } + defer func() { _ = file.Close() }() + + if header.Size <= 0 { + s.auditClaims(r, "upload.image", "error") + writeError(w, apperr.Wrap(apperr.ErrBadRequest, "bad_request", "file is empty")) + return + } + if header.Size > maxUploadBytes { + s.auditClaims(r, "upload.image", "error") + writeError(w, apperr.Wrap(apperr.ErrPayloadTooLarge, "payload_too_large", "file exceeds the 5MB limit")) + return + } + mime, err := sniffImageMIME(file) + if err != nil { + s.auditClaims(r, "upload.image", "error") + s.log.Error("upload sniff", "err", err) + writeError(w, apperr.Wrap(apperr.ErrInternal, "upload_failed", "failed to read upload")) + return + } + ext, ok := uploadExtForMIME(mime) + if !ok { + s.auditClaims(r, "upload.image", "error") + writeError(w, apperr.Wrap(apperr.ErrUnsupportedMediaType, "unsupported_media_type", "only png/jpeg/gif/webp/bmp images are accepted")) + return + } + name, err := s.storeUpload(file, ext) + if err != nil { + s.auditClaims(r, "upload.image", "error") + s.log.Error("upload store", "err", err) + writeError(w, apperr.Wrap(apperr.ErrInternal, "upload_failed", "failed to store upload")) + return + } + s.auditClaims(r, "upload.image", "ok") + writeJSON(w, http.StatusCreated, uploadResponse{ + URL: s.uploadURL(r, name), + Size: header.Size, + MIME: mime, + Name: name, + }) +} + +// guardUploadPaths rejects a static-upload request whose decoded path contains a +// parent-directory segment, before ServeMux can clean it and redirect the client +// to a non-upload path (which the SPA would then answer with index.html). The +// {name} route only ever matches one segment, so a `..` can only be an escape +// attempt; answering 404 here keeps every traversal shape uniformly rejected. +// Non-upload paths pass through untouched. +func guardUploadPaths(next http.Handler) http.Handler { + return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { + if strings.HasPrefix(r.URL.Path, uploadFilePrefix) && hasDotDotSegment(r.URL.Path) { + writeError(w, apperr.Wrap(apperr.ErrNotFound, "not_found", "file not found")) + return + } + next.ServeHTTP(w, r) + }) +} + +// hasDotDotSegment reports whether p has a literal ".." path segment. +func hasDotDotSegment(p string) bool { + for _, seg := range strings.Split(p, "/") { + if seg == ".." { + return true + } + } + return false +} + +// handleUploadFile serves a stored upload from /uploads. It is +// unauthenticated by design: the returned URL is handed to chat.image and to +// tags, neither of which can attach a bearer token. The random 128-bit +// name is the capability; the handler accepts only a strict hex+known-extension +// name and never lets a request path escape the uploads directory. +func (s *Server) handleUploadFile(w http.ResponseWriter, r *http.Request) { + name := r.PathValue("name") + if !validUploadName(name) { + writeError(w, apperr.Wrap(apperr.ErrNotFound, "not_found", "file not found")) + return + } + dir, err := s.uploadDir() + if err != nil { + writeError(w, apperr.Wrap(apperr.ErrNotFound, "not_found", "file not found")) + return + } + path := filepath.Join(dir, name) + if filepath.Dir(path) != filepath.Clean(dir) { + writeError(w, apperr.Wrap(apperr.ErrNotFound, "not_found", "file not found")) + return + } + mime, ok := uploadMIMEForExt(filepath.Ext(name)) + if !ok { + writeError(w, apperr.Wrap(apperr.ErrNotFound, "not_found", "file not found")) + return + } + f, err := os.Open(path) + if err != nil { + writeError(w, apperr.Wrap(apperr.ErrNotFound, "not_found", "file not found")) + return + } + defer func() { _ = f.Close() }() + info, err := f.Stat() + if err != nil || info.IsDir() { + writeError(w, apperr.Wrap(apperr.ErrNotFound, "not_found", "file not found")) + return + } + w.Header().Set("Content-Type", mime) + w.Header().Set("Cache-Control", "private, max-age=86400, immutable") + w.Header().Set("X-Content-Type-Options", "nosniff") + http.ServeContent(w, r, name, info.ModTime(), f) +} + +// writeUploadBodyError maps a multipart parse failure to its wire error: an +// over-limit body is 413, anything else (not multipart, malformed part) is 400. +func writeUploadBodyError(w http.ResponseWriter, err error) { + var maxErr *http.MaxBytesError + if errors.As(err, &maxErr) { + writeError(w, apperr.Wrap(apperr.ErrPayloadTooLarge, "payload_too_large", "file exceeds the 5MB limit")) + return + } + writeError(w, apperr.Wrap(apperr.ErrBadRequest, "bad_request", "invalid multipart body")) +} + +// sniffImageMIME detects the content type of the upload head with the stdlib +// sniffer (never the client-supplied Content-Type header, which is untrusted) +// and rewinds the part so the caller can copy it in full. +func sniffImageMIME(f multipart.File) (string, error) { + buf := make([]byte, uploadSniffBytes) + n, err := io.ReadFull(f, buf) + if err != nil && !errors.Is(err, io.EOF) && !errors.Is(err, io.ErrUnexpectedEOF) { + return "", fmt.Errorf("read upload head: %w", err) + } + if _, err := f.Seek(0, io.SeekStart); err != nil { + return "", fmt.Errorf("rewind upload: %w", err) + } + return http.DetectContentType(buf[:n]), nil +} + +// uploadExtForMIME maps a sniffed image MIME to the extension used on disk. The +// whitelist is deliberately raster-only: SVG (and HTML) would be served inline +// and could execute script, so they are not accepted. +func uploadExtForMIME(mime string) (string, bool) { + switch mime { + case "image/png": + return ".png", true + case "image/jpeg": + return ".jpg", true + case "image/gif": + return ".gif", true + case "image/webp": + return ".webp", true + case "image/bmp": + return ".bmp", true + default: + return "", false + } +} + +// uploadMIMEForExt returns the Content-Type for a stored upload extension. It is +// the read-side twin of uploadExtForMIME: only whitelisted extensions resolve, +// which is what makes validUploadName strict. +func uploadMIMEForExt(ext string) (string, bool) { + switch strings.ToLower(ext) { + case ".png": + return "image/png", true + case ".jpg", ".jpeg": + return "image/jpeg", true + case ".gif": + return "image/gif", true + case ".webp": + return "image/webp", true + case ".bmp": + return "image/bmp", true + default: + return "", false + } +} + +// storeUpload writes f to /uploads/. and returns the +// generated name. It writes to a dot-prefixed temp file first and renames it +// into place, so a partial upload is never visible under a servable name. The +// directory is created 0700 and the file inherits the temp file's 0600 mode. +func (s *Server) storeUpload(f io.Reader, ext string) (string, error) { + dir, err := s.uploadDir() + if err != nil { + return "", err + } + if err := os.MkdirAll(dir, 0o700); err != nil { + return "", fmt.Errorf("create upload dir: %w", err) + } + name, err := randomUploadName(ext) + if err != nil { + return "", err + } + tmp, err := os.CreateTemp(dir, ".upload-*") + if err != nil { + return "", fmt.Errorf("create upload temp: %w", err) + } + tmpName := tmp.Name() + if _, err := io.Copy(tmp, io.LimitReader(f, maxUploadBytes+1)); err != nil { + _ = tmp.Close() + _ = os.Remove(tmpName) + return "", fmt.Errorf("write upload: %w", err) + } + if err := tmp.Close(); err != nil { + _ = os.Remove(tmpName) + return "", fmt.Errorf("close upload: %w", err) + } + if err := os.Rename(tmpName, filepath.Join(dir, name)); err != nil { + _ = os.Remove(tmpName) + return "", fmt.Errorf("commit upload: %w", err) + } + return name, nil +} + +// uploadDir returns /uploads. The resolved data dir is injected by the +// composition root (Deps.DataDir, the same value resolveDataDir hands the DB and +// the runtime configs); the configured cfg.DataDir is the fallback so a Server +// built without the injection still resolves its uploads. +func (s *Server) uploadDir() (string, error) { + base := strings.TrimSpace(s.dataDir) + if base == "" { + base = strings.TrimSpace(s.cfg.DataDir) + } + if base == "" { + return "", fmt.Errorf("gateway: data directory not configured") + } + return filepath.Join(base, "uploads"), nil +} + +// randomUploadName returns a 128-bit random hex name with ext appended. It is +// the only filename component ever used: the client's original name is dropped. +func randomUploadName(ext string) (string, error) { + buf := make([]byte, uploadNameBytes) + if _, err := rand.Read(buf); err != nil { + return "", fmt.Errorf("random upload name: %w", err) + } + return hex.EncodeToString(buf) + ext, nil +} + +// validUploadName reports whether name is exactly a 32-lowercase-hex stem plus a +// whitelisted image extension. This is the path-traversal guard for the static +// route: separators, dots, dots-dots and any other shape are rejected before the +// filesystem is touched. +func validUploadName(name string) bool { + if name == "" || name != filepath.Base(name) || strings.ContainsAny(name, "/\\") { + return false + } + stem, ext, ok := strings.Cut(name, ".") + if !ok || len(stem) != uploadNameBytes*2 || ext == "" { + return false + } + for _, c := range stem { + if !isLowerHex(c) { + return false + } + } + _, ok = uploadMIMEForExt("." + ext) + return ok +} + +// isLowerHex reports whether c is an ASCII lowercase hex digit. +func isLowerHex(c rune) bool { + return (c >= '0' && c <= '9') || (c >= 'a' && c <= 'f') +} + +// uploadURL builds the absolute, same-origin URL for a stored upload. The scheme +// honours X-Forwarded-Proto (reverse proxy) and the authority is the request +// host, sanitized so a malformed Host header cannot smuggle path syntax into the +// returned URL; the configured listen address is the fallback. +func (s *Server) uploadURL(r *http.Request, name string) string { + scheme := "http" + if r.TLS != nil { + scheme = "https" + } + if proto := forwardedProto(r); proto != "" { + scheme = proto + } + host := sanitizeHost(r.Host) + if host == "" { + host = net.JoinHostPort(s.cfg.Server.Host, strconv.Itoa(s.cfg.Server.Port)) + } + return scheme + "://" + host + uploadFilePrefix + name +} + +// forwardedProto returns the first X-Forwarded-Proto value when it is http or +// https, else "". +func forwardedProto(r *http.Request) string { + v := r.Header.Get("X-Forwarded-Proto") + if v == "" { + return "" + } + v = strings.ToLower(strings.TrimSpace(strings.Split(v, ",")[0])) + if v == "http" || v == "https" { + return v + } + return "" +} + +// sanitizeHost rejects a host value containing path, userinfo or whitespace +// syntax; an empty result makes uploadURL fall back to the configured address. +func sanitizeHost(host string) string { + host = strings.TrimSpace(host) + if host == "" || strings.ContainsAny(host, "/\\@ \t") { + return "" + } + return host +} diff --git a/internal/services/gateway/upload_test.go b/internal/services/gateway/upload_test.go new file mode 100644 index 0000000..ddb6665 --- /dev/null +++ b/internal/services/gateway/upload_test.go @@ -0,0 +1,319 @@ +package gateway + +import ( + "bytes" + "context" + "encoding/json" + "io" + "mime/multipart" + "net/http" + "net/http/httptest" + "os" + "path/filepath" + "strconv" + "strings" + "testing" + "testing/fstest" + "time" + + "git.yeij.top/AskaEth/Cyrene_v2/internal/auth" +) + +// testPNG is the smallest byte string the stdlib sniffer recognizes as +// image/png (the 8-byte signature plus padding); the handler only sniffs the +// head, it never decodes the image. +var testPNG = append([]byte("\x89PNG\r\n\x1a\n"), make([]byte, 64)...) + +// uploadTestEnv builds a Server whose data dir is a fresh temp dir and returns +// the httptest server, the JWT service and the resolved data dir. +func uploadTestEnv(t *testing.T) (*httptest.Server, *auth.JWTService, string) { + t.Helper() + cfg := cfgDefault() + cfg.DataDir = t.TempDir() + _, ts, jwt := buildEnv(t, cfg, Deps{}) + return ts, jwt, cfg.DataDir +} + +// postUpload sends a multipart/form-data POST to /api/v1/upload. field is the +// part name (normally "file"); filename is the client-supplied name, which the +// handler must ignore. +func postUpload(t *testing.T, ts *httptest.Server, token, field, filename string, content []byte) (*http.Response, []byte) { + t.Helper() + var buf bytes.Buffer + mw := multipart.NewWriter(&buf) + part, err := mw.CreateFormFile(field, filename) + if err != nil { + t.Fatalf("create form file: %v", err) + } + if _, err := part.Write(content); err != nil { + t.Fatalf("write part: %v", err) + } + if err := mw.Close(); err != nil { + t.Fatalf("close writer: %v", err) + } + req, err := http.NewRequest(http.MethodPost, ts.URL+"/api/v1/upload", &buf) + if err != nil { + t.Fatalf("new request: %v", err) + } + req.Header.Set("Content-Type", mw.FormDataContentType()) + if token != "" { + req.Header.Set("Authorization", "Bearer "+token) + } + resp, err := ts.Client().Do(req) + if err != nil { + t.Fatalf("do request: %v", err) + } + defer func() { _ = resp.Body.Close() }() + body, err := io.ReadAll(resp.Body) + if err != nil { + t.Fatalf("read response: %v", err) + } + return resp, body +} + +// uploadData is the decoded upload response. +type uploadData struct { + URL string `json:"url"` + Size int64 `json:"size"` + MIME string `json:"mime"` + Name string `json:"name"` +} + +func decodeUploadData(t *testing.T, body []byte) uploadData { + t.Helper() + var out uploadData + if err := json.Unmarshal(decodeBody(t, body)["data"], &out); err != nil { + t.Fatalf("decode upload data: %v (body=%s)", err, body) + } + return out +} + +func TestUploadAndServeImage(t *testing.T) { + t.Parallel() + ts, jwt, dataDir := uploadTestEnv(t) + tok, err := jwt.Sign("acct", auth.RoleOwner, time.Hour) + if err != nil { + t.Fatalf("sign token: %v", err) + } + + resp, body := postUpload(t, ts, tok, "file", "../../evil.png", testPNG) + if resp.StatusCode != http.StatusCreated { + t.Fatalf("status = %d, want 201 (body=%s)", resp.StatusCode, body) + } + data := decodeUploadData(t, body) + if data.MIME != "image/png" { + t.Fatalf("mime = %q, want image/png", data.MIME) + } + if data.Size != int64(len(testPNG)) { + t.Fatalf("size = %d, want %d", data.Size, len(testPNG)) + } + if !strings.HasPrefix(data.URL, "http://") || !strings.Contains(data.URL, uploadFilePrefix) { + t.Fatalf("url = %q, want absolute %s URL", data.URL, uploadFilePrefix) + } + // The client's original filename must never reach disk or the response. + if strings.Contains(data.Name, "evil") || strings.ContainsAny(data.Name, "/\\") { + t.Fatalf("name = %q, want a server-generated hex name", data.Name) + } + if !validUploadName(data.Name) { + t.Fatalf("name = %q, not a valid storage name", data.Name) + } + if _, err := os.Stat(filepath.Join(dataDir, "uploads", data.Name)); err != nil { + t.Fatalf("stored file: %v", err) + } + + // The static route is a capability URL: no bearer token, and the bytes and + // caching/type headers must come back intact. + get, err := ts.Client().Get(data.URL) + if err != nil { + t.Fatalf("get upload: %v", err) + } + defer func() { _ = get.Body.Close() }() + if get.StatusCode != http.StatusOK { + t.Fatalf("get status = %d, want 200", get.StatusCode) + } + if ct := get.Header.Get("Content-Type"); ct != "image/png" { + t.Fatalf("content-type = %q, want image/png", ct) + } + if cl := get.Header.Get("Content-Length"); cl != strconv.Itoa(len(testPNG)) { + t.Fatalf("content-length = %q, want %d", cl, len(testPNG)) + } + if get.Header.Get("Cache-Control") == "" { + t.Fatal("cache-control is empty") + } + got, err := io.ReadAll(get.Body) + if err != nil { + t.Fatalf("read served body: %v", err) + } + if !bytes.Equal(got, testPNG) { + t.Fatalf("served body = %q, want the uploaded bytes", got) + } +} + +func TestUploadRejectsNonImage(t *testing.T) { + t.Parallel() + ts, jwt, _ := uploadTestEnv(t) + tok, _ := jwt.Sign("acct", auth.RoleOwner, time.Hour) + + resp, body := postUpload(t, ts, tok, "file", "notes.txt", []byte("just some plain text")) + if resp.StatusCode != http.StatusUnsupportedMediaType { + t.Fatalf("status = %d, want 415 (body=%s)", resp.StatusCode, body) + } + wantErrorShape(t, body) +} + +func TestUploadRejectsOversize(t *testing.T) { + t.Parallel() + ts, jwt, _ := uploadTestEnv(t) + tok, _ := jwt.Sign("acct", auth.RoleOwner, time.Hour) + + big := make([]byte, maxUploadBytes+1) + copy(big, testPNG) + resp, body := postUpload(t, ts, tok, "file", "big.png", big) + if resp.StatusCode != http.StatusRequestEntityTooLarge { + t.Fatalf("status = %d, want 413 (body=%s)", resp.StatusCode, body) + } + wantErrorShape(t, body) +} + +func TestUploadMissingFile(t *testing.T) { + t.Parallel() + ts, jwt, _ := uploadTestEnv(t) + tok, _ := jwt.Sign("acct", auth.RoleOwner, time.Hour) + + resp, body := postUpload(t, ts, tok, "wrongfield", "x.png", testPNG) + if resp.StatusCode != http.StatusBadRequest { + t.Fatalf("status = %d, want 400 (body=%s)", resp.StatusCode, body) + } + wantErrorShape(t, body) +} + +func TestUploadRequiresAuth(t *testing.T) { + t.Parallel() + ts, jwt, _ := uploadTestEnv(t) + viewer, _ := jwt.Sign("acct", auth.RoleViewer, time.Hour) + + if resp, body := postUpload(t, ts, "", "file", "x.png", testPNG); resp.StatusCode != http.StatusUnauthorized { + t.Fatalf("no token status = %d, want 401 (body=%s)", resp.StatusCode, body) + } + if resp, body := postUpload(t, ts, viewer, "file", "x.png", testPNG); resp.StatusCode != http.StatusForbidden { + t.Fatalf("viewer status = %d, want 403 (body=%s)", resp.StatusCode, body) + } +} + +func TestUploadAudits(t *testing.T) { + t.Parallel() + cfg := cfgDefault() + cfg.DataDir = t.TempDir() + store := newMemoryAuditStore() + _, ts, jwt := buildEnv(t, cfg, Deps{AuditStore: store}) + tok, _ := jwt.Sign("acct", auth.RoleOwner, time.Hour) + + if resp, body := postUpload(t, ts, tok, "file", "x.png", testPNG); resp.StatusCode != http.StatusCreated { + t.Fatalf("status = %d, want 201 (body=%s)", resp.StatusCode, body) + } + records, err := store.List(context.Background(), 0) + if err != nil { + t.Fatalf("list audit: %v", err) + } + found := false + for _, rec := range records { + if rec.What == "upload.image" && rec.Result == "ok" { + found = true + } + } + if !found { + t.Fatalf("no upload.image audit record in %+v", records) + } +} + +func TestUploadFileServedThroughSPA(t *testing.T) { + t.Parallel() + cfg := cfgDefault() + cfg.DataDir = t.TempDir() + // A non-nil frontend enables the SPA wrapper in Handler; the /api/v1/files/ + // route must still reach the API mux instead of falling back to index.html. + frontend := fstest.MapFS{"index.html": &fstest.MapFile{Data: []byte("spa")}} + _, ts, jwt := buildEnv(t, cfg, Deps{Frontend: frontend}) + tok, _ := jwt.Sign("acct", auth.RoleOwner, time.Hour) + + resp, body := postUpload(t, ts, tok, "file", "x.png", testPNG) + if resp.StatusCode != http.StatusCreated { + t.Fatalf("status = %d, want 201 (body=%s)", resp.StatusCode, body) + } + data := decodeUploadData(t, body) + get, err := ts.Client().Get(data.URL) + if err != nil { + t.Fatalf("get upload: %v", err) + } + defer func() { _ = get.Body.Close() }() + if get.StatusCode != http.StatusOK { + t.Fatalf("get status = %d, want 200 (SPA must not swallow the route)", get.StatusCode) + } + got, _ := io.ReadAll(get.Body) + if !bytes.Equal(got, testPNG) { + t.Fatalf("served body = %q, want the uploaded bytes", got) + } +} + +func TestUploadFileRejectsTraversal(t *testing.T) { + t.Parallel() + cfg := cfgDefault() + cfg.DataDir = t.TempDir() + // A non-nil frontend reproduces production routing: without the explicit + // guard, ServeMux would clean the raw `..` path and redirect to a path the SPA + // then answers with index.html. + frontend := fstest.MapFS{"index.html": &fstest.MapFile{Data: []byte("spa")}} + _, ts, _ := buildEnv(t, cfg, Deps{Frontend: frontend}) + // A file that really exists directly under the data dir (not in uploads/) + // must not be reachable through the static route. + if err := os.WriteFile(filepath.Join(cfg.DataDir, "secret.png"), testPNG, 0o600); err != nil { + t.Fatalf("write decoy: %v", err) + } + paths := []string{ + "/api/v1/files/", + "/api/v1/files/..%2f..%2f..%2fetc%2fpasswd", + "/api/v1/files/../../../../etc/passwd", + "/api/v1/files/%2e%2e%2fsecret.png", + "/api/v1/files/secret.png", + "/api/v1/files/00000000000000000000000000000000.png", + } + for _, p := range paths { + resp, err := ts.Client().Get(ts.URL + p) + if err != nil { + t.Fatalf("get %s: %v", p, err) + } + body, _ := io.ReadAll(resp.Body) + _ = resp.Body.Close() + if resp.StatusCode != http.StatusNotFound { + t.Fatalf("get %s status = %d, want 404", p, resp.StatusCode) + } + if bytes.Contains(body, []byte("root:")) { + t.Fatalf("get %s leaked /etc/passwd", p) + } + } +} + +func TestValidUploadName(t *testing.T) { + t.Parallel() + cases := []struct { + name string + want bool + }{ + {name: "0123456789abcdef0123456789abcdef.png", want: true}, + {name: "0123456789abcdef0123456789abcdef.webp", want: true}, + {name: "", want: false}, + {name: ".", want: false}, + {name: "..", want: false}, + {name: "abcd.png", want: false}, + {name: "0123456789abcdef0123456789abcdef.exe", want: false}, + {name: "0123456789abcdef0123456789abcdeZ.png", want: false}, + {name: "0123456789abcdef0123456789abcdef", want: false}, + {name: "../0123456789abcdef0123456789abcdef.png", want: false}, + {name: "0123456789abcdef0123456789abcdef.png/../x.png", want: false}, + } + for _, tc := range cases { + if got := validUploadName(tc.name); got != tc.want { + t.Errorf("validUploadName(%q) = %v, want %v", tc.name, got, tc.want) + } + } +}