feat(gateway): 图片上传端点(B1) — POST /v1/upload + GET /v1/files/{name}
- upload.go: multipart上传(owner/admin+审计)→ {url,size,mime,name}; MaxBytesReader 5MB + DetectContentType嗅探 + 白名单raster(拒SVG/HTML) + crypto/rand 128-bit随机名 + 0700目录
- 静态服务: 同端口 GET /v1/files/{name}, 能力URL, 路径穿越全形态拒绝(.. / %2f / %2e%2e), nosniff, Range
- errors: ErrPayloadTooLarge(413)/ErrUnsupportedMediaType(415)
- server.go 路由+Deps.DataDir; wire.go resolveDataDir 注入
- 8测试+真实二进制E2E, build/vet/test绿+交叉编译
- Docs/api-client v0.24 + api-reference §0.3 同步
This commit is contained in:
1 parent
3fc10fb86f
commit
7ecb43bd5d
7 files changed
+723
-3
No files matched your search
@@ -33,6 +33,8 @@
|
||||
| 10 | `/api/v1/audio/transcribe` | POST | JWT(owner) | ✅ | **语音→文本**:body `{audio:"<base64>",format?,deviceId?}` → `{data:{text, provider?}}`(走后端 STT,百炼 `qwen-audio-3.0-asr-flash`;**当结果为空文本/silent 降级时响应体带 `provider`**(如 `"silent"`,非空文本省略),客户端据此提示「没识别到声音」) |
|
||||
| 11 | `/api/v1/audio/speech` | POST | JWT(owner) | ✅ | **文本→音频**:body `{text,voice?,format?,instruction?}` → `200` 音频字节(`Content-Type: audio/<format>`,非 JSON envelope,走后端 TTS,百炼 `cosyvoice-v3-flash`,复刻音色可带 `instruction` 情绪指令;**静默降级空音频时仍 `500 tts_empty`,并附 `X-Voice-Provider: silent` 响应头**,客户端据此提示「没生成声音」) |
|
||||
| 12 | `/api/v1/audio/voice-clone` | POST | JWT(owner) | ✅ | **声音复刻建音色**:body `{url:"<公网参考音频URL>",prefix?}` → `{data:{voiceId}}`(走百炼 `voice-enrollment` create_voice) |
|
||||
| 13 | `/api/v1/upload` | POST | JWT(owner/admin) | ✅ | **图片上传**(B1):multipart/form-data,字段 `file`(raster image,5MB 上限)→ `201 {data:{url,size,mime,name}}`;`url` 为**同源访问地址**(`<host>/api/v1/files/<name>`,可直接作 `chat.image`)。校验:`MaxBytesReader` 防超大 → `DetectContentType` 嗅探(**不信客户端 Content-Type**)→ 白名单 png/jpeg/gif/webp/bmp(**拒 SVG/HTML**);文件名用 **crypto/rand 128-bit 随机名**(原始文件名丢弃),存 `<dataDir>/uploads/`(0700)。错误码:缺文件 `400 bad_request`、超限 `413 payload_too_large`、非白名单 `415 unsupported_media_type`、未授权 `401/403` |
|
||||
| 14 | `/api/v1/files/{name}` | GET | 无需(能力URL) | ✅ | **上传文件访问**(B1):从 `<dataDir>/uploads/` 读取,同名端口服务。`name` 仅接受「32 位小写 hex + 白名单扩展名」;**路径穿越全形态拒绝**(`..`/`%2f`/`%2e%2e` 等)。返回 `Content-Type`/`Content-Length`/`Cache-Control: private,max-age=86400,immutable`/`X-Content-Type-Options: nosniff`(支持 Range)。**设计取舍**:GET 不鉴权——该 URL 是**128-bit 随机能力地址**(`chat.image`/`<img>` 无法带 Bearer),POST 仍 owner/admin+审计;任何持有该 URL 者都能取文件,勿外泄到他人 |
|
||||
|
||||
> **语音 REST 源(`/api/v1/audio/*`,均 owner-only + 审计)**:原生/移动端用它上传/获取音频,**绕开 WS 4KB 帧限**。
|
||||
> - `POST /api/v1/audio/transcribe`:body `{audio:"<base64>", format?:"wav", deviceId?}` → `200 {data:{text}}`。`audio` 为 base64(或 data: URI);`format` 为编码(`wav`/`mp3`/`ogg`,缺省 `wav`);空 audio 回 `400`;未注入语音服务 `500 voice_unavailable`、识别失败 `500 stt_failed`。
|
||||
@@ -211,6 +213,7 @@
|
||||
- 语音:上行 `voice_input`(base64 音频或已转写 text),下行 `voice_output`(合成音频 base64 + text)。
|
||||
|
||||
## 5. 变更记录
|
||||
- `v0.24`(**B1 图片上传端点**):`POST /api/v1/upload`(**新增**,multipart `file`,owner/admin+审计 → `201 {data:{url,size,mime,name}}`)+ `GET /api/v1/files/{name}`(**新增**,同端口静态服务)。安全:`MaxBytesReader`(5MB)、`DetectContentType` 嗅探(不信客户端 Content-Type)、白名单 raster image(拒 SVG/HTML)、`crypto/rand` 128-bit 随机名(丢弃原始文件名)、非可执行上传目录(0700)、`nosniff`;路径穿越全形态拒绝(含编码 `%2f`/`%2e%2e`)。错误码 `400/413/415`(`internal/errors` 增 `ErrPayloadTooLarge`/`ErrUnsupportedMediaType`)。**设计取舍**:GET 不鉴权(能力 URL,`chat.image`/`<img>` 无法带 Bearer),POST 仍 owner/admin。`url` 为绝对地址(`X-Forwarded-Proto`/TLS + `r.Host`,因 `ParseImageRef` 只认 `http(s)://`)。`Deps.DataDir` 由 wire 注入(`resolveDataDir`)。
|
||||
- `v0.23`(**C2 语音 silent 降级信号**):三处统一用 **`provider`** 只在「结果为空」时出现(非空/成功时才省略)——`POST /api/v1/audio/transcribe` 空文本响应体带 `{text:"", provider:"silent"}`;WS `voice_output` 空音频 payload 带 `provider`;`POST /api/v1/audio/speech` 空音频仍 `500 tts_empty` 并加 `X-Voice-Provider: silent` 响应头(CORS `Access-Control-Expose-Headers` 已放开)。`voice.Chain/Service` 新增 `STTWithProvider/TTSWithProvider`(返回真实档位 cloud/local/edge-tts/espeak/silent),网关用可选接口 `voiceProviderReporter` 透传,自定义实现回退 `silent`。**含 `X-Voice-Provider`(`apperr` 包装、`providerSignal`);`DASHSCOPE_API_KEY` 无/降级时信号为 `silent`。**(`go test -race` 在本机受内核 TSAN VMA 限制失败,非本代码问题。)
|
||||
- `v0.22`(**B3 会话列表 lastMessage 预览**):`GET /api/v1/sessions` 与 `/api/v1/admin/sessions` 每个会话新增可选 **`lastMessage`** `{content,role,createdAt}`(content 折叠空白截前 60 rune + `…`;空会话省略),**单条 SQL**(LEFT JOIN 取每会话 `ORDER BY created_at DESC,rowid DESC LIMIT 1`,无 N+1,走 `idx_chat_msgs_session_created` 覆盖索引)。`unreadCount` 需已读游标设计,后置。
|
||||
- `v0.21`(**A4 会话 title**):`PATCH /api/v1/sessions/{id}`(**新增**,body `{title}` ≤80 rune → `{data:{id,title}}`;空/超长 400、不存在 404、主会话可改名);会话**首条用户消息到达时自动命名**(`history.append` 前 `ensureTitle`:折叠空白取前 20 rune,纯空白回落 `主会话`/`次级会话`),仅首次命名、不覆盖。`HistoryStore` 新增 `RenameSession`;隔离会话(`sess-xxx`)不在 `chat_sessions` 有行、不自动命名(其本不在 `/sessions` 列表)。**边界**:前端 `createSubSession(title?)` 的 POST body 含 `title` 但后端暂忽略(会被首条消息自动命名覆盖),未处理。
|
||||
|
||||
@@ -38,9 +38,9 @@
|
||||
|
||||
### 0.3 上传 / 媒体(重要确认)
|
||||
|
||||
- **无 `/upload` 端点**。图片走 `POST /api/v1/chat` 或 WS `chat` 的 `image` 字段(**base64 data: URI** 或 http(s) URL,入视觉真推理,见 §6 流式与直连)。
|
||||
- **无 REST 语音端点**。语音**只走 WS**:上行 `voice_input`(base64 音频或已转写 text),下行 `voice_output`(合成音频 base64)。见 §6.5。
|
||||
- 管理端导入知识文档(RAG)也走 JSON body 文本(`POST /api/v1/admin/rag`),非文件上传。
|
||||
- **`POST /api/v1/upload`**(B1)上传图片 → 返回**同源 URL**(`/api/v1/files/<name>`),可直接作 `chat.image`;也可继续用 **base64 data: URI** 或 http(s) URL 走 `POST /api/v1/chat` / WS `chat` 的 `image` 字段(入视觉真推理,见 §6 流式与直连)。
|
||||
- **REST 语音端点已存在**:`POST /api/v1/audio/transcribe`(STT)、`POST /api/v1/audio/speech`(TTS)、`POST /api/v1/audio/voice-clone`(声音复刻),均 owner-only + 审计,绕开 WS 4KB 帧限。语音也可走 WS `voice_input`(上行)/`voice_output`(下行)。
|
||||
- 管理端导入知识文档(RAG)走 JSON body 文本(`POST /api/v1/admin/rag`),非文件上传。
|
||||
|
||||
---
|
||||
|
||||
|
||||
@@ -684,6 +684,7 @@ func (a *app) wireFrontend() error {
|
||||
Frontend: resolveFrontendFS(a.log),
|
||||
Webhooks: a.channelBridge.WebhookHandlers(),
|
||||
DatabasePath: a.dbPath,
|
||||
DataDir: resolveDataDir(a.cfg),
|
||||
})
|
||||
if err != nil {
|
||||
return fmt.Errorf("gateway: %w", err)
|
||||
|
||||
@@ -96,6 +96,13 @@ var (
|
||||
// ErrBadRequest means the request is malformed (400).
|
||||
ErrBadRequest = newWithHTTP("bad_request", "bad request", http.StatusBadRequest)
|
||||
|
||||
// ErrPayloadTooLarge means the request body exceeds the allowed size (413).
|
||||
ErrPayloadTooLarge = newWithHTTP("payload_too_large", "payload too large", http.StatusRequestEntityTooLarge)
|
||||
|
||||
// ErrUnsupportedMediaType means the request carries a media type the endpoint
|
||||
// does not accept (415).
|
||||
ErrUnsupportedMediaType = newWithHTTP("unsupported_media_type", "unsupported media type", http.StatusUnsupportedMediaType)
|
||||
|
||||
// ErrRateLimited means the caller exceeded a rate limit (429).
|
||||
ErrRateLimited = newWithHTTP("rate_limited", "rate limited", http.StatusTooManyRequests)
|
||||
|
||||
|
||||
@@ -276,6 +276,11 @@ type Deps struct {
|
||||
// It backs the admin backup/restore endpoints. Optional; when empty those
|
||||
// endpoints answer honestly with a clear not-configured error.
|
||||
DatabasePath string
|
||||
// DataDir is the resolved data directory (the value resolveDataDir returns,
|
||||
// which may differ from cfg.DataDir when the OS default applies). It backs
|
||||
// the image-upload store (<DataDir>/uploads) and its static route. Optional;
|
||||
// when empty the upload handlers fall back to cfg.DataDir.
|
||||
DataDir string
|
||||
// Streaming marks the injected Loop as streaming-capable (its OnStream is
|
||||
// wired to publish ws.stream_chunk). When set the hub skips its own single
|
||||
// final stream_chunk emission, since deltas already stream through OnStream.
|
||||
@@ -316,6 +321,10 @@ type Server struct {
|
||||
streaming bool
|
||||
frontend fs.FS
|
||||
dbPath string
|
||||
// dataDir is the resolved data directory backing the upload store
|
||||
// (<dataDir>/uploads). Injected by the composition root; cfg.DataDir is the
|
||||
// fallback. See Deps.DataDir.
|
||||
dataDir string
|
||||
// llm is the hot-swappable provider stack (models.json). When nil the
|
||||
// admin /llm endpoints fall back to the env config. It is optional so tests
|
||||
// that do not wire it keep working.
|
||||
@@ -364,6 +373,7 @@ func New(cfg *config.Config, deps Deps) (*Server, error) {
|
||||
ownerRegistrar: deps.OwnerRegistrar,
|
||||
webhooks: deps.Webhooks,
|
||||
dbPath: deps.DatabasePath,
|
||||
dataDir: deps.DataDir,
|
||||
log: sourcedLog(deps.Log),
|
||||
}
|
||||
s.hub = newHub(s)
|
||||
@@ -395,6 +405,14 @@ func (s *Server) Handler() http.Handler {
|
||||
mux.Handle("POST /api/v1/audio/speech", s.auth.Authorize(auth.RoleOwner)(http.HandlerFunc(s.handleSpeech)))
|
||||
mux.Handle("POST /api/v1/audio/voice-clone", s.auth.Authorize(auth.RoleOwner)(http.HandlerFunc(s.handleVoiceClone)))
|
||||
|
||||
// Image upload (B1): a multipart POST that stores a raster image under the
|
||||
// resolved data dir and returns a same-origin URL usable directly as
|
||||
// chat.image. Upload is owner/admin + audited; the static route below is a
|
||||
// capability URL (unguessable name) because chat.image / <img> cannot carry a
|
||||
// bearer token. Both ride the one gateway port (single-binary red line).
|
||||
mux.Handle("POST /api/v1/upload", s.auth.Authorize(auth.RoleOwner, auth.RoleAdmin)(http.HandlerFunc(s.handleUpload)))
|
||||
mux.HandleFunc("GET /api/v1/files/{name}", s.handleUploadFile)
|
||||
|
||||
// Session/model reads and writes are the owner's private history; owner/admin only
|
||||
// (chat uses the same roles). Session metadata in GET /sessions stays readAuth for
|
||||
// any authenticated role, but the message content (and thinking) is owner/admin-only
|
||||
@@ -480,6 +498,7 @@ func (s *Server) Handler() http.Handler {
|
||||
}
|
||||
|
||||
var h http.Handler = mux
|
||||
h = guardUploadPaths(h)
|
||||
if s.frontend != nil {
|
||||
h = s.spa(h)
|
||||
}
|
||||
|
||||
@@ -0,0 +1,371 @@
|
||||
package gateway
|
||||
|
||||
import (
|
||||
"crypto/rand"
|
||||
"encoding/hex"
|
||||
"errors"
|
||||
"fmt"
|
||||
"io"
|
||||
"mime/multipart"
|
||||
"net"
|
||||
"net/http"
|
||||
"os"
|
||||
"path/filepath"
|
||||
"strconv"
|
||||
"strings"
|
||||
|
||||
apperr "git.yeij.top/AskaEth/Cyrene_v2/internal/errors"
|
||||
)
|
||||
|
||||
// maxUploadBytes caps a single uploaded image at 5 MiB. The multipart envelope
|
||||
// (boundaries, part headers) gets a little headroom on top of that; the decoded
|
||||
// file itself is checked against this exact limit.
|
||||
const (
|
||||
maxUploadBytes = 5 << 20
|
||||
uploadMultipartSlack = 1 << 20
|
||||
uploadNameBytes = 16
|
||||
uploadSniffBytes = 512
|
||||
)
|
||||
|
||||
// uploadFilePrefix is the same-origin static route that serves stored uploads
|
||||
// (registered in Handler). It lives under /api/v1/ so the gateway mux — not the
|
||||
// SPA fallback — owns it, and so it is part of the versioned client contract.
|
||||
const uploadFilePrefix = "/api/v1/files/"
|
||||
|
||||
// uploadResponse is the POST /api/v1/upload success payload. URL is an absolute
|
||||
// http(s) URL built from the request authority so it can be handed straight to
|
||||
// chat.image: the vision layer classifies only http(s):// as a URL and would
|
||||
// otherwise treat a bare path as base64. Name is the server-generated storage
|
||||
// name; the client's original filename is never echoed back or persisted.
|
||||
type uploadResponse struct {
|
||||
URL string `json:"url"`
|
||||
Size int64 `json:"size"`
|
||||
MIME string `json:"mime"`
|
||||
Name string `json:"name"`
|
||||
}
|
||||
|
||||
// handleUpload accepts a multipart/form-data request with a single `file` part,
|
||||
// validates it is a raster image within the size limit, stores it under
|
||||
// <dataDir>/uploads under a cryptographically random name, audits the write and
|
||||
// returns the URL to attach as chat.image. It is owner/admin only (registered in
|
||||
// Handler). The client's original filename is deliberately ignored: it is
|
||||
// attacker-controlled and could carry a path separator or overwrite a file.
|
||||
func (s *Server) handleUpload(w http.ResponseWriter, r *http.Request) {
|
||||
r.Body = http.MaxBytesReader(w, r.Body, maxUploadBytes+uploadMultipartSlack)
|
||||
if err := r.ParseMultipartForm(1 << 20); err != nil {
|
||||
s.auditClaims(r, "upload.image", "error")
|
||||
writeUploadBodyError(w, err)
|
||||
return
|
||||
}
|
||||
file, header, err := r.FormFile("file")
|
||||
if err != nil {
|
||||
s.auditClaims(r, "upload.image", "error")
|
||||
writeError(w, apperr.Wrap(apperr.ErrBadRequest, "bad_request", "file is required"))
|
||||
return
|
||||
}
|
||||
defer func() { _ = file.Close() }()
|
||||
|
||||
if header.Size <= 0 {
|
||||
s.auditClaims(r, "upload.image", "error")
|
||||
writeError(w, apperr.Wrap(apperr.ErrBadRequest, "bad_request", "file is empty"))
|
||||
return
|
||||
}
|
||||
if header.Size > maxUploadBytes {
|
||||
s.auditClaims(r, "upload.image", "error")
|
||||
writeError(w, apperr.Wrap(apperr.ErrPayloadTooLarge, "payload_too_large", "file exceeds the 5MB limit"))
|
||||
return
|
||||
}
|
||||
mime, err := sniffImageMIME(file)
|
||||
if err != nil {
|
||||
s.auditClaims(r, "upload.image", "error")
|
||||
s.log.Error("upload sniff", "err", err)
|
||||
writeError(w, apperr.Wrap(apperr.ErrInternal, "upload_failed", "failed to read upload"))
|
||||
return
|
||||
}
|
||||
ext, ok := uploadExtForMIME(mime)
|
||||
if !ok {
|
||||
s.auditClaims(r, "upload.image", "error")
|
||||
writeError(w, apperr.Wrap(apperr.ErrUnsupportedMediaType, "unsupported_media_type", "only png/jpeg/gif/webp/bmp images are accepted"))
|
||||
return
|
||||
}
|
||||
name, err := s.storeUpload(file, ext)
|
||||
if err != nil {
|
||||
s.auditClaims(r, "upload.image", "error")
|
||||
s.log.Error("upload store", "err", err)
|
||||
writeError(w, apperr.Wrap(apperr.ErrInternal, "upload_failed", "failed to store upload"))
|
||||
return
|
||||
}
|
||||
s.auditClaims(r, "upload.image", "ok")
|
||||
writeJSON(w, http.StatusCreated, uploadResponse{
|
||||
URL: s.uploadURL(r, name),
|
||||
Size: header.Size,
|
||||
MIME: mime,
|
||||
Name: name,
|
||||
})
|
||||
}
|
||||
|
||||
// guardUploadPaths rejects a static-upload request whose decoded path contains a
|
||||
// parent-directory segment, before ServeMux can clean it and redirect the client
|
||||
// to a non-upload path (which the SPA would then answer with index.html). The
|
||||
// {name} route only ever matches one segment, so a `..` can only be an escape
|
||||
// attempt; answering 404 here keeps every traversal shape uniformly rejected.
|
||||
// Non-upload paths pass through untouched.
|
||||
func guardUploadPaths(next http.Handler) http.Handler {
|
||||
return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
|
||||
if strings.HasPrefix(r.URL.Path, uploadFilePrefix) && hasDotDotSegment(r.URL.Path) {
|
||||
writeError(w, apperr.Wrap(apperr.ErrNotFound, "not_found", "file not found"))
|
||||
return
|
||||
}
|
||||
next.ServeHTTP(w, r)
|
||||
})
|
||||
}
|
||||
|
||||
// hasDotDotSegment reports whether p has a literal ".." path segment.
|
||||
func hasDotDotSegment(p string) bool {
|
||||
for _, seg := range strings.Split(p, "/") {
|
||||
if seg == ".." {
|
||||
return true
|
||||
}
|
||||
}
|
||||
return false
|
||||
}
|
||||
|
||||
// handleUploadFile serves a stored upload from <dataDir>/uploads. It is
|
||||
// unauthenticated by design: the returned URL is handed to chat.image and to
|
||||
// <img> tags, neither of which can attach a bearer token. The random 128-bit
|
||||
// name is the capability; the handler accepts only a strict hex+known-extension
|
||||
// name and never lets a request path escape the uploads directory.
|
||||
func (s *Server) handleUploadFile(w http.ResponseWriter, r *http.Request) {
|
||||
name := r.PathValue("name")
|
||||
if !validUploadName(name) {
|
||||
writeError(w, apperr.Wrap(apperr.ErrNotFound, "not_found", "file not found"))
|
||||
return
|
||||
}
|
||||
dir, err := s.uploadDir()
|
||||
if err != nil {
|
||||
writeError(w, apperr.Wrap(apperr.ErrNotFound, "not_found", "file not found"))
|
||||
return
|
||||
}
|
||||
path := filepath.Join(dir, name)
|
||||
if filepath.Dir(path) != filepath.Clean(dir) {
|
||||
writeError(w, apperr.Wrap(apperr.ErrNotFound, "not_found", "file not found"))
|
||||
return
|
||||
}
|
||||
mime, ok := uploadMIMEForExt(filepath.Ext(name))
|
||||
if !ok {
|
||||
writeError(w, apperr.Wrap(apperr.ErrNotFound, "not_found", "file not found"))
|
||||
return
|
||||
}
|
||||
f, err := os.Open(path)
|
||||
if err != nil {
|
||||
writeError(w, apperr.Wrap(apperr.ErrNotFound, "not_found", "file not found"))
|
||||
return
|
||||
}
|
||||
defer func() { _ = f.Close() }()
|
||||
info, err := f.Stat()
|
||||
if err != nil || info.IsDir() {
|
||||
writeError(w, apperr.Wrap(apperr.ErrNotFound, "not_found", "file not found"))
|
||||
return
|
||||
}
|
||||
w.Header().Set("Content-Type", mime)
|
||||
w.Header().Set("Cache-Control", "private, max-age=86400, immutable")
|
||||
w.Header().Set("X-Content-Type-Options", "nosniff")
|
||||
http.ServeContent(w, r, name, info.ModTime(), f)
|
||||
}
|
||||
|
||||
// writeUploadBodyError maps a multipart parse failure to its wire error: an
|
||||
// over-limit body is 413, anything else (not multipart, malformed part) is 400.
|
||||
func writeUploadBodyError(w http.ResponseWriter, err error) {
|
||||
var maxErr *http.MaxBytesError
|
||||
if errors.As(err, &maxErr) {
|
||||
writeError(w, apperr.Wrap(apperr.ErrPayloadTooLarge, "payload_too_large", "file exceeds the 5MB limit"))
|
||||
return
|
||||
}
|
||||
writeError(w, apperr.Wrap(apperr.ErrBadRequest, "bad_request", "invalid multipart body"))
|
||||
}
|
||||
|
||||
// sniffImageMIME detects the content type of the upload head with the stdlib
|
||||
// sniffer (never the client-supplied Content-Type header, which is untrusted)
|
||||
// and rewinds the part so the caller can copy it in full.
|
||||
func sniffImageMIME(f multipart.File) (string, error) {
|
||||
buf := make([]byte, uploadSniffBytes)
|
||||
n, err := io.ReadFull(f, buf)
|
||||
if err != nil && !errors.Is(err, io.EOF) && !errors.Is(err, io.ErrUnexpectedEOF) {
|
||||
return "", fmt.Errorf("read upload head: %w", err)
|
||||
}
|
||||
if _, err := f.Seek(0, io.SeekStart); err != nil {
|
||||
return "", fmt.Errorf("rewind upload: %w", err)
|
||||
}
|
||||
return http.DetectContentType(buf[:n]), nil
|
||||
}
|
||||
|
||||
// uploadExtForMIME maps a sniffed image MIME to the extension used on disk. The
|
||||
// whitelist is deliberately raster-only: SVG (and HTML) would be served inline
|
||||
// and could execute script, so they are not accepted.
|
||||
func uploadExtForMIME(mime string) (string, bool) {
|
||||
switch mime {
|
||||
case "image/png":
|
||||
return ".png", true
|
||||
case "image/jpeg":
|
||||
return ".jpg", true
|
||||
case "image/gif":
|
||||
return ".gif", true
|
||||
case "image/webp":
|
||||
return ".webp", true
|
||||
case "image/bmp":
|
||||
return ".bmp", true
|
||||
default:
|
||||
return "", false
|
||||
}
|
||||
}
|
||||
|
||||
// uploadMIMEForExt returns the Content-Type for a stored upload extension. It is
|
||||
// the read-side twin of uploadExtForMIME: only whitelisted extensions resolve,
|
||||
// which is what makes validUploadName strict.
|
||||
func uploadMIMEForExt(ext string) (string, bool) {
|
||||
switch strings.ToLower(ext) {
|
||||
case ".png":
|
||||
return "image/png", true
|
||||
case ".jpg", ".jpeg":
|
||||
return "image/jpeg", true
|
||||
case ".gif":
|
||||
return "image/gif", true
|
||||
case ".webp":
|
||||
return "image/webp", true
|
||||
case ".bmp":
|
||||
return "image/bmp", true
|
||||
default:
|
||||
return "", false
|
||||
}
|
||||
}
|
||||
|
||||
// storeUpload writes f to <dataDir>/uploads/<random>.<ext> and returns the
|
||||
// generated name. It writes to a dot-prefixed temp file first and renames it
|
||||
// into place, so a partial upload is never visible under a servable name. The
|
||||
// directory is created 0700 and the file inherits the temp file's 0600 mode.
|
||||
func (s *Server) storeUpload(f io.Reader, ext string) (string, error) {
|
||||
dir, err := s.uploadDir()
|
||||
if err != nil {
|
||||
return "", err
|
||||
}
|
||||
if err := os.MkdirAll(dir, 0o700); err != nil {
|
||||
return "", fmt.Errorf("create upload dir: %w", err)
|
||||
}
|
||||
name, err := randomUploadName(ext)
|
||||
if err != nil {
|
||||
return "", err
|
||||
}
|
||||
tmp, err := os.CreateTemp(dir, ".upload-*")
|
||||
if err != nil {
|
||||
return "", fmt.Errorf("create upload temp: %w", err)
|
||||
}
|
||||
tmpName := tmp.Name()
|
||||
if _, err := io.Copy(tmp, io.LimitReader(f, maxUploadBytes+1)); err != nil {
|
||||
_ = tmp.Close()
|
||||
_ = os.Remove(tmpName)
|
||||
return "", fmt.Errorf("write upload: %w", err)
|
||||
}
|
||||
if err := tmp.Close(); err != nil {
|
||||
_ = os.Remove(tmpName)
|
||||
return "", fmt.Errorf("close upload: %w", err)
|
||||
}
|
||||
if err := os.Rename(tmpName, filepath.Join(dir, name)); err != nil {
|
||||
_ = os.Remove(tmpName)
|
||||
return "", fmt.Errorf("commit upload: %w", err)
|
||||
}
|
||||
return name, nil
|
||||
}
|
||||
|
||||
// uploadDir returns <dataDir>/uploads. The resolved data dir is injected by the
|
||||
// composition root (Deps.DataDir, the same value resolveDataDir hands the DB and
|
||||
// the runtime configs); the configured cfg.DataDir is the fallback so a Server
|
||||
// built without the injection still resolves its uploads.
|
||||
func (s *Server) uploadDir() (string, error) {
|
||||
base := strings.TrimSpace(s.dataDir)
|
||||
if base == "" {
|
||||
base = strings.TrimSpace(s.cfg.DataDir)
|
||||
}
|
||||
if base == "" {
|
||||
return "", fmt.Errorf("gateway: data directory not configured")
|
||||
}
|
||||
return filepath.Join(base, "uploads"), nil
|
||||
}
|
||||
|
||||
// randomUploadName returns a 128-bit random hex name with ext appended. It is
|
||||
// the only filename component ever used: the client's original name is dropped.
|
||||
func randomUploadName(ext string) (string, error) {
|
||||
buf := make([]byte, uploadNameBytes)
|
||||
if _, err := rand.Read(buf); err != nil {
|
||||
return "", fmt.Errorf("random upload name: %w", err)
|
||||
}
|
||||
return hex.EncodeToString(buf) + ext, nil
|
||||
}
|
||||
|
||||
// validUploadName reports whether name is exactly a 32-lowercase-hex stem plus a
|
||||
// whitelisted image extension. This is the path-traversal guard for the static
|
||||
// route: separators, dots, dots-dots and any other shape are rejected before the
|
||||
// filesystem is touched.
|
||||
func validUploadName(name string) bool {
|
||||
if name == "" || name != filepath.Base(name) || strings.ContainsAny(name, "/\\") {
|
||||
return false
|
||||
}
|
||||
stem, ext, ok := strings.Cut(name, ".")
|
||||
if !ok || len(stem) != uploadNameBytes*2 || ext == "" {
|
||||
return false
|
||||
}
|
||||
for _, c := range stem {
|
||||
if !isLowerHex(c) {
|
||||
return false
|
||||
}
|
||||
}
|
||||
_, ok = uploadMIMEForExt("." + ext)
|
||||
return ok
|
||||
}
|
||||
|
||||
// isLowerHex reports whether c is an ASCII lowercase hex digit.
|
||||
func isLowerHex(c rune) bool {
|
||||
return (c >= '0' && c <= '9') || (c >= 'a' && c <= 'f')
|
||||
}
|
||||
|
||||
// uploadURL builds the absolute, same-origin URL for a stored upload. The scheme
|
||||
// honours X-Forwarded-Proto (reverse proxy) and the authority is the request
|
||||
// host, sanitized so a malformed Host header cannot smuggle path syntax into the
|
||||
// returned URL; the configured listen address is the fallback.
|
||||
func (s *Server) uploadURL(r *http.Request, name string) string {
|
||||
scheme := "http"
|
||||
if r.TLS != nil {
|
||||
scheme = "https"
|
||||
}
|
||||
if proto := forwardedProto(r); proto != "" {
|
||||
scheme = proto
|
||||
}
|
||||
host := sanitizeHost(r.Host)
|
||||
if host == "" {
|
||||
host = net.JoinHostPort(s.cfg.Server.Host, strconv.Itoa(s.cfg.Server.Port))
|
||||
}
|
||||
return scheme + "://" + host + uploadFilePrefix + name
|
||||
}
|
||||
|
||||
// forwardedProto returns the first X-Forwarded-Proto value when it is http or
|
||||
// https, else "".
|
||||
func forwardedProto(r *http.Request) string {
|
||||
v := r.Header.Get("X-Forwarded-Proto")
|
||||
if v == "" {
|
||||
return ""
|
||||
}
|
||||
v = strings.ToLower(strings.TrimSpace(strings.Split(v, ",")[0]))
|
||||
if v == "http" || v == "https" {
|
||||
return v
|
||||
}
|
||||
return ""
|
||||
}
|
||||
|
||||
// sanitizeHost rejects a host value containing path, userinfo or whitespace
|
||||
// syntax; an empty result makes uploadURL fall back to the configured address.
|
||||
func sanitizeHost(host string) string {
|
||||
host = strings.TrimSpace(host)
|
||||
if host == "" || strings.ContainsAny(host, "/\\@ \t") {
|
||||
return ""
|
||||
}
|
||||
return host
|
||||
}
|
||||
@@ -0,0 +1,319 @@
|
||||
package gateway
|
||||
|
||||
import (
|
||||
"bytes"
|
||||
"context"
|
||||
"encoding/json"
|
||||
"io"
|
||||
"mime/multipart"
|
||||
"net/http"
|
||||
"net/http/httptest"
|
||||
"os"
|
||||
"path/filepath"
|
||||
"strconv"
|
||||
"strings"
|
||||
"testing"
|
||||
"testing/fstest"
|
||||
"time"
|
||||
|
||||
"git.yeij.top/AskaEth/Cyrene_v2/internal/auth"
|
||||
)
|
||||
|
||||
// testPNG is the smallest byte string the stdlib sniffer recognizes as
|
||||
// image/png (the 8-byte signature plus padding); the handler only sniffs the
|
||||
// head, it never decodes the image.
|
||||
var testPNG = append([]byte("\x89PNG\r\n\x1a\n"), make([]byte, 64)...)
|
||||
|
||||
// uploadTestEnv builds a Server whose data dir is a fresh temp dir and returns
|
||||
// the httptest server, the JWT service and the resolved data dir.
|
||||
func uploadTestEnv(t *testing.T) (*httptest.Server, *auth.JWTService, string) {
|
||||
t.Helper()
|
||||
cfg := cfgDefault()
|
||||
cfg.DataDir = t.TempDir()
|
||||
_, ts, jwt := buildEnv(t, cfg, Deps{})
|
||||
return ts, jwt, cfg.DataDir
|
||||
}
|
||||
|
||||
// postUpload sends a multipart/form-data POST to /api/v1/upload. field is the
|
||||
// part name (normally "file"); filename is the client-supplied name, which the
|
||||
// handler must ignore.
|
||||
func postUpload(t *testing.T, ts *httptest.Server, token, field, filename string, content []byte) (*http.Response, []byte) {
|
||||
t.Helper()
|
||||
var buf bytes.Buffer
|
||||
mw := multipart.NewWriter(&buf)
|
||||
part, err := mw.CreateFormFile(field, filename)
|
||||
if err != nil {
|
||||
t.Fatalf("create form file: %v", err)
|
||||
}
|
||||
if _, err := part.Write(content); err != nil {
|
||||
t.Fatalf("write part: %v", err)
|
||||
}
|
||||
if err := mw.Close(); err != nil {
|
||||
t.Fatalf("close writer: %v", err)
|
||||
}
|
||||
req, err := http.NewRequest(http.MethodPost, ts.URL+"/api/v1/upload", &buf)
|
||||
if err != nil {
|
||||
t.Fatalf("new request: %v", err)
|
||||
}
|
||||
req.Header.Set("Content-Type", mw.FormDataContentType())
|
||||
if token != "" {
|
||||
req.Header.Set("Authorization", "Bearer "+token)
|
||||
}
|
||||
resp, err := ts.Client().Do(req)
|
||||
if err != nil {
|
||||
t.Fatalf("do request: %v", err)
|
||||
}
|
||||
defer func() { _ = resp.Body.Close() }()
|
||||
body, err := io.ReadAll(resp.Body)
|
||||
if err != nil {
|
||||
t.Fatalf("read response: %v", err)
|
||||
}
|
||||
return resp, body
|
||||
}
|
||||
|
||||
// uploadData is the decoded upload response.
|
||||
type uploadData struct {
|
||||
URL string `json:"url"`
|
||||
Size int64 `json:"size"`
|
||||
MIME string `json:"mime"`
|
||||
Name string `json:"name"`
|
||||
}
|
||||
|
||||
func decodeUploadData(t *testing.T, body []byte) uploadData {
|
||||
t.Helper()
|
||||
var out uploadData
|
||||
if err := json.Unmarshal(decodeBody(t, body)["data"], &out); err != nil {
|
||||
t.Fatalf("decode upload data: %v (body=%s)", err, body)
|
||||
}
|
||||
return out
|
||||
}
|
||||
|
||||
func TestUploadAndServeImage(t *testing.T) {
|
||||
t.Parallel()
|
||||
ts, jwt, dataDir := uploadTestEnv(t)
|
||||
tok, err := jwt.Sign("acct", auth.RoleOwner, time.Hour)
|
||||
if err != nil {
|
||||
t.Fatalf("sign token: %v", err)
|
||||
}
|
||||
|
||||
resp, body := postUpload(t, ts, tok, "file", "../../evil.png", testPNG)
|
||||
if resp.StatusCode != http.StatusCreated {
|
||||
t.Fatalf("status = %d, want 201 (body=%s)", resp.StatusCode, body)
|
||||
}
|
||||
data := decodeUploadData(t, body)
|
||||
if data.MIME != "image/png" {
|
||||
t.Fatalf("mime = %q, want image/png", data.MIME)
|
||||
}
|
||||
if data.Size != int64(len(testPNG)) {
|
||||
t.Fatalf("size = %d, want %d", data.Size, len(testPNG))
|
||||
}
|
||||
if !strings.HasPrefix(data.URL, "http://") || !strings.Contains(data.URL, uploadFilePrefix) {
|
||||
t.Fatalf("url = %q, want absolute %s URL", data.URL, uploadFilePrefix)
|
||||
}
|
||||
// The client's original filename must never reach disk or the response.
|
||||
if strings.Contains(data.Name, "evil") || strings.ContainsAny(data.Name, "/\\") {
|
||||
t.Fatalf("name = %q, want a server-generated hex name", data.Name)
|
||||
}
|
||||
if !validUploadName(data.Name) {
|
||||
t.Fatalf("name = %q, not a valid storage name", data.Name)
|
||||
}
|
||||
if _, err := os.Stat(filepath.Join(dataDir, "uploads", data.Name)); err != nil {
|
||||
t.Fatalf("stored file: %v", err)
|
||||
}
|
||||
|
||||
// The static route is a capability URL: no bearer token, and the bytes and
|
||||
// caching/type headers must come back intact.
|
||||
get, err := ts.Client().Get(data.URL)
|
||||
if err != nil {
|
||||
t.Fatalf("get upload: %v", err)
|
||||
}
|
||||
defer func() { _ = get.Body.Close() }()
|
||||
if get.StatusCode != http.StatusOK {
|
||||
t.Fatalf("get status = %d, want 200", get.StatusCode)
|
||||
}
|
||||
if ct := get.Header.Get("Content-Type"); ct != "image/png" {
|
||||
t.Fatalf("content-type = %q, want image/png", ct)
|
||||
}
|
||||
if cl := get.Header.Get("Content-Length"); cl != strconv.Itoa(len(testPNG)) {
|
||||
t.Fatalf("content-length = %q, want %d", cl, len(testPNG))
|
||||
}
|
||||
if get.Header.Get("Cache-Control") == "" {
|
||||
t.Fatal("cache-control is empty")
|
||||
}
|
||||
got, err := io.ReadAll(get.Body)
|
||||
if err != nil {
|
||||
t.Fatalf("read served body: %v", err)
|
||||
}
|
||||
if !bytes.Equal(got, testPNG) {
|
||||
t.Fatalf("served body = %q, want the uploaded bytes", got)
|
||||
}
|
||||
}
|
||||
|
||||
func TestUploadRejectsNonImage(t *testing.T) {
|
||||
t.Parallel()
|
||||
ts, jwt, _ := uploadTestEnv(t)
|
||||
tok, _ := jwt.Sign("acct", auth.RoleOwner, time.Hour)
|
||||
|
||||
resp, body := postUpload(t, ts, tok, "file", "notes.txt", []byte("just some plain text"))
|
||||
if resp.StatusCode != http.StatusUnsupportedMediaType {
|
||||
t.Fatalf("status = %d, want 415 (body=%s)", resp.StatusCode, body)
|
||||
}
|
||||
wantErrorShape(t, body)
|
||||
}
|
||||
|
||||
func TestUploadRejectsOversize(t *testing.T) {
|
||||
t.Parallel()
|
||||
ts, jwt, _ := uploadTestEnv(t)
|
||||
tok, _ := jwt.Sign("acct", auth.RoleOwner, time.Hour)
|
||||
|
||||
big := make([]byte, maxUploadBytes+1)
|
||||
copy(big, testPNG)
|
||||
resp, body := postUpload(t, ts, tok, "file", "big.png", big)
|
||||
if resp.StatusCode != http.StatusRequestEntityTooLarge {
|
||||
t.Fatalf("status = %d, want 413 (body=%s)", resp.StatusCode, body)
|
||||
}
|
||||
wantErrorShape(t, body)
|
||||
}
|
||||
|
||||
func TestUploadMissingFile(t *testing.T) {
|
||||
t.Parallel()
|
||||
ts, jwt, _ := uploadTestEnv(t)
|
||||
tok, _ := jwt.Sign("acct", auth.RoleOwner, time.Hour)
|
||||
|
||||
resp, body := postUpload(t, ts, tok, "wrongfield", "x.png", testPNG)
|
||||
if resp.StatusCode != http.StatusBadRequest {
|
||||
t.Fatalf("status = %d, want 400 (body=%s)", resp.StatusCode, body)
|
||||
}
|
||||
wantErrorShape(t, body)
|
||||
}
|
||||
|
||||
func TestUploadRequiresAuth(t *testing.T) {
|
||||
t.Parallel()
|
||||
ts, jwt, _ := uploadTestEnv(t)
|
||||
viewer, _ := jwt.Sign("acct", auth.RoleViewer, time.Hour)
|
||||
|
||||
if resp, body := postUpload(t, ts, "", "file", "x.png", testPNG); resp.StatusCode != http.StatusUnauthorized {
|
||||
t.Fatalf("no token status = %d, want 401 (body=%s)", resp.StatusCode, body)
|
||||
}
|
||||
if resp, body := postUpload(t, ts, viewer, "file", "x.png", testPNG); resp.StatusCode != http.StatusForbidden {
|
||||
t.Fatalf("viewer status = %d, want 403 (body=%s)", resp.StatusCode, body)
|
||||
}
|
||||
}
|
||||
|
||||
func TestUploadAudits(t *testing.T) {
|
||||
t.Parallel()
|
||||
cfg := cfgDefault()
|
||||
cfg.DataDir = t.TempDir()
|
||||
store := newMemoryAuditStore()
|
||||
_, ts, jwt := buildEnv(t, cfg, Deps{AuditStore: store})
|
||||
tok, _ := jwt.Sign("acct", auth.RoleOwner, time.Hour)
|
||||
|
||||
if resp, body := postUpload(t, ts, tok, "file", "x.png", testPNG); resp.StatusCode != http.StatusCreated {
|
||||
t.Fatalf("status = %d, want 201 (body=%s)", resp.StatusCode, body)
|
||||
}
|
||||
records, err := store.List(context.Background(), 0)
|
||||
if err != nil {
|
||||
t.Fatalf("list audit: %v", err)
|
||||
}
|
||||
found := false
|
||||
for _, rec := range records {
|
||||
if rec.What == "upload.image" && rec.Result == "ok" {
|
||||
found = true
|
||||
}
|
||||
}
|
||||
if !found {
|
||||
t.Fatalf("no upload.image audit record in %+v", records)
|
||||
}
|
||||
}
|
||||
|
||||
func TestUploadFileServedThroughSPA(t *testing.T) {
|
||||
t.Parallel()
|
||||
cfg := cfgDefault()
|
||||
cfg.DataDir = t.TempDir()
|
||||
// A non-nil frontend enables the SPA wrapper in Handler; the /api/v1/files/
|
||||
// route must still reach the API mux instead of falling back to index.html.
|
||||
frontend := fstest.MapFS{"index.html": &fstest.MapFile{Data: []byte("<html>spa</html>")}}
|
||||
_, ts, jwt := buildEnv(t, cfg, Deps{Frontend: frontend})
|
||||
tok, _ := jwt.Sign("acct", auth.RoleOwner, time.Hour)
|
||||
|
||||
resp, body := postUpload(t, ts, tok, "file", "x.png", testPNG)
|
||||
if resp.StatusCode != http.StatusCreated {
|
||||
t.Fatalf("status = %d, want 201 (body=%s)", resp.StatusCode, body)
|
||||
}
|
||||
data := decodeUploadData(t, body)
|
||||
get, err := ts.Client().Get(data.URL)
|
||||
if err != nil {
|
||||
t.Fatalf("get upload: %v", err)
|
||||
}
|
||||
defer func() { _ = get.Body.Close() }()
|
||||
if get.StatusCode != http.StatusOK {
|
||||
t.Fatalf("get status = %d, want 200 (SPA must not swallow the route)", get.StatusCode)
|
||||
}
|
||||
got, _ := io.ReadAll(get.Body)
|
||||
if !bytes.Equal(got, testPNG) {
|
||||
t.Fatalf("served body = %q, want the uploaded bytes", got)
|
||||
}
|
||||
}
|
||||
|
||||
func TestUploadFileRejectsTraversal(t *testing.T) {
|
||||
t.Parallel()
|
||||
cfg := cfgDefault()
|
||||
cfg.DataDir = t.TempDir()
|
||||
// A non-nil frontend reproduces production routing: without the explicit
|
||||
// guard, ServeMux would clean the raw `..` path and redirect to a path the SPA
|
||||
// then answers with index.html.
|
||||
frontend := fstest.MapFS{"index.html": &fstest.MapFile{Data: []byte("<html>spa</html>")}}
|
||||
_, ts, _ := buildEnv(t, cfg, Deps{Frontend: frontend})
|
||||
// A file that really exists directly under the data dir (not in uploads/)
|
||||
// must not be reachable through the static route.
|
||||
if err := os.WriteFile(filepath.Join(cfg.DataDir, "secret.png"), testPNG, 0o600); err != nil {
|
||||
t.Fatalf("write decoy: %v", err)
|
||||
}
|
||||
paths := []string{
|
||||
"/api/v1/files/",
|
||||
"/api/v1/files/..%2f..%2f..%2fetc%2fpasswd",
|
||||
"/api/v1/files/../../../../etc/passwd",
|
||||
"/api/v1/files/%2e%2e%2fsecret.png",
|
||||
"/api/v1/files/secret.png",
|
||||
"/api/v1/files/00000000000000000000000000000000.png",
|
||||
}
|
||||
for _, p := range paths {
|
||||
resp, err := ts.Client().Get(ts.URL + p)
|
||||
if err != nil {
|
||||
t.Fatalf("get %s: %v", p, err)
|
||||
}
|
||||
body, _ := io.ReadAll(resp.Body)
|
||||
_ = resp.Body.Close()
|
||||
if resp.StatusCode != http.StatusNotFound {
|
||||
t.Fatalf("get %s status = %d, want 404", p, resp.StatusCode)
|
||||
}
|
||||
if bytes.Contains(body, []byte("root:")) {
|
||||
t.Fatalf("get %s leaked /etc/passwd", p)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
func TestValidUploadName(t *testing.T) {
|
||||
t.Parallel()
|
||||
cases := []struct {
|
||||
name string
|
||||
want bool
|
||||
}{
|
||||
{name: "0123456789abcdef0123456789abcdef.png", want: true},
|
||||
{name: "0123456789abcdef0123456789abcdef.webp", want: true},
|
||||
{name: "", want: false},
|
||||
{name: ".", want: false},
|
||||
{name: "..", want: false},
|
||||
{name: "abcd.png", want: false},
|
||||
{name: "0123456789abcdef0123456789abcdef.exe", want: false},
|
||||
{name: "0123456789abcdef0123456789abcdeZ.png", want: false},
|
||||
{name: "0123456789abcdef0123456789abcdef", want: false},
|
||||
{name: "../0123456789abcdef0123456789abcdef.png", want: false},
|
||||
{name: "0123456789abcdef0123456789abcdef.png/../x.png", want: false},
|
||||
}
|
||||
for _, tc := range cases {
|
||||
if got := validUploadName(tc.name); got != tc.want {
|
||||
t.Errorf("validUploadName(%q) = %v, want %v", tc.name, got, tc.want)
|
||||
}
|
||||
}
|
||||
}
|
||||
Reference in new issue
Block a user