fix(gateway): 修 Snapshot 交出活 map 导致进程被 fatal 打死(并发 map 迭代+写入)

现场:cy.log:560,2026-10-03 21:00:28.403,栈顶
encoding/json.mapEncoder.encode ← writeJSON(rest.go:383) ← adminObservabilityMetrics(observability.go:89),
不可 recover,网关 recover 中间件接不住 ⇒ 单二进制进程整体退出(端口 10942 零监听、
所有后台服务含 backthink 心跳一起没;比第一次心跳 tick 早 32.6 秒)。

根因:MetricsCollector.Snapshot 在锁内只新造了 StatusCounts,却把 collector 自己的
活 map m.perMethod / m.perPath 直接交出去;锁外(handler 的 writeJSON)才开始遍历,
而每个请求的 logging 中间件(middleware.go:79)在别的 goroutine 里 Record 写这两个 map
⇒ 写有锁、迭代没锁。面板并行轮询十几个端点,/admin/observability/metrics 被打了 29 次
且常见同毫秒两个请求,必踩。

修法:Snapshot 改为交出副本(新增 copyCounts / copyStatusCounts,都在锁内遍历),
并把「必须在锁外可安全编码」的原因写进文档注释。对外读侧语义不变(仍是全量累计)。

测试(新增 observability_metrics_test.go,242 行):① 快照与 collector 双向隔离(确定性)
② 并发「写 + 编码快照」(-race 下复现生产竞态)③ 真 HTTP + 中间件 + 并行轮询
/configmetric 端点(= 21:00:28 现场)。

验证:go build/vet 绿;-race ./internal/services/gateway/... ok;全量 go test ./... 58 包 ok / 0 FAIL;
linux/amd64 + darwin/arm64 交叉编译绿。
变异验证:把 Snapshot 改回交活 map → 确定性测试 FAIL("snapshot shares the collector's perPath map")
+ -race 报出与生产同一处的 DATA RACE(Record 写在 observability_metrics.go:64)。
This commit is contained in:
AskaEth committed 2026-10-03 21:38:59 +08:00
1 parent 08b3a01752
commit 5847705e35
2 files changed
+273 -7

No files matched your search

@@ -100,13 +100,16 @@ func (m *MetricsCollector) prune(now time.Time) {
// Snapshot returns a point-in-time copy of the metrics. qps counts the requests
// started within metricsQPSSpan of now.
//
// Every map in the result is a fresh copy, never the collector's own: the caller
// serializes the value AFTER this lock is released (the admin handler JSON-encodes
// it), so returning the live maps would let a concurrent Record mutate a map while
// the encoder iterates it — the Go runtime answers that with an unrecoverable
// fatal "concurrent map iteration and map write" that takes the whole process
// down (the 2026-10-03 21:00:28 crash on GET /api/v1/admin/observability/metrics).
func (m *MetricsCollector) Snapshot(now time.Time) AdminMetrics {
m.mu.Lock()
defer m.mu.Unlock()
statuses := make(map[string]int64, len(m.statuses))
for k, v := range m.statuses {
statuses[strconv.Itoa(k)] = v
}
var qps int64
for _, t := range m.recent {
if now.Sub(t) <= metricsQPSSpan {
@@ -122,10 +125,31 @@ func (m *MetricsCollector) Snapshot(now time.Time) AdminMetrics {
QPS: float64(qps),
AvgLatencyMs: float64(avg),
MaxLatencyMs: m.maxDur,
PerMethod: m.perMethod,
PerPath: m.perPath,
StatusCounts: statuses,
PerMethod: copyCounts(m.perMethod),
PerPath: copyCounts(m.perPath),
StatusCounts: copyStatusCounts(m.statuses),
LLMCalls: m.llm,
SubagentEvents: m.subagent,
}
}
// copyCounts returns a detached copy of a string-keyed counter map, so the
// snapshot can be encoded after the collector's lock is released. The caller must
// hold m.mu while copying (the copy iterates the live map).
func copyCounts(src map[string]int64) map[string]int64 {
out := make(map[string]int64, len(src))
for k, v := range src {
out[k] = v
}
return out
}
// copyStatusCounts renders the int-keyed status counter as the string-keyed wire
// map the AdminMetrics JSON shape declares. The caller must hold m.mu while copying.
func copyStatusCounts(src map[int]int64) map[string]int64 {
out := make(map[string]int64, len(src))
for k, v := range src {
out[strconv.Itoa(k)] = v
}
return out
}