Merge branch 'main' into feat/preflight-confirm-audit-version
This commit is contained in:
@@ -12,6 +12,7 @@
|
||||
| [`integration/heicode-desktop-client-api.md`](./integration/heicode-desktop-client-api.md) | 桌面客户端对接 HM(模板 Agent 模型),已生产验证 |
|
||||
| [`integration/heicode-am-contract.md`](./integration/heicode-am-contract.md) | HM ↔ AM(agent_management)接口契约 |
|
||||
| [`integration/heicode-swarm-deferred.md`](./integration/heicode-swarm-deferred.md) | **蜂群(Swarm)现状裁定与后续跟踪入口**:HM 当前不实现 swarm runtime,旧 sub/蜂群文档作废后的上下文迁移映射与归属(AM/Swarm 侧) |
|
||||
| [`integration/telemetry-production-enablement-checklist.md`](./integration/telemetry-production-enablement-checklist.md) | **客户端错误遥测生产开启前置 Checklist(#44 定稿)**:代码侧控制核验、生产配置、隐私/法务签字硬前置、上线/回滚验证;开启 `HEICODE_TELEMETRY_ENABLED` 前必过 |
|
||||
| [`deployment/azure-production-deploy-guardrails.md`](./deployment/azure-production-deploy-guardrails.md) | Azure VM / PostgreSQL / Redis / Agent / NewAPI 生产部署前的安全守卫、环境变量注入和验证计划 |
|
||||
|
||||
旧 Agent API 草案、旧 sub / 蜂群任务编排、旧里程碑、旧架构说明和旧上手材料不再作为实施依据(相关文档已删除)。**蜂群相关文档的作废/迁移映射与「HM 不实现 swarm runtime、新能力归 AM/Swarm 侧跟踪」的结论见 [`integration/heicode-swarm-deferred.md`](./integration/heicode-swarm-deferred.md)(删除≠丢上下文,迁移到此追踪入口)。** 当前实施模型以 `heicode.md`、`plan.md` 与 `integration/heicode-hm-template-agent-model.md` 为准;生产部署操作以安全守卫文档约束,且不得覆盖产品/架构主线。
|
||||
|
||||
@@ -308,6 +308,37 @@ signature = base64( ed25519_sign( device_priv, sha256(canonical) ) )
|
||||
|
||||
---
|
||||
|
||||
## 5.2 Swarm 运行查询 🟡(#45 Phase1 · 只读预览)
|
||||
|
||||
多 Agent 蜂群运行(`agent_swarm` / HeiCode Swarm)的**只读查询**。数据全部来自 HM 已持久化的运行时回调(Swarm → HM 带签名回调),**无需实时调 Swarm**,故不受 `agent_swarm#2` 契约冻结阻塞。
|
||||
|
||||
| 方法 | 路径 | 说明 | 状态 |
|
||||
|---|---|---|---|
|
||||
| GET | `/api/heicode/swarms` | 列出我的 swarm 运行 | 🟢 本地数据 |
|
||||
| GET | `/api/heicode/swarms/:id` | 单个运行状态(:id = deployment_id / swarm_id / correlation_id 任一) | 🟢 |
|
||||
| GET | `/api/heicode/swarms/:id/events?after=&limit=` | 事件增量拉取(`after`=上次返回的 `next_after`,oldest-first) | 🟢 |
|
||||
| GET | `/api/heicode/swarms/:id/artifacts` | 从已存事件派生的产物 | 🟢(派生) |
|
||||
| POST | `/api/heicode/swarms/:id/stop` | 停止运行(**写**) | 🟡 待 `agent_swarm#2` 冻结 + `SWARM_RUNTIME_ENABLED=true` |
|
||||
|
||||
```json
|
||||
// GET /api/heicode/swarms/:id
|
||||
{ "success": true, "data": {
|
||||
"deployment_id":"dep_…", "swarm_id":"swarm-…", "correlation_id":"…",
|
||||
"status":"running", "phase":"…", "runtime_state":"…", "failure_reason":"",
|
||||
"created_at":"…","updated_at":"…","runtime_last_sync_at":"…" }}
|
||||
|
||||
// GET /api/heicode/swarms/:id/events?after=120
|
||||
{ "success": true, "data": {
|
||||
"items":[ {"id":121,"event_type":"task.completed","task_id":"…","result":"ok","occurred_at":"…","payload":{…}} ],
|
||||
"next_after":121, "count":1 }}
|
||||
```
|
||||
|
||||
- 状态机(契约 §4):`waiting_approval → running →(blocked ⇄ running)→ completed/failed/stopped`。
|
||||
- `events` 用 `id` 游标(`next_after`)增量轮询;事件 `payload` **已脱敏**(递归剔除 `secret_ref`/credentials/大字段 + `RedactText` 兜底,绝不下发 `azkv://` secret_ref 或 sk-/Bearer)。事件查询按**当前用户**作用域(防跨用户泄漏)。
|
||||
- `stop` 在契约冻结前**一律不伪造成功**:未启用 → `POLICY_REJECTED`;即使 `SWARM_RUNTIME_ENABLED=true` 也返回 `NOT_IMPLEMENTED`(未真正转发运行时),直到 `agent_swarm#2` 冻结后接上真实 stop。
|
||||
|
||||
---
|
||||
|
||||
## 6. 直连 Agent(客户端 ↔ agent,A2A 协议)
|
||||
|
||||
> 客户端**直接连 agent 子域名**(§5 的 `subdomain`),不经过 HM。agent 是 AM 的 `coding_a2a_agent`,走 **A2A 协议**。
|
||||
|
||||
@@ -0,0 +1,53 @@
|
||||
# 客户端错误遥测 — 生产开启前置 Checklist(定稿)
|
||||
|
||||
> 工单:#44(拆分自 #32 EPIC)· 状态:**工程项已就绪;开启前置 = 隐私披露文档已发布 + 评审通过**
|
||||
> 适用:把 `HEICODE_TELEMETRY_ENABLED` 从默认 `false` 翻到 `true` 之前的强制检查清单。
|
||||
|
||||
## 0. 一句话
|
||||
|
||||
遥测**默认关闭**(摄入端点回 410 kill switch)。在本清单**全部勾选**前,生产 **不得** 开启;尤其是 §3 的隐私披露是硬前置——工程已把代码侧防线做齐,但「设备 ID 可关联账号」属隐私敏感,必须先在隐私文档如实披露并发布。
|
||||
|
||||
## 1. 代码侧控制(已实现,可现场核验)
|
||||
|
||||
| 控制 | 实现 | 核验方式 |
|
||||
|---|---|---|
|
||||
| 默认关闭 + kill switch | `HEICODE_TELEMETRY_ENABLED` 默认 `false`,关时摄入端点返回 **410** | `GET /api/heicode/config` → `telemetry.enabled=false`;`POST /api/heicode/telemetry/events` → 410 |
|
||||
| 不计费、不进 consume log | 独立 `telemetry_events` 表,摄入不写消费日志/不动 quota | 代码 `controller/heicode_telemetry.go`;开启后抽查无 consume log |
|
||||
| 仅设备配对可上送 | 需 V2 设备签名 + `X-Heicode-Device-Id`;会话-only 拒绝(403) | 无设备头请求 → 403 |
|
||||
| 批量/尺寸限制 | 顶层数组 1–20 条、≤256KB | 超限 → 413 |
|
||||
| context 字段白名单(#42) | 仅 `route/retryable/phase/exit_code/duration_ms/attempt`,其余键丢弃 | `TestFilterTelemetryContext_Whitelist` |
|
||||
| 单字段尺寸上限 | `stack_top`/`context` 脱敏后截断 8KiB | `TestCapTelemetryField` |
|
||||
| 服务端二次脱敏 | `stack_top`/`context` 剥离 sk-/Bearer/URL token/JSON 密钥字段 | `TestTelemetryToModel_RedactsSecrets` |
|
||||
| 数据保留期(#43) | `HEICODE_TELEMETRY_RETENTION_DAYS`(默认 30)+ master-only 每日清理 | `GET /api/heicode/config` → `retention_days`;`TestDeleteTelemetryEventsBefore` |
|
||||
|
||||
**禁止采集(白名单已强制)**:prompt、代码正文、token、邮箱、完整文件路径、用户名路径、IP 原文等可识别信息——非白名单 `context` 键一律丢弃 + 服务端脱敏兜底。
|
||||
|
||||
## 2. 生产配置确认(开启时设置 + 复核)
|
||||
|
||||
- [ ] `HEICODE_TELEMETRY_ENABLED=true`(仅在 §3 全部完成后)
|
||||
- [ ] `HEICODE_TELEMETRY_RETENTION_DAYS` 已确认(建议 ≤90;默认 30)
|
||||
- [ ] `HEICODE_TELEMETRY_RETENTION_INTERVAL_HOURS` 已确认(默认 24)
|
||||
- [ ] 仅 master 节点跑清理任务(`IsMasterNode`,已在 `main.go` 内置)
|
||||
- [ ] `GET /api/heicode/config` 返回的 `telemetry` 块与上述配置一致
|
||||
|
||||
## 3. 隐私披露前置(硬性,开启的真正闸门)
|
||||
|
||||
> 这一节是**文档 + 评审**条件,不是代码项;由对应文档 PR 经评审通过即满足。
|
||||
|
||||
- [ ] **heicodeDocs 隐私文档如实披露**「客户端错误遥测;**设备 ID 可关联账号、非匿名**;采集范围(崩溃类别/错误码哈希/脱敏调用栈/运行环境);保留期;不采集 prompt/代码/token/邮箱/完整路径/IP 原文」(跟踪:#34,草案已在 #34 给出)
|
||||
- [ ] 该隐私披露 PR **已评审通过并合并**(在 #44 / #34 留 PR 链接为开启决定留痕)
|
||||
- [ ] 隐私文档**已发布上线**(用户可见),且与实际采集行为一致(`法律声明.md`「隐私披露必须与真实采集行为一致」)
|
||||
|
||||
## 4. 上线 / 回滚验证(开启当次执行)
|
||||
|
||||
- [ ] 开启前:`POST /api/heicode/telemetry/events` 返回 **410**(确认 kill switch 基线)
|
||||
- [ ] 开启后:客户端真实上送一批 → `200 {accepted:n}`,落 `telemetry_events` 表
|
||||
- [ ] 抽查入库行:`stack_top`/`context` 已脱敏;`context` 仅白名单键;无 prompt/邮箱/路径/IP
|
||||
- [ ] 确认**无** consume log / quota 变化(遥测不计费)
|
||||
- [ ] **回滚演练**:把 `HEICODE_TELEMETRY_ENABLED` 改回 `false` → 端点回 410、客户端停送(kill switch 可用)
|
||||
- [ ] 保留期清理任务在日志中可见(`secret/telemetry retention task started`)
|
||||
|
||||
## 5. 结论门
|
||||
|
||||
**只有 §1 已核验 + §2 已确认 + §3 隐私披露已发布 + §4 验证通过,才允许在生产保持 `HEICODE_TELEMETRY_ENABLED=true`。**
|
||||
任一项不满足 → 维持默认关闭(410)。本清单作为 #44 的「checklist 定稿」交付;§3 的隐私披露文档(#34)是开启的前置条件。
|
||||
@@ -0,0 +1,241 @@
|
||||
package controller
|
||||
|
||||
import (
|
||||
"errors"
|
||||
"strconv"
|
||||
"strings"
|
||||
|
||||
"github.com/gin-gonic/gin"
|
||||
"github.com/heicode/manager/common"
|
||||
"github.com/heicode/manager/model"
|
||||
"gorm.io/gorm"
|
||||
)
|
||||
|
||||
// HM-side Swarm Run query (#45 Phase1: list / status / events?after / artifacts / stop).
|
||||
//
|
||||
// 设计要点(关键):**只读查询全部基于 HM 本地已持久化的数据**——Swarm 运行时通过带签名回调
|
||||
// (`/api/agent/callbacks/runtime-events`,见 agent_callback.go)把生命周期事件推给 HM,HM 落
|
||||
// 到 AgentDeployment + AgentCallbackEvent。因此 list/status/events/artifacts **无需**调用 Swarm
|
||||
// 运行时,也就**不依赖 `agent_swarm#2` 尚未冻结的拉取契约**,返工风险低。
|
||||
//
|
||||
// 仅 `stop`(写操作)需要真正调用 Swarm 运行时;在契约冻结 + `SWARM_RUNTIME_ENABLED=true` 前
|
||||
// 默认关闭并明确提示(不 mock,不臆造未冻结的写接口)。
|
||||
//
|
||||
// 字段口径对齐 agent_swarm/docs/integration/runtime-contract.md(草案):
|
||||
// deployment_id ↔ swarm_id ↔ manager_deployment_id 三者映射;事件按 swarm_id/deployment_id 持久化。
|
||||
|
||||
// findUserSwarmDeployment 按 :id(匹配 deployment_id / runtime_swarm_id / correlation_id)加载
|
||||
// 当前用户的 swarm 部署。非本人或非 swarm 模式 → 失败。
|
||||
func findUserSwarmDeployment(c *gin.Context) (model.AgentDeployment, bool) {
|
||||
userID := c.GetInt("id")
|
||||
id := strings.TrimSpace(c.Param("id"))
|
||||
var dep model.AgentDeployment
|
||||
err := model.DB.Where(
|
||||
"user_id = ? AND (deployment_id = ? OR runtime_swarm_id = ? OR correlation_id = ?)",
|
||||
strconv.Itoa(userID), id, id, id,
|
||||
).First(&dep).Error
|
||||
if err != nil {
|
||||
if errors.Is(err, gorm.ErrRecordNotFound) {
|
||||
agentError(c, "POLICY_REJECTED", "swarm run not found")
|
||||
} else {
|
||||
agentError(c, "DEPLOYMENT_CONFLICT", "failed to load swarm run")
|
||||
}
|
||||
return model.AgentDeployment{}, false
|
||||
}
|
||||
if !isSwarmDeployment(dep) {
|
||||
agentError(c, "POLICY_REJECTED", "deployment is not a swarm run")
|
||||
return model.AgentDeployment{}, false
|
||||
}
|
||||
return dep, true
|
||||
}
|
||||
|
||||
func isSwarmDeployment(dep model.AgentDeployment) bool {
|
||||
return strings.EqualFold(dep.SubMode, "swarm") || strings.TrimSpace(dep.RuntimeSwarmID) != ""
|
||||
}
|
||||
|
||||
// swarmDeploymentView 是 swarm 运行的状态摘要视图(脱敏:不含 plan/payload 等大字段与凭据)。
|
||||
func swarmDeploymentView(dep model.AgentDeployment) gin.H {
|
||||
return gin.H{
|
||||
"deployment_id": dep.DeploymentID,
|
||||
"swarm_id": dep.RuntimeSwarmID,
|
||||
"runtime_deployment_id": dep.RuntimeDeploymentID,
|
||||
"correlation_id": dep.CorrelationID,
|
||||
"status": dep.Status,
|
||||
"phase": dep.Phase,
|
||||
"runtime_state": dep.RuntimeState,
|
||||
"failure_reason": dep.FailureReason,
|
||||
"created_at": dep.CreatedAtText,
|
||||
"updated_at": dep.UpdatedAtText,
|
||||
"runtime_last_sync_at": dep.RuntimeLastSyncAtText,
|
||||
}
|
||||
}
|
||||
|
||||
// swarmSensitivePayloadKeys 是事件 payload 中**绝不下发**给客户端的键(凭据/大字段)。
|
||||
// 递归剔除(#45 复审 #1:回调 envelope 可能含 secret_ref,如 approval.requested)。
|
||||
var swarmSensitivePayloadKeys = map[string]bool{
|
||||
"secret_ref": true, "secretref": true, "credentials": true, "credential": true,
|
||||
"secret": true, "token": true, "access_token": true, "refresh_token": true,
|
||||
"api_key": true, "apikey": true, "private_key": true, "access_key": true,
|
||||
"password": true, "passwd": true,
|
||||
// 大字段/内部结构,避免顺带泄漏
|
||||
"plan": true, "payload": true, "permission_manifest": true, "env": true,
|
||||
}
|
||||
|
||||
// stripSensitiveKeys 递归删除敏感键(键名小写匹配 swarmSensitivePayloadKeys)。
|
||||
func stripSensitiveKeys(v any) any {
|
||||
switch t := v.(type) {
|
||||
case map[string]any:
|
||||
out := make(map[string]any, len(t))
|
||||
for k, val := range t {
|
||||
if swarmSensitivePayloadKeys[strings.ToLower(strings.TrimSpace(k))] {
|
||||
continue
|
||||
}
|
||||
out[k] = stripSensitiveKeys(val)
|
||||
}
|
||||
return out
|
||||
case []any:
|
||||
out := make([]any, 0, len(t))
|
||||
for _, item := range t {
|
||||
out = append(out, stripSensitiveKeys(item))
|
||||
}
|
||||
return out
|
||||
default:
|
||||
return v
|
||||
}
|
||||
}
|
||||
|
||||
// sanitizeSwarmPayload 递归剔除敏感/大字段键,再对序列化结果跑一次 RedactText 兜底
|
||||
// (剥离 sk-/Bearer/URL token/JSON 密钥字段)。
|
||||
func sanitizeSwarmPayload(raw string) map[string]any {
|
||||
m := unmarshalResourceJSON(raw)
|
||||
if m == nil {
|
||||
return nil
|
||||
}
|
||||
cleaned, _ := stripSensitiveKeys(m).(map[string]any)
|
||||
if b, err := common.Marshal(cleaned); err == nil {
|
||||
var out map[string]any
|
||||
if err := common.UnmarshalJsonStr(model.RedactText(string(b)), &out); err == nil {
|
||||
return out
|
||||
}
|
||||
}
|
||||
return cleaned
|
||||
}
|
||||
|
||||
// swarmEventView maps a persisted callback event to the client view (payload 脱敏)。
|
||||
func swarmEventView(e model.AgentCallbackEvent) gin.H {
|
||||
return gin.H{
|
||||
"id": e.Id, // 作为下一次 ?after= 的游标
|
||||
"event_id": e.EventID,
|
||||
"event_type": e.EventType,
|
||||
"task_id": e.TaskID,
|
||||
"agent_instance_id": e.AgentInstanceID,
|
||||
"result": e.Result,
|
||||
"occurred_at": e.OccurredAt,
|
||||
"created_at_ms": e.CreatedAtMs,
|
||||
"payload": sanitizeSwarmPayload(e.PayloadJSON),
|
||||
}
|
||||
}
|
||||
|
||||
// HeicodeListSwarms: GET /api/heicode/swarms — 列出当前用户的 swarm 运行(读 HM 本地部署表)。
|
||||
func HeicodeListSwarms(c *gin.Context) {
|
||||
userID := c.GetInt("id")
|
||||
if userID <= 0 {
|
||||
agentError(c, "POLICY_REJECTED", "authentication required")
|
||||
return
|
||||
}
|
||||
if model.DB == nil {
|
||||
agentError(c, "DEPLOYMENT_PERSIST_FAILED", "database not initialised")
|
||||
return
|
||||
}
|
||||
var rows []model.AgentDeployment
|
||||
err := model.DB.Where(
|
||||
"user_id = ? AND (LOWER(sub_mode) = ? OR runtime_swarm_id <> '')",
|
||||
strconv.Itoa(userID), "swarm",
|
||||
).Order("created_at_ms desc, id desc").Limit(200).Find(&rows).Error
|
||||
if err != nil {
|
||||
agentError(c, "DEPLOYMENT_CONFLICT", "failed to list swarm runs")
|
||||
return
|
||||
}
|
||||
items := make([]gin.H, 0, len(rows))
|
||||
for _, r := range rows {
|
||||
items = append(items, swarmDeploymentView(r))
|
||||
}
|
||||
common.ApiSuccess(c, gin.H{"items": items, "total": len(items)})
|
||||
}
|
||||
|
||||
// HeicodeGetSwarmStatus: GET /api/heicode/swarms/:id — 单个 swarm 运行状态。
|
||||
func HeicodeGetSwarmStatus(c *gin.Context) {
|
||||
dep, ok := findUserSwarmDeployment(c)
|
||||
if !ok {
|
||||
return
|
||||
}
|
||||
common.ApiSuccess(c, swarmDeploymentView(dep))
|
||||
}
|
||||
|
||||
// HeicodeListSwarmEvents: GET /api/heicode/swarms/:id/events?after=&limit= — 事件增量拉取。
|
||||
func HeicodeListSwarmEvents(c *gin.Context) {
|
||||
dep, ok := findUserSwarmDeployment(c)
|
||||
if !ok {
|
||||
return
|
||||
}
|
||||
after, _ := strconv.Atoi(strings.TrimSpace(c.Query("after")))
|
||||
limit, _ := strconv.Atoi(strings.TrimSpace(c.Query("limit")))
|
||||
events, err := model.ListSwarmCallbackEventsAfter(strconv.Itoa(c.GetInt("id")), dep.DeploymentID, dep.RuntimeSwarmID, after, limit)
|
||||
if err != nil {
|
||||
agentError(c, "DEPLOYMENT_CONFLICT", "failed to list swarm events")
|
||||
return
|
||||
}
|
||||
items := make([]gin.H, 0, len(events))
|
||||
nextAfter := after
|
||||
for _, e := range events {
|
||||
items = append(items, swarmEventView(e))
|
||||
if e.Id > nextAfter {
|
||||
nextAfter = e.Id
|
||||
}
|
||||
}
|
||||
common.ApiSuccess(c, gin.H{"items": items, "next_after": nextAfter, "count": len(items)})
|
||||
}
|
||||
|
||||
// HeicodeListSwarmArtifacts: GET /api/heicode/swarms/:id/artifacts — 从已持久化事件中筛产物。
|
||||
// 当前从 event_type 含 "artifact" 的回调事件派生(真实数据);专用 artifact 端点待 agent_swarm#2 冻结后补。
|
||||
func HeicodeListSwarmArtifacts(c *gin.Context) {
|
||||
dep, ok := findUserSwarmDeployment(c)
|
||||
if !ok {
|
||||
return
|
||||
}
|
||||
events, err := model.ListSwarmCallbackEventsAfter(strconv.Itoa(c.GetInt("id")), dep.DeploymentID, dep.RuntimeSwarmID, 0, 1000)
|
||||
if err != nil {
|
||||
agentError(c, "DEPLOYMENT_CONFLICT", "failed to list swarm artifacts")
|
||||
return
|
||||
}
|
||||
items := make([]gin.H, 0)
|
||||
for _, e := range events {
|
||||
if strings.Contains(strings.ToLower(e.EventType), "artifact") {
|
||||
items = append(items, swarmEventView(e))
|
||||
}
|
||||
}
|
||||
common.ApiSuccess(c, gin.H{"items": items, "total": len(items),
|
||||
"note": "derived from persisted runtime events; dedicated artifact contract pending agent_swarm#2 freeze"})
|
||||
}
|
||||
|
||||
// HeicodeStopSwarm: POST /api/heicode/swarms/:id/stop — 停止 swarm 运行(写操作)。
|
||||
// 这是唯一需要真正调用 Swarm 运行时的操作。在 agent_swarm#2 契约冻结 + SWARM_RUNTIME_ENABLED=true
|
||||
// 前默认关闭并明确提示(不臆造未冻结的写接口)。冻结后在此接入运行时 stop(草案路径
|
||||
// /api/agent/swarm/deployments/{id}/stop)。
|
||||
func HeicodeStopSwarm(c *gin.Context) {
|
||||
dep, ok := findUserSwarmDeployment(c)
|
||||
if !ok {
|
||||
return
|
||||
}
|
||||
// 运行时 stop 的真实接入随 agent_swarm#2 契约冻结落地。**在此之前一律不伪造 accepted**
|
||||
// (#45 复审 #2:开关打开也不能返回 accepted:true 误导客户端/审计)。无论开关如何,均返回
|
||||
// 明确的「未实现/待契约」语义,直到真正接上运行时 stop。
|
||||
_ = dep
|
||||
if !common.GetEnvOrDefaultBool("SWARM_RUNTIME_ENABLED", false) {
|
||||
agentError(c, "POLICY_REJECTED",
|
||||
"swarm stop not enabled: set SWARM_RUNTIME_ENABLED=true after agent_swarm#2 runtime-contract freeze")
|
||||
return
|
||||
}
|
||||
agentError(c, "NOT_IMPLEMENTED",
|
||||
"swarm stop runtime wiring is pending agent_swarm#2 contract freeze; not forwarded to runtime")
|
||||
}
|
||||
@@ -0,0 +1,46 @@
|
||||
package controller
|
||||
|
||||
import (
|
||||
"encoding/json"
|
||||
"testing"
|
||||
|
||||
"github.com/stretchr/testify/require"
|
||||
)
|
||||
|
||||
// #45 复审 #1:事件 payload 必须脱敏 —— 递归剔除 secret_ref / credentials / 大字段,
|
||||
// 并对结果再跑 RedactText 兜底,绝不把 azkv:// secret_ref 或 sk-/Bearer 下发给客户端。
|
||||
func TestSanitizeSwarmPayload_StripsSecrets(t *testing.T) {
|
||||
raw := `{
|
||||
"task_id":"t1",
|
||||
"approval":{"secret_ref":"azkv://heicode-kv.vault.azure.net/secrets/git-pat","note":"deploy"},
|
||||
"credentials":{"access_key":"AKIA123","secret_access_key":"xxx"},
|
||||
"headers":{"authorization":"Bearer aZ09tokenVALUE"},
|
||||
"api_key":"sk-abcDEF1234567890",
|
||||
"stack":["plain frame","key=sk-leak0987654321ABCD"],
|
||||
"ok":true
|
||||
}`
|
||||
out := sanitizeSwarmPayload(raw)
|
||||
b, err := json.Marshal(out)
|
||||
require.NoError(t, err)
|
||||
s := string(b)
|
||||
|
||||
// 敏感键被递归剔除
|
||||
require.NotContains(t, s, "secret_ref")
|
||||
require.NotContains(t, s, "azkv://")
|
||||
require.NotContains(t, s, "git-pat")
|
||||
require.NotContains(t, s, "credentials")
|
||||
require.NotContains(t, s, "AKIA123")
|
||||
require.NotContains(t, s, "api_key")
|
||||
// RedactText 兜底:残留在普通字段里的 sk-/Bearer 也被打码
|
||||
require.NotContains(t, s, "sk-leak0987654321ABCD")
|
||||
require.NotContains(t, s, "aZ09tokenVALUE")
|
||||
// 非敏感内容保留
|
||||
require.Contains(t, s, "t1")
|
||||
require.Contains(t, s, "ok")
|
||||
}
|
||||
|
||||
func TestSanitizeSwarmPayload_EmptyAndPlain(t *testing.T) {
|
||||
require.Empty(t, sanitizeSwarmPayload(""))
|
||||
out := sanitizeSwarmPayload(`{"route":"chat","n":3}`)
|
||||
require.Equal(t, "chat", out["route"])
|
||||
}
|
||||
@@ -0,0 +1,302 @@
|
||||
package controller
|
||||
|
||||
import (
|
||||
"bytes"
|
||||
"crypto/hmac"
|
||||
"crypto/sha256"
|
||||
"encoding/hex"
|
||||
"errors"
|
||||
"fmt"
|
||||
"io"
|
||||
"net/http"
|
||||
"net/url"
|
||||
"sort"
|
||||
"strings"
|
||||
"time"
|
||||
|
||||
"github.com/heicode/manager/common"
|
||||
"github.com/heicode/manager/model"
|
||||
)
|
||||
|
||||
// AWS resource discovery (#5). Pure-Go AWS Signature V4 (no AWS SDK), mirroring
|
||||
// the REST approach of azure_resource_discovery.go. Uses the Resource Groups
|
||||
// Tagging API GetResources (AWS JSON 1.1), then normalizes ARNs to discoveredCloudResource.
|
||||
//
|
||||
// ⚠️ 覆盖范围(#5 复审 #2,据实声明):GetResources 仅返回**已打标签或曾打标签**的资源,
|
||||
// **不覆盖完全未打标签的 EC2/RDS/S3 等**。因此本适配器是「**tagged resources only**」的发现,
|
||||
// 不等于账号内全量资源;补全需后续加服务级 List/Describe(DescribeInstances / DescribeDBInstances /
|
||||
// ListBuckets 等)。在此之前 #5 的「全量发现」诉求不应据本适配器关闭。
|
||||
//
|
||||
// Creds (from cloud_account metadata/secret): access_key_id, secret_access_key,
|
||||
// region, optional session_token.
|
||||
|
||||
type awsCloudDiscoveryProvider struct{}
|
||||
|
||||
func (awsCloudDiscoveryProvider) name() string { return "aws" }
|
||||
|
||||
type awsDiscoveryCredentials struct {
|
||||
AccessKeyID string
|
||||
SecretAccessKey string
|
||||
SessionToken string
|
||||
Region string
|
||||
}
|
||||
|
||||
func (c awsDiscoveryCredentials) validate() error {
|
||||
if strings.TrimSpace(c.AccessKeyID) == "" {
|
||||
return errors.New("AWS access_key_id required")
|
||||
}
|
||||
if strings.TrimSpace(c.SecretAccessKey) == "" {
|
||||
return errors.New("AWS secret_access_key required")
|
||||
}
|
||||
if strings.TrimSpace(c.Region) == "" {
|
||||
return errors.New("AWS region required")
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
func (awsCloudDiscoveryProvider) discover(account model.ResourceBinding, creds map[string]any) ([]discoveredCloudResource, error) {
|
||||
// region may live on the binding's external_id when not in creds.
|
||||
if mapString(creds, "region") == "" && strings.TrimSpace(account.ExternalId) != "" {
|
||||
creds["region"] = account.ExternalId
|
||||
}
|
||||
return discoverAWS(creds, time.Now(), &http.Client{Timeout: 20 * time.Second})
|
||||
}
|
||||
|
||||
// hmacSHA256 / sha256Hex — SigV4 primitives.
|
||||
func hmacSHA256(key, data []byte) []byte {
|
||||
h := hmac.New(sha256.New, key)
|
||||
h.Write(data)
|
||||
return h.Sum(nil)
|
||||
}
|
||||
|
||||
func sha256Hex(data []byte) string {
|
||||
sum := sha256.Sum256(data)
|
||||
return hex.EncodeToString(sum[:])
|
||||
}
|
||||
|
||||
// awsSigV4Authorization computes the SigV4 Authorization header value + x-amz-date
|
||||
// for the given request. signHeaders are the headers (besides host/x-amz-date) to
|
||||
// include in the signature (e.g. content-type, x-amz-target). Pure & deterministic
|
||||
// given t — verified against AWS's official "get-vanilla" test vector.
|
||||
func awsSigV4Authorization(method, rawURL string, signHeaders map[string]string, payload []byte,
|
||||
cred awsDiscoveryCredentials, service string, t time.Time) (authorization, amzDate string, err error) {
|
||||
|
||||
u, err := url.Parse(rawURL)
|
||||
if err != nil {
|
||||
return "", "", err
|
||||
}
|
||||
amzDate = t.UTC().Format("20060102T150405Z")
|
||||
dateStamp := t.UTC().Format("20060102")
|
||||
|
||||
// Canonical headers must include host + x-amz-date. Lowercase names, trim values, sort.
|
||||
headers := map[string]string{}
|
||||
for k, v := range signHeaders {
|
||||
headers[strings.ToLower(strings.TrimSpace(k))] = strings.TrimSpace(v)
|
||||
}
|
||||
headers["host"] = u.Host
|
||||
headers["x-amz-date"] = amzDate
|
||||
if strings.TrimSpace(cred.SessionToken) != "" {
|
||||
headers["x-amz-security-token"] = strings.TrimSpace(cred.SessionToken)
|
||||
}
|
||||
names := make([]string, 0, len(headers))
|
||||
for k := range headers {
|
||||
names = append(names, k)
|
||||
}
|
||||
sort.Strings(names)
|
||||
var canonicalHeaders strings.Builder
|
||||
for _, n := range names {
|
||||
canonicalHeaders.WriteString(n + ":" + headers[n] + "\n")
|
||||
}
|
||||
signedHeaders := strings.Join(names, ";")
|
||||
|
||||
canonicalURI := u.EscapedPath()
|
||||
if canonicalURI == "" {
|
||||
canonicalURI = "/"
|
||||
}
|
||||
// Canonical query string: sort by key, RFC3986-encoded.
|
||||
canonicalQuery := canonicalizeQuery(u.Query())
|
||||
|
||||
payloadHash := sha256Hex(payload)
|
||||
canonicalRequest := method + "\n" + canonicalURI + "\n" + canonicalQuery + "\n" +
|
||||
canonicalHeaders.String() + "\n" + signedHeaders + "\n" + payloadHash
|
||||
|
||||
credentialScope := dateStamp + "/" + cred.Region + "/" + service + "/aws4_request"
|
||||
stringToSign := "AWS4-HMAC-SHA256\n" + amzDate + "\n" + credentialScope + "\n" + sha256Hex([]byte(canonicalRequest))
|
||||
|
||||
kDate := hmacSHA256([]byte("AWS4"+cred.SecretAccessKey), []byte(dateStamp))
|
||||
kRegion := hmacSHA256(kDate, []byte(cred.Region))
|
||||
kService := hmacSHA256(kRegion, []byte(service))
|
||||
kSigning := hmacSHA256(kService, []byte("aws4_request"))
|
||||
signature := hex.EncodeToString(hmacSHA256(kSigning, []byte(stringToSign)))
|
||||
|
||||
authorization = fmt.Sprintf("AWS4-HMAC-SHA256 Credential=%s/%s, SignedHeaders=%s, Signature=%s",
|
||||
cred.AccessKeyID, credentialScope, signedHeaders, signature)
|
||||
return authorization, amzDate, nil
|
||||
}
|
||||
|
||||
func canonicalizeQuery(values url.Values) string {
|
||||
if len(values) == 0 {
|
||||
return ""
|
||||
}
|
||||
keys := make([]string, 0, len(values))
|
||||
for k := range values {
|
||||
keys = append(keys, k)
|
||||
}
|
||||
sort.Strings(keys)
|
||||
parts := make([]string, 0, len(keys))
|
||||
for _, k := range keys {
|
||||
vs := append([]string(nil), values[k]...)
|
||||
sort.Strings(vs)
|
||||
for _, v := range vs {
|
||||
parts = append(parts, awsURIEncode(k, true)+"="+awsURIEncode(v, true))
|
||||
}
|
||||
}
|
||||
return strings.Join(parts, "&")
|
||||
}
|
||||
|
||||
// awsURIEncode is RFC3986 percent-encoding per SigV4 rules.
|
||||
func awsURIEncode(s string, encodeSlash bool) string {
|
||||
var b strings.Builder
|
||||
for _, c := range []byte(s) {
|
||||
switch {
|
||||
case (c >= 'A' && c <= 'Z') || (c >= 'a' && c <= 'z') || (c >= '0' && c <= '9') ||
|
||||
c == '-' || c == '_' || c == '.' || c == '~':
|
||||
b.WriteByte(c)
|
||||
case c == '/' && !encodeSlash:
|
||||
b.WriteByte(c)
|
||||
default:
|
||||
b.WriteString(fmt.Sprintf("%%%02X", c))
|
||||
}
|
||||
}
|
||||
return b.String()
|
||||
}
|
||||
|
||||
// awsResourceTagMapping is one entry of the Resource Groups Tagging API response.
|
||||
type awsResourceTagMapping struct {
|
||||
ResourceARN string `json:"ResourceARN"`
|
||||
Tags []struct {
|
||||
Key string `json:"Key"`
|
||||
Value string `json:"Value"`
|
||||
} `json:"Tags"`
|
||||
}
|
||||
|
||||
type awsGetResourcesResponse struct {
|
||||
ResourceTagMappingList []awsResourceTagMapping `json:"ResourceTagMappingList"`
|
||||
PaginationToken string `json:"PaginationToken"`
|
||||
}
|
||||
|
||||
// discoverAWS enumerates the account's tagged resources via the Resource Groups
|
||||
// Tagging API (paginated). Real implementation (no mock).
|
||||
func discoverAWS(creds map[string]any, now time.Time, httpClient *http.Client) ([]discoveredCloudResource, error) {
|
||||
cred := awsDiscoveryCredentials{
|
||||
AccessKeyID: mapString(creds, "access_key_id"),
|
||||
SecretAccessKey: mapString(creds, "secret_access_key"),
|
||||
SessionToken: mapString(creds, "session_token"),
|
||||
Region: mapString(creds, "region"),
|
||||
}
|
||||
if err := cred.validate(); err != nil {
|
||||
return nil, err
|
||||
}
|
||||
endpoint := fmt.Sprintf("https://tagging.%s.amazonaws.com/", cred.Region)
|
||||
const target = "ResourceGroupsTaggingAPI_20170126.GetResources"
|
||||
|
||||
out := []discoveredCloudResource{}
|
||||
paginationToken := ""
|
||||
for page := 0; page < 50; page++ {
|
||||
bodyMap := map[string]any{"ResourcesPerPage": 100}
|
||||
if paginationToken != "" {
|
||||
bodyMap["PaginationToken"] = paginationToken
|
||||
}
|
||||
body, err := common.Marshal(bodyMap)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
signHeaders := map[string]string{
|
||||
"content-type": "application/x-amz-json-1.1",
|
||||
"x-amz-target": target,
|
||||
}
|
||||
auth, amzDate, err := awsSigV4Authorization(http.MethodPost, endpoint, signHeaders, body, cred, "tagging", now)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
req, err := http.NewRequest(http.MethodPost, endpoint, bytes.NewReader(body))
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
req.Header.Set("Content-Type", "application/x-amz-json-1.1")
|
||||
req.Header.Set("X-Amz-Target", target)
|
||||
req.Header.Set("X-Amz-Date", amzDate)
|
||||
req.Header.Set("Authorization", auth)
|
||||
if strings.TrimSpace(cred.SessionToken) != "" {
|
||||
req.Header.Set("X-Amz-Security-Token", cred.SessionToken)
|
||||
}
|
||||
resp, err := httpClient.Do(req)
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("AWS GetResources request failed: %w", err)
|
||||
}
|
||||
raw, _ := io.ReadAll(io.LimitReader(resp.Body, 8<<20))
|
||||
resp.Body.Close()
|
||||
if resp.StatusCode < http.StatusOK || resp.StatusCode >= http.StatusMultipleChoices {
|
||||
return nil, fmt.Errorf("AWS GetResources failed (%d): %s", resp.StatusCode, strings.TrimSpace(string(raw)))
|
||||
}
|
||||
var payload awsGetResourcesResponse
|
||||
if err := common.Unmarshal(raw, &payload); err != nil {
|
||||
return nil, err
|
||||
}
|
||||
for _, m := range payload.ResourceTagMappingList {
|
||||
if d, ok := awsARNToResource(m.ResourceARN); ok {
|
||||
tags := map[string]any{}
|
||||
for _, t := range m.Tags {
|
||||
tags[t.Key] = t.Value
|
||||
}
|
||||
d.Metadata = map[string]any{"region": cred.Region, "tags": tags}
|
||||
out = append(out, d)
|
||||
}
|
||||
}
|
||||
paginationToken = strings.TrimSpace(payload.PaginationToken)
|
||||
if paginationToken == "" {
|
||||
break
|
||||
}
|
||||
}
|
||||
return out, nil
|
||||
}
|
||||
|
||||
// awsARNToResource parses arn:partition:service:region:account:resource into a
|
||||
// normalized resource. NativeType is CloudFormation-style (AWS::<Service>::<Type>)
|
||||
// so classifyCloudResourceType maps it to vm/database/blob consistently.
|
||||
func awsARNToResource(arn string) (discoveredCloudResource, bool) {
|
||||
arn = strings.TrimSpace(arn)
|
||||
if !strings.HasPrefix(arn, "arn:") {
|
||||
return discoveredCloudResource{}, false
|
||||
}
|
||||
parts := strings.SplitN(arn, ":", 6)
|
||||
if len(parts) < 6 {
|
||||
return discoveredCloudResource{}, false
|
||||
}
|
||||
service := parts[2]
|
||||
region := parts[3]
|
||||
resource := parts[5]
|
||||
// resource may be "type/id", "type:id", or just "id" (e.g. s3 bucket).
|
||||
resType, resName := "", resource
|
||||
if i := strings.IndexAny(resource, "/:"); i >= 0 {
|
||||
resType = resource[:i]
|
||||
resName = resource[i+1:]
|
||||
}
|
||||
native := "AWS::" + strings.ToUpper(service)
|
||||
if resType != "" {
|
||||
native += "::" + resType
|
||||
} else if strings.EqualFold(service, "s3") {
|
||||
native += "::Bucket"
|
||||
}
|
||||
name := resName
|
||||
if name == "" {
|
||||
name = arn
|
||||
}
|
||||
return discoveredCloudResource{
|
||||
ExternalId: arn,
|
||||
Name: name,
|
||||
NativeType: native,
|
||||
Location: region,
|
||||
BindingScope: "aws:" + arn,
|
||||
}, true
|
||||
}
|
||||
@@ -0,0 +1,269 @@
|
||||
package controller
|
||||
|
||||
import (
|
||||
"errors"
|
||||
"fmt"
|
||||
"strings"
|
||||
|
||||
"github.com/gin-gonic/gin"
|
||||
"github.com/heicode/manager/common"
|
||||
"github.com/heicode/manager/model"
|
||||
"gorm.io/gorm"
|
||||
)
|
||||
|
||||
// Cloud resource discovery — provider-agnostic layer (#5).
|
||||
//
|
||||
// 历史上只有 Azure 发现(controller/azure_resource_discovery.go,硬要求 provider==azure)。
|
||||
// 本文件抽象出统一的 provider 接口,让 AWS / GCP 走同一条「发现 → 归一化 → 落库」路径,
|
||||
// 并对 vm / database / blob 做跨云一致的分类(DoD #3)。Azure 适配器复用既有 ARM 客户端,
|
||||
// 故三家行为一致;具体云的签名/鉴权在 aws_resource_discovery.go / gcp_resource_discovery.go。
|
||||
|
||||
// discoveredCloudResource 是各 provider 适配器输出的**归一化**资源视图。upsert 时由统一
|
||||
// 路径补 source_account_id 与 classified_type,并写入 ResourceBinding(resource_type=cloud_resource)。
|
||||
type discoveredCloudResource struct {
|
||||
ExternalId string // ARN / Azure resource id / GCP asset name —— 跨 provider 的稳定外部 id
|
||||
Name string // 展示名
|
||||
NativeType string // provider 原生类型字符串(如 Microsoft.Compute/virtualMachines、AWS::EC2::Instance、compute.googleapis.com/Instance)
|
||||
Location string // region / location
|
||||
BindingScope string // 可选;为空时默认 "<provider>:<external_id>"
|
||||
Metadata map[string]any // provider 特定附加字段
|
||||
}
|
||||
|
||||
// cloudDiscoveryProvider 是「连接云账号 → 列资源」的统一抽象(DoD #3)。
|
||||
type cloudDiscoveryProvider interface {
|
||||
name() string
|
||||
// discover 用 creds(已合并 metadata + secret JSON)列出 account 下的资源。
|
||||
discover(account model.ResourceBinding, creds map[string]any) ([]discoveredCloudResource, error)
|
||||
}
|
||||
|
||||
func cloudDiscoveryProviderByName(provider string) (cloudDiscoveryProvider, bool) {
|
||||
switch strings.ToLower(strings.TrimSpace(provider)) {
|
||||
case "azure":
|
||||
return azureCloudDiscoveryProvider{}, true
|
||||
case "aws", "amazon":
|
||||
return awsCloudDiscoveryProvider{}, true
|
||||
case "gcp", "google", "gce":
|
||||
return gcpCloudDiscoveryProvider{}, true
|
||||
}
|
||||
return nil, false
|
||||
}
|
||||
|
||||
// classifyCloudResourceType 把 provider 原生类型映射到跨云一致的逻辑类别(DoD #3):
|
||||
// vm / database / blob / cloud_resource(兜底)。基于不区分大小写的子串匹配,覆盖三家常见服务。
|
||||
func classifyCloudResourceType(nativeType string) string {
|
||||
t := strings.ToLower(nativeType)
|
||||
containsAny := func(s string, subs ...string) bool {
|
||||
for _, sub := range subs {
|
||||
if strings.Contains(s, sub) {
|
||||
return true
|
||||
}
|
||||
}
|
||||
return false
|
||||
}
|
||||
switch {
|
||||
// 数据库(放在 vm 之前:避免 "sqlvirtual..." 之类被误判为 vm;数据库特征更强)
|
||||
case containsAny(t, "sql", "postgres", "mysql", "mariadb", "rds", "database", "cosmos",
|
||||
"dynamodb", "spanner", "cloudsql", "bigtable", "redis", "documentdb", "memcache"):
|
||||
return "database"
|
||||
case containsAny(t, "virtualmachine", "ec2::instance", "ec2/instance", "compute/instance",
|
||||
"compute.googleapis.com/instance", "virtualmachinescaleset", "instance"):
|
||||
return "vm"
|
||||
case containsAny(t, "storageaccount", "::s3::", "s3:::", "/buckets/", "storage/bucket",
|
||||
"storage.googleapis.com/bucket", "blob", "bucket"):
|
||||
return "blob"
|
||||
default:
|
||||
return "cloud_resource"
|
||||
}
|
||||
}
|
||||
|
||||
// upsertDiscoveredCloudResources 把归一化资源幂等写入 ResourceBinding(按 user+type+provider+external_id
|
||||
// 去重)。统一所有 provider 的落库格式:resource_type 固定为 cloud_resource(与既有 Azure 行为一致,
|
||||
// 不让发现的资源直接变成可绑定的 vm/db/blob),跨云类别放在 metadata.classified_type。
|
||||
func upsertDiscoveredCloudResources(account model.ResourceBinding, provider string, discovered []discoveredCloudResource) ([]resourceResponse, error) {
|
||||
items := make([]resourceResponse, 0, len(discovered))
|
||||
for _, d := range discovered {
|
||||
if strings.TrimSpace(d.ExternalId) == "" {
|
||||
continue
|
||||
}
|
||||
var resource model.ResourceBinding
|
||||
err := model.DB.Where(
|
||||
"user_id = ? AND resource_type = ? AND provider = ? AND external_id = ?",
|
||||
account.UserId, "cloud_resource", provider, d.ExternalId,
|
||||
).First(&resource).Error
|
||||
if err != nil && !errors.Is(err, gorm.ErrRecordNotFound) {
|
||||
return nil, err
|
||||
}
|
||||
if err := applyDiscoveredResourceFields(&resource, account, provider, d); err != nil {
|
||||
return nil, err
|
||||
}
|
||||
if resource.Id == 0 {
|
||||
if err := model.DB.Create(&resource).Error; err != nil {
|
||||
return nil, err
|
||||
}
|
||||
} else if err := model.DB.Save(&resource).Error; err != nil {
|
||||
return nil, err
|
||||
}
|
||||
items = append(items, resourceToResponse(resource))
|
||||
}
|
||||
return items, nil
|
||||
}
|
||||
|
||||
// applyDiscoveredResourceFields 把一条归一化发现结果写入 ResourceBinding 字段(DB 无关,可单测)。
|
||||
// 关键安全约束(#5 复审 #1):**SecretRef 恒为空**——发现出的子资源绝不继承云账号 secret_ref,
|
||||
// 否则会把账号级凭据引用扩散到每个 VM/S3/DB 并经 resourceToResponse / grant manifest 下发给 agent。
|
||||
// 发现是只读清单;如需 agent 读取某资源,应另行 grant/审批/最小权限 secret。
|
||||
func applyDiscoveredResourceFields(resource *model.ResourceBinding, account model.ResourceBinding, provider string, d discoveredCloudResource) error {
|
||||
metadata := map[string]any{}
|
||||
for k, v := range d.Metadata {
|
||||
metadata[k] = v
|
||||
}
|
||||
metadata["source_account_id"] = account.Id
|
||||
metadata["classified_type"] = classifyCloudResourceType(d.NativeType)
|
||||
if strings.TrimSpace(d.NativeType) != "" {
|
||||
metadata["native_type"] = d.NativeType
|
||||
}
|
||||
if strings.TrimSpace(d.Location) != "" {
|
||||
metadata["location"] = d.Location
|
||||
}
|
||||
metadataJSON, err := marshalResourceJSON(metadata)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
permissionScopeJSON, err := marshalResourceJSON(map[string]any{"actions": []string{provider + ":read"}})
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
constraintsJSON, err := marshalResourceJSON(unmarshalResourceJSON(account.Constraints))
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
name := strings.TrimSpace(d.Name)
|
||||
if name == "" {
|
||||
name = d.ExternalId
|
||||
}
|
||||
bindingScope := strings.TrimSpace(d.BindingScope)
|
||||
if bindingScope == "" {
|
||||
bindingScope = fmt.Sprintf("%s:%s", provider, d.ExternalId)
|
||||
}
|
||||
resource.UserId = account.UserId
|
||||
resource.TenantId = account.TenantId
|
||||
resource.ProjectId = account.ProjectId
|
||||
resource.BindingScope = bindingScope
|
||||
resource.Name = name
|
||||
resource.ResourceType = "cloud_resource"
|
||||
resource.Provider = provider
|
||||
resource.ExternalId = d.ExternalId
|
||||
resource.SecretRef = "" // 见上:绝不继承账号 secret_ref
|
||||
resource.Metadata = metadataJSON
|
||||
resource.PermissionScope = permissionScopeJSON
|
||||
resource.Constraints = constraintsJSON
|
||||
resource.Status = "active"
|
||||
return nil
|
||||
}
|
||||
|
||||
// DiscoverCloudResources: POST /api/resources/:id/discover-cloud — provider-agnostic
|
||||
// discovery dispatcher (#5). Routes by the account's provider to the matching
|
||||
// adapter (azure/aws/gcp), then upserts via the unified path.
|
||||
func DiscoverCloudResources(c *gin.Context) {
|
||||
userId := c.GetInt("id")
|
||||
var account model.ResourceBinding
|
||||
if err := model.DB.Where("id = ? AND user_id = ?", c.Param("id"), userId).First(&account).Error; err != nil {
|
||||
if errors.Is(err, gorm.ErrRecordNotFound) {
|
||||
common.ApiErrorMsg(c, "resource not found")
|
||||
return
|
||||
}
|
||||
common.ApiError(c, err)
|
||||
return
|
||||
}
|
||||
if account.ResourceType != "cloud_account" || account.Status != "active" {
|
||||
common.ApiErrorMsg(c, "resource must be an active cloud_account")
|
||||
return
|
||||
}
|
||||
impl, ok := cloudDiscoveryProviderByName(account.Provider)
|
||||
if !ok {
|
||||
common.ApiErrorMsg(c, "unsupported cloud provider: "+account.Provider+" (supported: azure, aws, gcp)")
|
||||
return
|
||||
}
|
||||
|
||||
// creds = metadata(非密)合并 secret JSON(密)。secret 优先级在适配器内部按需处理。
|
||||
creds := unmarshalResourceJSON(account.Metadata)
|
||||
if creds == nil {
|
||||
creds = map[string]any{}
|
||||
}
|
||||
if strings.TrimSpace(account.SecretRef) != "" {
|
||||
secretClient, err := newSecretStoreClientFromEnv()
|
||||
if err != nil {
|
||||
common.ApiError(c, err)
|
||||
return
|
||||
}
|
||||
secretData, err := secretClient.getJSONSecret(account.SecretRef)
|
||||
if err != nil {
|
||||
common.ApiError(c, err)
|
||||
return
|
||||
}
|
||||
for k, v := range secretData {
|
||||
creds[k] = v // secret 覆盖同名非密字段
|
||||
}
|
||||
}
|
||||
|
||||
discovered, err := impl.discover(account, creds)
|
||||
if err != nil {
|
||||
common.ApiError(c, err)
|
||||
return
|
||||
}
|
||||
items, err := upsertDiscoveredCloudResources(account, impl.name(), discovered)
|
||||
if err != nil {
|
||||
common.ApiError(c, err)
|
||||
return
|
||||
}
|
||||
common.ApiSuccess(c, gin.H{
|
||||
"account": resourceToResponse(account),
|
||||
"provider": impl.name(),
|
||||
"items": items,
|
||||
"discovered": len(items),
|
||||
})
|
||||
}
|
||||
|
||||
// azureCloudDiscoveryProvider 让既有 Azure ARM 发现实现统一接口(复用 azure_resource_discovery.go
|
||||
// 的 client/token/list),保证三家走同一抽象。
|
||||
type azureCloudDiscoveryProvider struct{}
|
||||
|
||||
func (azureCloudDiscoveryProvider) name() string { return "azure" }
|
||||
|
||||
func (azureCloudDiscoveryProvider) discover(account model.ResourceBinding, creds map[string]any) ([]discoveredCloudResource, error) {
|
||||
cred := azureDiscoveryCredentials{
|
||||
SubscriptionID: firstString(creds, nil, account.ExternalId, "subscription_id"),
|
||||
TenantID: firstString(creds, nil, account.TenantId, "tenant_id"),
|
||||
ClientID: mapString(creds, "client_id"),
|
||||
ClientSecret: mapString(creds, "client_secret"),
|
||||
}
|
||||
if err := cred.validate(); err != nil {
|
||||
return nil, err
|
||||
}
|
||||
client := newAzureResourceDiscoveryClientFromEnv()
|
||||
token, err := client.clientCredentialsToken(cred)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
arm, err := client.listSubscriptionResources(cred.SubscriptionID, token)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
out := make([]discoveredCloudResource, 0, len(arm))
|
||||
for _, r := range arm {
|
||||
out = append(out, discoveredCloudResource{
|
||||
ExternalId: r.ID,
|
||||
Name: r.Name,
|
||||
NativeType: r.Type,
|
||||
Location: r.Location,
|
||||
BindingScope: fmt.Sprintf("azure:%s:%s", cred.SubscriptionID, r.ID),
|
||||
Metadata: map[string]any{
|
||||
"subscription_id": cred.SubscriptionID,
|
||||
"tenant_id": cred.TenantID,
|
||||
"resource_group": azureResourceGroupFromID(r.ID),
|
||||
"tags": r.Tags,
|
||||
},
|
||||
})
|
||||
}
|
||||
return out, nil
|
||||
}
|
||||
@@ -0,0 +1,150 @@
|
||||
package controller
|
||||
|
||||
import (
|
||||
"crypto"
|
||||
"crypto/rand"
|
||||
"crypto/rsa"
|
||||
"crypto/sha256"
|
||||
"crypto/x509"
|
||||
"encoding/base64"
|
||||
"encoding/pem"
|
||||
"strings"
|
||||
"testing"
|
||||
"time"
|
||||
|
||||
"github.com/heicode/manager/model"
|
||||
"github.com/stretchr/testify/require"
|
||||
)
|
||||
|
||||
// #5: 跨云类别分类 vm/database/blob 一致。
|
||||
func TestClassifyCloudResourceType(t *testing.T) {
|
||||
cases := map[string]string{
|
||||
// vm
|
||||
"Microsoft.Compute/virtualMachines": "vm",
|
||||
"AWS::EC2::Instance": "vm",
|
||||
"compute.googleapis.com/Instance": "vm",
|
||||
// database
|
||||
"Microsoft.Sql/servers/databases": "database",
|
||||
"AWS::RDS::DBInstance": "database",
|
||||
"sqladmin.googleapis.com/Instance": "database",
|
||||
"AWS::DynamoDB::Table": "database",
|
||||
// blob / object storage
|
||||
"Microsoft.Storage/storageAccounts": "blob",
|
||||
"AWS::S3::Bucket": "blob",
|
||||
"storage.googleapis.com/Bucket": "blob",
|
||||
// fallback
|
||||
"Microsoft.Network/virtualNetworks": "cloud_resource",
|
||||
"AWS::IAM::Role": "cloud_resource",
|
||||
}
|
||||
for native, want := range cases {
|
||||
require.Equal(t, want, classifyCloudResourceType(native), "classify %s", native)
|
||||
}
|
||||
}
|
||||
|
||||
// #5: ARN 解析 → 归一化 + 正确分类。
|
||||
func TestAWSARNToResource(t *testing.T) {
|
||||
ec2, ok := awsARNToResource("arn:aws:ec2:us-east-1:123456789012:instance/i-0abc123")
|
||||
require.True(t, ok)
|
||||
require.Equal(t, "us-east-1", ec2.Location)
|
||||
require.Equal(t, "i-0abc123", ec2.Name)
|
||||
require.Equal(t, "vm", classifyCloudResourceType(ec2.NativeType))
|
||||
|
||||
s3, ok := awsARNToResource("arn:aws:s3:::my-bucket")
|
||||
require.True(t, ok)
|
||||
require.Equal(t, "my-bucket", s3.Name)
|
||||
require.Equal(t, "blob", classifyCloudResourceType(s3.NativeType))
|
||||
|
||||
rds, ok := awsARNToResource("arn:aws:rds:eu-west-1:123:db:prod-pg")
|
||||
require.True(t, ok)
|
||||
require.Equal(t, "database", classifyCloudResourceType(rds.NativeType))
|
||||
|
||||
_, ok = awsARNToResource("not-an-arn")
|
||||
require.False(t, ok)
|
||||
}
|
||||
|
||||
// #5: SigV4 实现正确性 —— 对齐 AWS 官方 "get-vanilla" 测试向量。
|
||||
func TestAWSSigV4_VanillaVector(t *testing.T) {
|
||||
cred := awsDiscoveryCredentials{
|
||||
AccessKeyID: "AKIDEXAMPLE",
|
||||
SecretAccessKey: "wJalrXUtnFEMI/K7MDENG+bPxRfiCYEXAMPLEKEY",
|
||||
Region: "us-east-1",
|
||||
}
|
||||
tm := time.Date(2015, 8, 30, 12, 36, 0, 0, time.UTC)
|
||||
auth, amzDate, err := awsSigV4Authorization("GET", "https://example.amazonaws.com/", map[string]string{}, []byte(""), cred, "service", tm)
|
||||
require.NoError(t, err)
|
||||
require.Equal(t, "20150830T123600Z", amzDate)
|
||||
require.Contains(t, auth, "Credential=AKIDEXAMPLE/20150830/us-east-1/service/aws4_request")
|
||||
require.Contains(t, auth, "SignedHeaders=host;x-amz-date")
|
||||
// 官方向量期望签名
|
||||
require.Contains(t, auth, "Signature=5fa00fa31553b73ebf1942676e86291e8372ff2a2260956d9b8aae1d763fbf31")
|
||||
}
|
||||
|
||||
// #5 复审 #3:GCP searchAllResources URL —— scope 的 slash 不得被 %2F 转义。
|
||||
func TestGCPSearchAllResourcesURL(t *testing.T) {
|
||||
u := gcpSearchAllResourcesURL("my-proj", "")
|
||||
require.Equal(t, "https://cloudasset.googleapis.com/v1/projects/my-proj:searchAllResources?pageSize=500", u)
|
||||
require.NotContains(t, u, "%2F", "scope slash 不应被转义")
|
||||
|
||||
u2 := gcpSearchAllResourcesURL("my-proj", "tok en/+")
|
||||
require.Contains(t, u2, "&pageToken=tok+en%2F%2B")
|
||||
}
|
||||
|
||||
// #5 复审 #1:发现出的资源**绝不继承**账号 secret_ref;元数据带 classified_type,不含账号凭据引用。
|
||||
func TestApplyDiscoveredResourceFields_NoSecretInheritance(t *testing.T) {
|
||||
account := model.ResourceBinding{
|
||||
Id: 42, UserId: 7, ResourceType: "cloud_account", Provider: "aws",
|
||||
SecretRef: "azkv://heicode-kv.vault.azure.net/secrets/aws-keys",
|
||||
}
|
||||
d := discoveredCloudResource{
|
||||
ExternalId: "arn:aws:ec2:us-east-1:123:instance/i-1",
|
||||
Name: "i-1", NativeType: "AWS::EC2::Instance", Location: "us-east-1",
|
||||
Metadata: map[string]any{"region": "us-east-1"},
|
||||
}
|
||||
var r model.ResourceBinding
|
||||
require.NoError(t, applyDiscoveredResourceFields(&r, account, "aws", d))
|
||||
|
||||
require.Equal(t, "", r.SecretRef, "发现出的子资源绝不继承账号 secret_ref")
|
||||
require.Equal(t, "cloud_resource", r.ResourceType)
|
||||
require.Equal(t, "aws", r.Provider)
|
||||
require.Equal(t, 7, r.UserId)
|
||||
require.NotContains(t, r.Metadata, "azkv://", "metadata 不得含账号凭据引用")
|
||||
require.Contains(t, r.Metadata, "\"classified_type\":\"vm\"")
|
||||
require.Contains(t, r.PermissionScope, "aws:read")
|
||||
}
|
||||
|
||||
// #5: GCP SA JWT —— RS256 断言可被对应公钥验签,且 claims 正确。
|
||||
func TestBuildGCPAssertion_RS256Roundtrip(t *testing.T) {
|
||||
key, err := rsa.GenerateKey(rand.Reader, 2048)
|
||||
require.NoError(t, err)
|
||||
der, err := x509.MarshalPKCS8PrivateKey(key)
|
||||
require.NoError(t, err)
|
||||
pemStr := string(pem.EncodeToMemory(&pem.Block{Type: "PRIVATE KEY", Bytes: der}))
|
||||
|
||||
cred := gcpDiscoveryCredentials{
|
||||
ClientEmail: "svc@proj.iam.gserviceaccount.com",
|
||||
PrivateKey: pemStr,
|
||||
ProjectID: "proj",
|
||||
}
|
||||
tm := time.Date(2026, 6, 10, 0, 0, 0, 0, time.UTC)
|
||||
jwt, err := buildGCPAssertion(cred, "https://www.googleapis.com/auth/cloud-platform", "https://oauth2.googleapis.com/token", tm)
|
||||
require.NoError(t, err)
|
||||
|
||||
parts := strings.Split(jwt, ".")
|
||||
require.Len(t, parts, 3)
|
||||
|
||||
// 验签:signingInput = header.payload
|
||||
signingInput := parts[0] + "." + parts[1]
|
||||
digest := sha256.Sum256([]byte(signingInput))
|
||||
sig, err := base64.RawURLEncoding.DecodeString(parts[2])
|
||||
require.NoError(t, err)
|
||||
require.NoError(t, rsa.VerifyPKCS1v15(&key.PublicKey, crypto.SHA256, digest[:], sig), "RS256 签名应可被公钥验证")
|
||||
|
||||
// header alg + claims
|
||||
hdr, err := base64.RawURLEncoding.DecodeString(parts[0])
|
||||
require.NoError(t, err)
|
||||
require.Contains(t, string(hdr), "RS256")
|
||||
claims, err := base64.RawURLEncoding.DecodeString(parts[1])
|
||||
require.NoError(t, err)
|
||||
require.Contains(t, string(claims), "svc@proj.iam.gserviceaccount.com")
|
||||
require.Contains(t, string(claims), "cloud-platform")
|
||||
}
|
||||
@@ -0,0 +1,236 @@
|
||||
package controller
|
||||
|
||||
import (
|
||||
"crypto"
|
||||
"crypto/rand"
|
||||
"crypto/rsa"
|
||||
"crypto/sha256"
|
||||
"crypto/x509"
|
||||
"encoding/base64"
|
||||
"encoding/pem"
|
||||
"errors"
|
||||
"fmt"
|
||||
"io"
|
||||
"net/http"
|
||||
"net/url"
|
||||
"strings"
|
||||
"time"
|
||||
|
||||
"github.com/heicode/manager/common"
|
||||
"github.com/heicode/manager/model"
|
||||
)
|
||||
|
||||
// GCP resource discovery (#5). Pure-Go service-account flow (no GCP SDK):
|
||||
// build an RS256-signed JWT assertion from the SA private key, exchange it for an
|
||||
// OAuth2 access token, then call Cloud Asset Inventory searchAllResources (REST)
|
||||
// to enumerate the project's resources. Normalizes assets to discoveredCloudResource.
|
||||
//
|
||||
// Creds (from cloud_account metadata/secret): client_email, private_key (PEM),
|
||||
// project_id. token_uri optional (defaults to Google's).
|
||||
|
||||
type gcpCloudDiscoveryProvider struct{}
|
||||
|
||||
func (gcpCloudDiscoveryProvider) name() string { return "gcp" }
|
||||
|
||||
func (gcpCloudDiscoveryProvider) discover(account model.ResourceBinding, creds map[string]any) ([]discoveredCloudResource, error) {
|
||||
if mapString(creds, "project_id") == "" && strings.TrimSpace(account.ExternalId) != "" {
|
||||
creds["project_id"] = account.ExternalId
|
||||
}
|
||||
return discoverGCP(creds, time.Now(), &http.Client{Timeout: 20 * time.Second})
|
||||
}
|
||||
|
||||
type gcpDiscoveryCredentials struct {
|
||||
ClientEmail string
|
||||
PrivateKey string // PEM
|
||||
ProjectID string
|
||||
TokenURI string
|
||||
}
|
||||
|
||||
func (c gcpDiscoveryCredentials) validate() error {
|
||||
if strings.TrimSpace(c.ClientEmail) == "" {
|
||||
return errors.New("GCP client_email required")
|
||||
}
|
||||
if strings.TrimSpace(c.PrivateKey) == "" {
|
||||
return errors.New("GCP private_key required")
|
||||
}
|
||||
if strings.TrimSpace(c.ProjectID) == "" {
|
||||
return errors.New("GCP project_id required")
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
func b64url(b []byte) string { return base64.RawURLEncoding.EncodeToString(b) }
|
||||
|
||||
// parseRSAPrivateKeyPEM parses a PEM private key (PKCS#8 or PKCS#1) into an RSA key.
|
||||
func parseRSAPrivateKeyPEM(pemStr string) (*rsa.PrivateKey, error) {
|
||||
block, _ := pem.Decode([]byte(strings.TrimSpace(pemStr)))
|
||||
if block == nil {
|
||||
return nil, errors.New("GCP private_key is not valid PEM")
|
||||
}
|
||||
if key, err := x509.ParsePKCS1PrivateKey(block.Bytes); err == nil {
|
||||
return key, nil
|
||||
}
|
||||
keyAny, err := x509.ParsePKCS8PrivateKey(block.Bytes)
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("GCP private_key parse failed: %w", err)
|
||||
}
|
||||
rsaKey, ok := keyAny.(*rsa.PrivateKey)
|
||||
if !ok {
|
||||
return nil, errors.New("GCP private_key is not an RSA key")
|
||||
}
|
||||
return rsaKey, nil
|
||||
}
|
||||
|
||||
// buildGCPAssertion builds the RS256-signed JWT assertion for the OAuth2 token
|
||||
// exchange. Pure & deterministic given t — unit-testable with a generated key.
|
||||
func buildGCPAssertion(cred gcpDiscoveryCredentials, scope string, aud string, t time.Time) (string, error) {
|
||||
key, err := parseRSAPrivateKeyPEM(cred.PrivateKey)
|
||||
if err != nil {
|
||||
return "", err
|
||||
}
|
||||
header, err := common.Marshal(map[string]any{"alg": "RS256", "typ": "JWT"})
|
||||
if err != nil {
|
||||
return "", err
|
||||
}
|
||||
iat := t.UTC().Unix()
|
||||
claims, err := common.Marshal(map[string]any{
|
||||
"iss": cred.ClientEmail,
|
||||
"scope": scope,
|
||||
"aud": aud,
|
||||
"iat": iat,
|
||||
"exp": iat + 3600,
|
||||
})
|
||||
if err != nil {
|
||||
return "", err
|
||||
}
|
||||
signingInput := b64url(header) + "." + b64url(claims)
|
||||
digest := sha256.Sum256([]byte(signingInput))
|
||||
sig, err := rsa.SignPKCS1v15(rand.Reader, key, crypto.SHA256, digest[:])
|
||||
if err != nil {
|
||||
return "", err
|
||||
}
|
||||
return signingInput + "." + b64url(sig), nil
|
||||
}
|
||||
|
||||
func gcpAccessToken(cred gcpDiscoveryCredentials, t time.Time, httpClient *http.Client) (string, error) {
|
||||
tokenURI := strings.TrimSpace(cred.TokenURI)
|
||||
if tokenURI == "" {
|
||||
tokenURI = "https://oauth2.googleapis.com/token"
|
||||
}
|
||||
assertion, err := buildGCPAssertion(cred, "https://www.googleapis.com/auth/cloud-platform", tokenURI, t)
|
||||
if err != nil {
|
||||
return "", err
|
||||
}
|
||||
form := url.Values{}
|
||||
form.Set("grant_type", "urn:ietf:params:oauth:grant-type:jwt-bearer")
|
||||
form.Set("assertion", assertion)
|
||||
req, err := http.NewRequest(http.MethodPost, tokenURI, strings.NewReader(form.Encode()))
|
||||
if err != nil {
|
||||
return "", err
|
||||
}
|
||||
req.Header.Set("Content-Type", "application/x-www-form-urlencoded")
|
||||
resp, err := httpClient.Do(req)
|
||||
if err != nil {
|
||||
return "", fmt.Errorf("GCP token exchange failed: %w", err)
|
||||
}
|
||||
defer resp.Body.Close()
|
||||
raw, _ := io.ReadAll(io.LimitReader(resp.Body, 1<<20))
|
||||
if resp.StatusCode < http.StatusOK || resp.StatusCode >= http.StatusMultipleChoices {
|
||||
return "", fmt.Errorf("GCP token exchange failed (%d): %s", resp.StatusCode, strings.TrimSpace(string(raw)))
|
||||
}
|
||||
var payload struct {
|
||||
AccessToken string `json:"access_token"`
|
||||
}
|
||||
if err := common.Unmarshal(raw, &payload); err != nil {
|
||||
return "", err
|
||||
}
|
||||
if strings.TrimSpace(payload.AccessToken) == "" {
|
||||
return "", errors.New("GCP token response missing access_token")
|
||||
}
|
||||
return payload.AccessToken, nil
|
||||
}
|
||||
|
||||
type gcpAsset struct {
|
||||
Name string `json:"name"`
|
||||
AssetType string `json:"assetType"`
|
||||
DisplayName string `json:"displayName"`
|
||||
Location string `json:"location"`
|
||||
}
|
||||
|
||||
type gcpSearchResponse struct {
|
||||
Results []gcpAsset `json:"results"`
|
||||
NextPageToken string `json:"nextPageToken"`
|
||||
}
|
||||
|
||||
// gcpSearchAllResourcesURL 构造 Cloud Asset Inventory searchAllResources 端点。
|
||||
// #5 复审 #3:scope 里的 slash 是路径模板的一部分,**不能** PathEscape 成 %2F —— 正确形如
|
||||
// /v1/projects/<id>:searchAllResources。projectID 仍按单段转义,pageToken 走 query 转义。
|
||||
func gcpSearchAllResourcesURL(projectID, pageToken string) string {
|
||||
endpoint := "https://cloudasset.googleapis.com/v1/projects/" + url.PathEscape(projectID) + ":searchAllResources?pageSize=500"
|
||||
if strings.TrimSpace(pageToken) != "" {
|
||||
endpoint += "&pageToken=" + url.QueryEscape(pageToken)
|
||||
}
|
||||
return endpoint
|
||||
}
|
||||
|
||||
func discoverGCP(creds map[string]any, now time.Time, httpClient *http.Client) ([]discoveredCloudResource, error) {
|
||||
cred := gcpDiscoveryCredentials{
|
||||
ClientEmail: mapString(creds, "client_email"),
|
||||
PrivateKey: mapString(creds, "private_key"),
|
||||
ProjectID: mapString(creds, "project_id"),
|
||||
TokenURI: mapString(creds, "token_uri"),
|
||||
}
|
||||
if err := cred.validate(); err != nil {
|
||||
return nil, err
|
||||
}
|
||||
token, err := gcpAccessToken(cred, now, httpClient)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
|
||||
out := []discoveredCloudResource{}
|
||||
pageToken := ""
|
||||
for page := 0; page < 50; page++ {
|
||||
endpoint := gcpSearchAllResourcesURL(cred.ProjectID, pageToken)
|
||||
req, err := http.NewRequest(http.MethodGet, endpoint, nil)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
req.Header.Set("Authorization", "Bearer "+token)
|
||||
resp, err := httpClient.Do(req)
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("GCP searchAllResources failed: %w", err)
|
||||
}
|
||||
raw, _ := io.ReadAll(io.LimitReader(resp.Body, 8<<20))
|
||||
resp.Body.Close()
|
||||
if resp.StatusCode < http.StatusOK || resp.StatusCode >= http.StatusMultipleChoices {
|
||||
return nil, fmt.Errorf("GCP searchAllResources failed (%d): %s", resp.StatusCode, strings.TrimSpace(string(raw)))
|
||||
}
|
||||
var payload gcpSearchResponse
|
||||
if err := common.Unmarshal(raw, &payload); err != nil {
|
||||
return nil, err
|
||||
}
|
||||
for _, a := range payload.Results {
|
||||
if strings.TrimSpace(a.Name) == "" {
|
||||
continue
|
||||
}
|
||||
name := strings.TrimSpace(a.DisplayName)
|
||||
if name == "" {
|
||||
name = lastPathSegment(a.Name)
|
||||
}
|
||||
out = append(out, discoveredCloudResource{
|
||||
ExternalId: a.Name,
|
||||
Name: name,
|
||||
NativeType: a.AssetType, // e.g. compute.googleapis.com/Instance
|
||||
Location: a.Location,
|
||||
BindingScope: "gcp:" + a.Name,
|
||||
Metadata: map[string]any{"project_id": cred.ProjectID, "asset_type": a.AssetType},
|
||||
})
|
||||
}
|
||||
pageToken = strings.TrimSpace(payload.NextPageToken)
|
||||
if pageToken == "" {
|
||||
break
|
||||
}
|
||||
}
|
||||
return out, nil
|
||||
}
|
||||
@@ -1,6 +1,9 @@
|
||||
package model
|
||||
|
||||
import "errors"
|
||||
import (
|
||||
"errors"
|
||||
"strings"
|
||||
)
|
||||
|
||||
type AgentCallbackEvent struct {
|
||||
Id int `gorm:"primaryKey" json:"id"`
|
||||
@@ -55,6 +58,41 @@ func InsertAgentCallbackEvent(row *AgentCallbackEvent) (bool, error) {
|
||||
return true, nil
|
||||
}
|
||||
|
||||
// ListSwarmCallbackEventsAfter returns persisted swarm runtime events for one run
|
||||
// (matched by deployment_id OR swarm_id) with an id-based `after` cursor for
|
||||
// incremental polling (#45 events?after). Ordered oldest-first so the client can
|
||||
// append; the caller uses the last returned Id as the next `after`. Reading from
|
||||
// HM-persisted callback rows means this needs no live Swarm call.
|
||||
func ListSwarmCallbackEventsAfter(userID, deploymentID, swarmID string, afterID, limit int) ([]AgentCallbackEvent, error) {
|
||||
if DB == nil {
|
||||
return nil, nil
|
||||
}
|
||||
q := DB.Model(&AgentCallbackEvent{})
|
||||
switch {
|
||||
case deploymentID != "" && swarmID != "":
|
||||
q = q.Where("deployment_id = ? OR swarm_id = ?", deploymentID, swarmID)
|
||||
case deploymentID != "":
|
||||
q = q.Where("deployment_id = ?", deploymentID)
|
||||
case swarmID != "":
|
||||
q = q.Where("swarm_id = ?", swarmID)
|
||||
default:
|
||||
return nil, nil
|
||||
}
|
||||
// 防跨用户泄漏:即使 runtime_swarm_id/deployment_id 碰撞或误写,也按 user_id 收口(#45 复审 #3)。
|
||||
if strings.TrimSpace(userID) != "" {
|
||||
q = q.Where("user_id = ?", userID)
|
||||
}
|
||||
if afterID > 0 {
|
||||
q = q.Where("id > ?", afterID)
|
||||
}
|
||||
if limit <= 0 || limit > 1000 {
|
||||
limit = 200
|
||||
}
|
||||
var items []AgentCallbackEvent
|
||||
err := q.Order("id asc").Limit(limit).Find(&items).Error
|
||||
return items, err
|
||||
}
|
||||
|
||||
func ListAgentCallbackEvents(f ListAgentCallbackEventsFilter) ([]AgentCallbackEvent, error) {
|
||||
if DB == nil {
|
||||
return nil, nil
|
||||
|
||||
@@ -0,0 +1,50 @@
|
||||
package model
|
||||
|
||||
import (
|
||||
"testing"
|
||||
|
||||
"github.com/stretchr/testify/require"
|
||||
)
|
||||
|
||||
// #45: events?after 游标 + user 作用域 —— 按 deployment_id/swarm_id 过滤,id>after 增量,oldest-first;
|
||||
// 传入 userID 时按 user_id 收口(防跨用户泄漏,复审 #3)。
|
||||
func TestListSwarmCallbackEventsAfter(t *testing.T) {
|
||||
require.NoError(t, LOG_DB.Where("1 = 1").Delete(&AgentCallbackEvent{}).Error)
|
||||
mk := func(eventID, uid, dep, swarm, etype string) {
|
||||
_, err := InsertAgentCallbackEvent(&AgentCallbackEvent{
|
||||
EventID: eventID, UserID: uid, DeploymentID: dep, SwarmID: swarm, EventType: etype,
|
||||
})
|
||||
require.NoError(t, err)
|
||||
}
|
||||
mk("e1", "7", "dep_A", "swarm_A", "deployment.status_changed")
|
||||
mk("e2", "7", "dep_A", "swarm_A", "task.created")
|
||||
mk("e3", "7", "dep_A", "swarm_A", "artifact.produced")
|
||||
mk("e4", "7", "dep_B", "swarm_B", "task.created") // 另一个 run
|
||||
mk("e5", "9", "dep_A", "swarm_A", "task.created") // 同 dep/swarm 但别的用户 → 不应泄漏给 user 7
|
||||
|
||||
// user 7 + dep_A:3 条(e5 属 user 9,被排除),oldest-first
|
||||
all, err := ListSwarmCallbackEventsAfter("7", "dep_A", "swarm_A", 0, 100)
|
||||
require.NoError(t, err)
|
||||
require.Len(t, all, 3)
|
||||
require.Equal(t, "e1", all[0].EventID)
|
||||
for _, e := range all {
|
||||
require.NotEqual(t, "e5", e.EventID, "不得返回别的用户的事件")
|
||||
}
|
||||
|
||||
// 游标:after = 第一条 id → 其后 2 条
|
||||
rest, err := ListSwarmCallbackEventsAfter("7", "dep_A", "swarm_A", all[0].Id, 100)
|
||||
require.NoError(t, err)
|
||||
require.Len(t, rest, 2)
|
||||
require.Equal(t, "e2", rest[0].EventID)
|
||||
|
||||
// user 9 只看到自己的 e5
|
||||
u9, err := ListSwarmCallbackEventsAfter("9", "dep_A", "swarm_A", 0, 100)
|
||||
require.NoError(t, err)
|
||||
require.Len(t, u9, 1)
|
||||
require.Equal(t, "e5", u9[0].EventID)
|
||||
|
||||
// 空标识 → 空
|
||||
none, err := ListSwarmCallbackEventsAfter("7", "", "", 0, 100)
|
||||
require.NoError(t, err)
|
||||
require.Empty(t, none)
|
||||
}
|
||||
@@ -45,6 +45,8 @@ func TestMain(m *testing.M) {
|
||||
&UserSubscription{},
|
||||
&TelemetryEvent{},
|
||||
&PreflightConfirmation{},
|
||||
&AgentDeployment{},
|
||||
&AgentCallbackEvent{},
|
||||
); err != nil {
|
||||
panic("failed to migrate: " + err.Error())
|
||||
}
|
||||
|
||||
@@ -225,6 +225,9 @@ func SetApiRouter(router *gin.Engine) {
|
||||
resourceRoute.POST("/", controller.CreateResource)
|
||||
resourceRoute.POST("/:id/secret", controller.UpsertResourceSecret)
|
||||
resourceRoute.POST("/:id/azure/discover", controller.DiscoverAzureResources)
|
||||
// Provider-agnostic cloud discovery (#5): routes by the account's provider
|
||||
// (azure/aws/gcp). The azure-specific route above is kept for back-compat.
|
||||
resourceRoute.POST("/:id/discover-cloud", controller.DiscoverCloudResources)
|
||||
resourceRoute.PUT("/:id", controller.UpdateResource)
|
||||
resourceRoute.DELETE("/:id", controller.DeleteResource)
|
||||
}
|
||||
@@ -548,6 +551,13 @@ func SetApiRouter(router *gin.Engine) {
|
||||
// Client error-telemetry ingest (#24). Device-paired; never bills.
|
||||
// Gated by HEICODE_TELEMETRY_ENABLED (default off -> 410 kill switch).
|
||||
heicodeAgentRoute.POST("/telemetry/events", controller.HeicodeTelemetryEvents)
|
||||
// Swarm Run query (#45 Phase1). Read-only views served from HM-persisted
|
||||
// callback data (no live Swarm call); stop is gated pending agent_swarm#2 freeze.
|
||||
heicodeAgentRoute.GET("/swarms", controller.HeicodeListSwarms)
|
||||
heicodeAgentRoute.GET("/swarms/:id", controller.HeicodeGetSwarmStatus)
|
||||
heicodeAgentRoute.GET("/swarms/:id/events", controller.HeicodeListSwarmEvents)
|
||||
heicodeAgentRoute.GET("/swarms/:id/artifacts", controller.HeicodeListSwarmArtifacts)
|
||||
heicodeAgentRoute.POST("/swarms/:id/stop", controller.HeicodeStopSwarm)
|
||||
}
|
||||
|
||||
// Client↔agent access control (HM-provided, AM-optional). Called by the
|
||||
|
||||
Reference in New Issue
Block a user