diff --git a/docs/heicode-manager-sub-swarm-progress-checklist.md b/docs/heicode-manager-sub-swarm-progress-checklist.md index 91f6862f..ac6bff00 100644 --- a/docs/heicode-manager-sub-swarm-progress-checklist.md +++ b/docs/heicode-manager-sub-swarm-progress-checklist.md @@ -70,16 +70,16 @@ Heicode 桌面客户端 | 优先级 | 任务 | 当前缺口 | 是否 Manager 可独立做 | 验收标准 | |---|---|---|---|---| -| P0 | 把 `/api/swarms` adapter 文档化并固定字段 | 代码已有入口，但文档要明确它和 Runtime `POST /api/swarms` / `/api/agnet/deployments` 的映射 | 是 | 文档写清 request/response、source、`deployment_id <-> swarm_id` 映射 | -| P0 | 增加 Swarm Run 显示字段 | 页面需要明确展示 `runtime_deployment_id`、`runtime_swarm_id`、`runtime_state`、source | 是 | 用户态 deployment 详情和后台页面能看到 Runtime 映射 | -| P0 | 回调事件类型收敛 | 目前统一 `swarm-events` 已能接，但需要按蜂群事件标准列出必填字段和展示规则 | 是 | `task.created/claimed/running/completed/failed/handoff/approval/artifact` 都有 schema | -| P0 | Runtime 联调配置模板 | 需要把 `AGNET_RUNTIME_*` 配置、token、callback URL、签名方式整理给蜂群侧 | 是 | 蜂群项目按模板能调用 Manager callback | +| P0 | 把 `/api/swarms` adapter 文档化并固定字段 | 已完成：`docs/integration/蜂群模式-AgentManager对接任务清单.md` 和 `docs/integration/普通sub敏捷模式-AgentManager对接任务清单.md` 已写清 Runtime create、callback、source、`deployment_id <-> swarm_id` 映射 | 是 | 文档可直接发给 Agent Manager / 蜂群侧核对 | +| P0 | 增加 Swarm Run 显示字段 | 已完成：用户态 deployment 详情展示 `runtime_deployment_id`、`runtime_swarm_id`、`runtime_state`，并在相关记录里展示 source | 是 | 用户态 deployment 详情和后台页面能看到 Runtime 映射 | +| P0 | 回调事件类型收敛 | 已完成：`GET /api/agnet/callbacks/swarm-events/schema` 输出事件类型、分类和必填字段；callback 接收端按 schema 校验关键 task/handoff/artifact/approval 字段 | 是 | `task.created/claimed/running/completed/failed/handoff/approval/artifact` 都有 schema | +| P0 | Runtime 联调配置模板 | 已完成：两份 Agent Manager 对接任务清单已写清 `AGNET_RUNTIME_*`、callback URL、service token/HMAC 方式和验收步骤 | 是 | 蜂群项目按模板能调用 Manager callback | | P1 | 审批结果回传 Runtime 联调 | Manager adapter 已有；仍需要 Runtime 提供接收接口并验证状态继续/停止 | 需要 Runtime 接口 | 审批通过/拒绝后 Runtime 状态能继续或停止 | -| P1 | Artifact 展示优化 | 已能在页面展示 artifact 类型和 URI；仍需等 Runtime 输出真实 code_patch/document/test_report/deployment_manifest | 需要 Runtime 数据 | artifact 页面/详情能按类型展示摘要和链接 | -| P1 | 任务图/Agent 状态展示占位 | Manager 还没有真实 task graph/claim/heartbeat 页面结构 | Manager 可先做展示结构，真实数据需 Runtime | 有空态和字段，不宣称真实已运行 | -| P1 | 日志/指标真实来源标识 | deployment 详情、timeline 已强化 runtime/simulated 来源；logs/metrics 仍需 Runtime 真实数据 | 需要 Runtime 数据 | 页面和 API 响应能区分来源 | -| P1 | 高危审批客户端联动文档 | Manager 有 API，桌面客户端需要知道如何轮询/展示/提交审批 | 是 | 客户端文档补齐 approval flow | -| P2 | 蜂群模式验收脚本 | Manager 已有默认模拟链路和定向单测；还需要整理生产 curl 脚本 | 是 | 本地/生产能跑出 callback、artifact、approval、timeline 可见 | +| P1 | Artifact 展示优化 | Manager 端已完成：页面展示 artifact 类型、摘要和 URI；真实 `code_patch/document/test_report/deployment_manifest` 仍需 Runtime 输出 | 需要 Runtime 数据 | artifact 页面/详情能按类型展示摘要和链接 | +| P1 | 任务图/Agent 状态展示占位 | 已完成：页面从 `task.*` / `handoff.*` callback 聚合 Agent task map；无真实数据时显示 Runtime callback 空态 | Manager 可先做展示结构，真实数据需 Runtime | 有空态和字段，不宣称真实已运行 | +| P1 | 日志/指标真实来源标识 | 已完成：logs/metrics API 返回 `data_source`、`runtime_source`，当前明确是 Manager control-plane / estimated，不伪装 Runtime 真实指标 | 需要 Runtime 数据 | 页面和 API 响应能区分来源 | +| P1 | 高危审批客户端联动文档 | 已完成：`docs/integration/heicode-desktop-sub-agile-api.md` 已包含 approval 查询、approve/reject、awaiting_approval 流程 | 是 | 客户端文档补齐 approval flow | +| P2 | 蜂群模式验收脚本 | 已完成：`scripts/agnet_sub_mode_smoke.py` 支持 schema 检查、生产健康检查、可选 simulate-events、可选真实 callback smoke | 是 | 本地/生产能跑出 callback、artifact、approval、timeline 可见 | ## 五、需要蜂群项目配合的事项 diff --git a/docs/integration/AgentManager蜂群Runtime接口实现要求.md b/docs/integration/AgentManager蜂群Runtime接口实现要求.md new file mode 100644 index 00000000..9b4a8296 --- /dev/null +++ b/docs/integration/AgentManager蜂群Runtime接口实现要求.md @@ -0,0 +1,624 @@ +# Agent Manager 蜂群 Runtime 接口实现要求 + +更新时间：2026-05-28 +面向对象：Agent Manager / HeiCode-Swarm Runtime 开发负责人 +用途：Agent Manager 按本文实现接口、字段和回调后，Heicode Manager 可直接联调蜂群模式。 + +## 1. 结论 + +Heicode Manager 已负责控制面和记录面： + +- 生成 deployment / swarm 创建请求。 +- 传递 `resource_grants`、`secret_ref`、预算、模型网关上下文和 callback 地址。 +- 保存 `deployment_id <-> runtime_deployment_id <-> swarm_id` 映射。 +- 接收 Runtime callback，落库 artifact / approval / timeline / audit。 +- 用户 approve/reject 后，把审批决定回传 Runtime。 + +Agent Manager / Swarm Runtime 需要负责执行层： + +- 创建真实 Swarm Run。 +- 生成真实 task graph。 +- 管理 Agent claim / heartbeat / handoff / retry / blocked / failed 状态机。 +- 真实执行任务并产出 artifact。 +- 在高危动作前暂停并回调审批请求。 +- 接收审批结果后继续或停止。 +- 回传日志、指标、用量和最终结果。 + +## 2. 总体调用链 + +```text +Heicode Desktop Client + -> Heicode Manager + POST /api/swarms 或 /api/agnet/user/deployments + -> Agent Manager / Swarm Runtime + POST /api/swarms + <- Runtime response + runtime_deployment_id / swarm_id / status + <- Runtime callback + POST /api/agnet/callbacks/swarm-events + -> Runtime approval decision + POST /api/swarms/{swarm_id}/approvals/{approval_id} +``` + +## 3. Agent Manager 必须提供的接口 + +| 优先级 | 方法 | 路径 | 必须 | 用途 | +|---|---|---|---:|---| +| P0 | `GET` | `/api/agnet/health` | 是 | 健康检查 | +| P0 | `POST` | `/api/swarms` | 是 | 创建真实 Swarm Run | +| P0 | `POST` | `/api/swarms/{swarm_id}/stop` | 是 | 停止 Swarm Run | +| P0 | `POST` | `/api/swarms/{swarm_id}/approvals/{approval_id}` | 是 | 接收 Manager 审批决定 | +| P0 | callback | Manager `/api/agnet/callbacks/swarm-events` | 是 | 回写状态、task、handoff、artifact、approval | +| P1 | `GET` | `/api/swarms/{swarm_id}` | 建议 | 查询 Runtime 详情 | +| P1 | `GET` | `/api/swarms/{swarm_id}/tasks` | 建议 | 查询 task graph | +| P1 | `GET` | `/api/swarms/{swarm_id}/logs` | 建议 | 查询日志 | +| P1 | `GET` | `/api/swarms/{swarm_id}/metrics` | 建议 | 查询指标 | + +如果短期无法提供 `/api/swarms`，可以临时确认兼容路径，例如 `/tasks`。但这只能作为联调过渡，不作为最终生产契约。 + +## 4. 认证与公共 Header + +Manager 调 Agent Manager 时携带： + +```http +Authorization: Bearer +X-User-ID: +X-Binding-Scope: +X-Correlation-ID: +X-Idempotency-Key: manager- +Content-Type: application/json +``` + +要求： + +1. `Authorization` 用于服务间鉴权。 +2. `X-Correlation-ID` 必须贯穿 create、callback、approval decision、logs、metrics。 +3. `X-Idempotency-Key` 必须支持幂等；重复创建请求不能生成多个真实 Swarm Run。 +4. Agent Manager 不得要求 Manager 传长期明文密钥。 + +## 5. `GET /api/agnet/health` + +### 响应 + +```json +{ + "success": true, + "data": { + "status": "healthy", + "service": "agent-manager-swarm-runtime", + "version": "1.0.0", + "runtime": "aks", + "time": "2026-05-28T10:00:00Z" + } +} +``` + +验收： + +- HTTP 200。 +- `status` 为 `healthy` / `ok` / `up` 之一。 +- 不返回密钥、Token、连接串。 + +## 6. `POST /api/swarms` + +### 请求体 + +```json +{ + "orchestration_plan": { + "intent_id": "task_123", + "template_hint": "heicode-task", + "objective": "完成本轮用户目标", + "sub_mode": "agile", + "risk_level": "medium", + "budget": { + "max_tokens": 120000, + "max_cost_usd": 8, + "max_duration_sec": 3600 + }, + "user_context": { + "user_id": "22", + "channel_id": "heicode", + "binding_scope": "task-task-123" + }, + "billing_context": { + "provider": "newapi", + "default_model_id": "model_xxx", + "allowed_model_ids": ["model_xxx"], + "secret_ref": "azkv://heicode-kv.vault.azure.net/secrets/model-gateway-key" + }, + "agile_context": { + "iteration": "2026-05-28", + "stage": "development", + "checkpoint": "draft_created", + "acceptance_criteria": [ + "接口返回成功", + "artifact 可回写到 timeline", + "不出现明文密钥" + ], + "next_action": "continue", + "requires_user_approval": false + }, + "agents": [ + { + "role_template": "backend", + "goal": "完成后端实现和测试", + "default_model_id": "model_xxx", + "resource_grants": [] + } + ], + "resource_grants": [] + }, + "agents": [ + { + "role": "backend", + "resource_grants": [] + } + ], + "resource_grants": [ + { + "grant_id": "grant-task-123-git", + "resource_id": "repo-main", + "resource_type": "git", + "permission_scope": ["repo:read", "repo:write:feature-branches"], + "secret_ref": "azkv://heicode-kv.vault.azure.net/secrets/repo-main", + "ref": "azkv://heicode-kv.vault.azure.net/secrets/repo-main", + "target_role": "backend", + "constraints": { + "allowed_paths": "heicode/**" + }, + "metadata": { + "repo": "heicode-manager" + } + } + ], + "callback": { + "url": "https://code.xinghanlab.com/api/agnet/callbacks/swarm-events", + "signing_secret_ref": "azkv://heicode-kv.vault.azure.net/secrets/agnet-callback-signing-key", + "subscribed_events": [ + "deployment.status_changed", + "task.created", + "task.claimed", + "task.heartbeat", + "task.blocked", + "task.retried", + "task.failed", + "task.completed", + "handoff.requested", + "handoff.completed", + "approval.requested", + "artifact.created", + "timeline.updated", + "budget.alert" + ] + }, + "metadata": { + "manager_deployment_id": "dep_xxx", + "heicode_deployment_id": "dep_xxx", + "heicode_runtime_bridge": true, + "correlation_id": "corr_xxx" + } +} +``` + +### Agent Manager 必须消费的字段 + +| 字段 | 必须 | 说明 | +|---|---:|---| +| `orchestration_plan.objective` | 是 | 用户目标 | +| `orchestration_plan.sub_mode` | 是 | `agile` / `waterfall`；蜂群执行时也要保留此组织方式 | +| `orchestration_plan.risk_level` | 是 | 高危动作必须走审批 | +| `orchestration_plan.budget` | 是 | token / cost / duration 上限 | +| `billing_context.provider` | 是 | 当前为 `newapi` | +| `billing_context.secret_ref` | 是 | 模型网关密钥引用，只能是 `azkv://...` | +| `agents[].role` 或 `role_template` | 是 | 子 Agent 角色 | +| `resource_grants[]` | 是 | 资源授权清单 | +| `resource_grants[].secret_ref` | 凭据资源必填 | 只允许 Key Vault 引用，不允许明文 | +| `callback.url` | 是 | Runtime 回调 Manager 的地址 | +| `callback.signing_secret_ref` | 建议 | HMAC 签名密钥引用 | +| `metadata.manager_deployment_id` | 是 | Manager 侧 deployment id | +| `metadata.correlation_id` | 是 | 全链路追踪 | + +### 响应 + +```json +{ + "success": true, + "data": { + "deployment_id": "runtime-dep-123", + "swarm_id": "swarm-123", + "status": "running", + "created_at": "2026-05-28T10:00:00Z", + "estimated_ready_at": "2026-05-28T10:02:00Z" + } +} +``` + +兼容响应： + +```json +{ + "deployment_id": "runtime-dep-123", + "swarm_id": "swarm-123", + "status": "running" +} +``` + +Manager 映射规则： + +| Agent Manager 返回字段 | Manager 保存字段 | +|---|---| +| `data.deployment_id` / `deployment_id` / `id` | `runtime_deployment_id` | +| `data.swarm_id` / `swarm_id` / `runtime_swarm_id` | `runtime_swarm_id` | +| `data.status` / `status` / `runtime_status` | `runtime_state` | + +## 7. Runtime 必须生成的 task graph + +Agent Manager 创建 Swarm Run 后，必须在 Runtime 内部生成任务图，并通过 callback 回写。 + +每个 task 至少包含： + +```json +{ + "task_id": "task-backend-1", + "title": "实现后端接口", + "description": "完成 API、校验和测试", + "agent_role": "backend", + "status": "pending", + "depends_on": ["task-design-1"], + "attempt": 1 +} +``` + +状态建议： + +| status | 说明 | +|---|---| +| `pending` | 等待执行 | +| `claimed` | 已被 Agent 领取 | +| `running` | 执行中 | +| `blocked` | 阻塞 | +| `handoff_requested` | 等待交接 | +| `retrying` | 重试中 | +| `completed` | 完成 | +| `failed` | 失败 | + +## 8. Runtime 回调 Manager + +Manager 回调地址： + +```http +POST https://code.xinghanlab.com/api/agnet/callbacks/swarm-events +X-Agnet-Service-Token: +X-Agnet-Event-Id: +X-Correlation-ID: +Content-Type: application/json +``` + +也支持 HMAC： + +```http +X-Agnet-Timestamp: +X-Agnet-Signature: sha256= +``` + +HMAC 签名内容： + +```text +timestamp + "." + event_id + "." + raw_body +``` + +联调前可拉取 Manager 当前接受的事件 schema： + +```http +GET https://code.xinghanlab.com/api/agnet/callbacks/swarm-events/schema +``` + +### 通用 callback envelope + +```json +{ + "event_id": "evt_123", + "idempotency_key": "evt_123", + "event_type": "task.claimed", + "deployment_id": "dep_xxx", + "swarm_id": "swarm-123", + "agent_instance_id": "agent-backend-1", + "task_id": "task-backend-1", + "occurred_at": "2026-05-28T10:10:00Z", + "correlation_id": "corr_xxx", + "source": "agent-manager-runtime", + "payload": {} +} +``` + +要求： + +1. `event_id` 全局唯一。 +2. 重试同一事件必须复用相同 `event_id` 或 `idempotency_key`。 +3. `deployment_id` 优先使用 Manager deployment id。 +4. 如果只知道 `swarm_id`，Manager 也会按 `runtime_swarm_id` 查找 deployment。 +5. `source` 要能区分真实 Runtime，例如 `agent-manager-runtime`，不要写 `simulated`。 + +## 9. 必须回调的事件和字段 + +| event_type | 必填字段 | 说明 | +|---|---|---| +| `deployment.status_changed` | `payload.status` | Runtime 整体状态变化 | +| `task.created` | `task_id`, `payload.title` | 任务图新增任务 | +| `task.claimed` | `task_id`, `payload.agent_role` | Agent 领取任务 | +| `task.running` | `task_id`, `payload.agent_role` | Agent 开始执行 | +| `task.heartbeat` | `task_id`, `payload.agent_role` | Agent 心跳 | +| `task.blocked` | `task_id`, `payload.reason` | 任务阻塞 | +| `task.retried` | `task_id`, `payload.attempt` | 任务重试 | +| `task.failed` | `task_id`, `payload.reason` | 任务失败 | +| `task.completed` | `task_id` | 任务完成 | +| `handoff.requested` | `task_id`, `payload.from_role`, `payload.to_role` | 请求交接 | +| `handoff.completed` | `task_id`, `payload.from_role`, `payload.to_role` | 完成交接 | +| `artifact.created` | `artifact.artifact_id` 或 `payload.artifact_id` | 产物生成 | +| `approval.requested` | `payload.approval_id`, `payload.operation`, `payload.risk_level` | 高危审批 | +| `timeline.updated` | `payload.title` | 用户可见时间线 | +| `budget.alert` | `payload.threshold_pct` | 预算告警 | + +## 10. artifact.created + +Runtime 产生中间产物或最终交付物时回调： + +```json +{ + "event_id": "evt_artifact_1", + "event_type": "artifact.created", + "deployment_id": "dep_xxx", + "swarm_id": "swarm-123", + "task_id": "task-backend-1", + "source": "agent-manager-runtime", + "artifact": { + "artifact_id": "art_backend_patch_1", + "artifact_type": "code_patch", + "title": "Backend API patch", + "summary": "新增任务创建和状态查询接口", + "uri": "git://heicode-manager/branches/feature/task-api", + "checksum": "sha256:abc123", + "metadata": { + "agent_role": "backend", + "redacted": true + } + } +} +``` + +支持的 `artifact_type`： + +| 类型 | 说明 | +|---|---| +| `code_patch` | 代码补丁或分支 | +| `document` | 文档 | +| `test_report` | 测试报告 | +| `deployment_manifest` | 部署清单 | +| `log_bundle` | 日志包 | +| `other` | 其他 | + +要求： + +1. 大文件不要内联到 callback body。 +2. `summary` 可展示给用户，不得包含密钥。 +3. `uri` 可以是 `git://`、`artifact://`、`azblob://`、`https://`。 +4. metadata 必须脱敏。 + +## 11. approval.requested 与审批结果 + +高危动作前 Runtime 必须暂停，并回调： + +```json +{ + "event_id": "evt_approval_1", + "event_type": "approval.requested", + "deployment_id": "dep_xxx", + "swarm_id": "swarm-123", + "task_id": "task-deploy-1", + "source": "agent-manager-runtime", + "payload": { + "approval_id": "runtime-approval-1", + "operation": "deploy.production", + "resource_id": "prod-env", + "resource_type": "azure", + "resource_scope": "resource-group/heicode-prod", + "target_role": "ops", + "risk_level": "high", + "requires_credential": true, + "secret_ref": "azkv://heicode-kv.vault.azure.net/secrets/prod-deploy", + "ttl_seconds": 600, + "reason": "需要部署到生产环境" + } +} +``` + +Manager / 客户端审批后，Manager 调 Runtime： + +```http +POST /api/swarms/{swarm_id}/approvals/{approval_id} +Authorization: Bearer +X-Correlation-ID: +X-Idempotency-Key: approval-decision-- +Content-Type: application/json +``` + +请求： + +```json +{ + "approval_id": "runtime-approval-1", + "decision": "approved", + "reason": "用户已确认", + "manager_deployment_id": "dep_xxx", + "swarm_id": "swarm-123", + "decided_by": "user:22", + "decided_at": "2026-05-28T10:20:00Z", + "credential_lease": { + "lease_id": "lease_xxx", + "credential_ref": "lease://runtime/runtime-approval-1", + "expires_at": "2026-05-28T10:30:00Z" + } +} +``` + +`decision` 枚举： + +| decision | Runtime 行为 | +|---|---| +| `approved` | 继续原高危动作 | +| `rejected` | 停止该动作，回调 `timeline.updated` 或 `task.failed` | + +要求： + +1. Runtime 收到 `approved` 后才可以继续高危动作。 +2. Runtime 收到 `rejected` 后不能继续执行该动作。 +3. Runtime 不得把 `credential_ref` 展开写入日志或 artifact metadata。 +4. Runtime 必须把审批结果后的状态继续通过 callback 回写。 + +## 12. 日志、指标和用量 + +### 日志 + +建议提供： + +```http +GET /api/swarms/{swarm_id}/logs?limit=100&cursor= +``` + +响应： + +```json +{ + "success": true, + "data": { + "items": [ + { + "timestamp": "2026-05-28T10:21:00Z", + "level": "info", + "agent_role": "backend", + "task_id": "task-backend-1", + "message": "test completed", + "redacted": true + } + ], + "next_cursor": "" + } +} +``` + +### 指标 + +建议提供： + +```http +GET /api/swarms/{swarm_id}/metrics?window=15m&step=60s +``` + +响应： + +```json +{ + "success": true, + "data": { + "swarm_id": "swarm-123", + "agent_metrics": [ + { + "agent_instance_id": "agent-backend-1", + "agent_role": "backend", + "task_id": "task-backend-1", + "status": "running", + "cpu_percent": 12.5, + "memory_bytes": 268435456, + "uptime_seconds": 300 + } + ] + } +} +``` + +### 用量 + +Runtime 调模型时必须带关联字段： + +| 字段 | 说明 | +|---|---| +| `manager_deployment_id` | Manager deployment id | +| `swarm_id` | Runtime swarm id | +| `task_id` | 当前任务 | +| `agent_role` | Agent 角色 | +| `model_id` | 使用模型 | +| `correlation_id` | 全链路追踪 | + +Runtime 可通过 `budget.alert` 或 `timeline.updated` 回传用量摘要。 + +## 13. 安全约束 + +必须遵守： + +1. 禁止在请求、callback、日志、artifact metadata 中出现明文密码、Token、私钥、连接串、云 access key、模型 key。 +2. 长期凭据只能通过 `azkv:///secrets/` 传引用。 +3. Runtime 可以把短期租约写成 `lease://...`，但不能写真实密钥。 +4. `secret_ref`、`credential_ref` 只能作为引用使用，不能在用户可见内容中展开。 +5. 所有 callback payload 必须脱敏。 +6. 高危动作必须先 `approval.requested`，不能先执行后补审批。 + +## 14. 错误码建议 + +| HTTP | code | 场景 | +|---:|---|---| +| 401 | `UNAUTHORIZED` | service token 无效 | +| 403 | `FORBIDDEN` | 权限不足 | +| 404 | `SWARM_NOT_FOUND` | swarm 不存在 | +| 409 | `IDEMPOTENCY_CONFLICT` | 幂等键冲突 | +| 422 | `VALIDATION_ERROR` | 请求字段缺失 | +| 422 | `SECRET_REF_INVALID` | `secret_ref` 非 `azkv://...` | +| 422 | `POLICY_REJECTED` | 高危策略拒绝 | +| 500 | `INTERNAL_ERROR` | Runtime 内部错误 | + +错误响应： + +```json +{ + "success": false, + "error": { + "code": "VALIDATION_ERROR", + "message": "resource_grants[0].secret_ref is required", + "request_id": "corr_xxx" + } +} +``` + +## 15. 联调验收清单 + +| 步骤 | 操作 | 通过标准 | +|---:|---|---| +| 1 | Manager 调 `GET /api/agnet/health` | 返回 healthy | +| 2 | Manager 调 `POST /api/swarms` | 返回真实 `deployment_id` 和 `swarm_id` | +| 3 | Runtime 回调 `deployment.status_changed` | Manager timeline 可见 | +| 4 | Runtime 回调 `task.created` | Manager task flow / Agent task map 可见 | +| 5 | Runtime 回调 `task.claimed` / `task.heartbeat` | Manager 可看到 Agent 领取和心跳 | +| 6 | Runtime 回调 `handoff.requested` / `handoff.completed` | Manager 可看到交接 | +| 7 | Runtime 回调 `artifact.created` | Manager artifact 列表可见 | +| 8 | Runtime 回调 `approval.requested` | Manager approval 列表出现 pending | +| 9 | 客户端 / Manager approve | Runtime 收到 decision 并继续 | +| 10 | 客户端 / Manager reject | Runtime 收到 decision 并停止对应动作 | +| 11 | Runtime 回传 logs / metrics / usage | Manager 可查询或 timeline 可见 | +| 12 | Runtime 回调最终 `deployment.status_changed=completed` | Manager 和客户端看到完成 | + +## 16. 当前不能误报完成的项 + +以下项只有在 Agent Manager 真实实现并联调后才能算完成： + +1. 真实 Swarm Run 创建成功。 +2. 真实 task graph 生成。 +3. 真实 Agent claim / heartbeat。 +4. 真实 handoff / retry / blocked。 +5. 真实 artifact 产出。 +6. 审批后 Runtime 继续或停止。 +7. Runtime 真实日志和指标。 +8. 子 Agent 模型用量归属。 +9. 最终交付物回到客户端。 + +Manager 本地模拟事件、测试桩、空态页面和 schema 校验只能证明 Manager 接收和展示能力，不能证明蜂群生产闭环完成。 + diff --git a/docs/integration/普通sub敏捷模式-AgentManager对接任务清单.md b/docs/integration/普通sub敏捷模式-AgentManager对接任务清单.md index a6a1627e..7f157a80 100644 --- a/docs/integration/普通sub敏捷模式-AgentManager对接任务清单.md +++ b/docs/integration/普通sub敏捷模式-AgentManager对接任务清单.md @@ -252,6 +252,14 @@ Agent Manager 至少要支持以下阶段语义，并通过 callback 回写当 普通 sub 敏捷下，客户端主要看 timeline。Agent Manager 应持续回写可展示事件。 +联调前可以先请求 Manager 当前接受的 callback schema： + +```http +GET https://code.xinghanlab.com/api/agnet/callbacks/swarm-events/schema +``` + +该接口只返回事件类型、分类和必填字段，不返回任何 token 或密钥。普通 sub 敏捷重点核对 `phase.changed`、`timeline.updated`、`artifact.created`、`approval.requested`、`sk_tool.*` 和 `budget.alert`。 + ```json { "event_id": "evt_timeline_1", diff --git a/docs/integration/蜂群模式-AgentManager对接任务清单.md b/docs/integration/蜂群模式-AgentManager对接任务清单.md index 6866d5f2..0165b1d5 100644 --- a/docs/integration/蜂群模式-AgentManager对接任务清单.md +++ b/docs/integration/蜂群模式-AgentManager对接任务清单.md @@ -200,6 +200,14 @@ X-Agnet-Signature: sha256= timestamp + "." + event_id + "." + raw_body ``` +Agent Manager / 蜂群侧可以先拉取 Manager 当前接受的事件 schema： + +```http +GET https://code.xinghanlab.com/api/agnet/callbacks/swarm-events/schema +``` + +该接口不返回任何 token 或密钥，只返回事件类型、分类和必填字段，用于联调前核对 `task.*`、`handoff.*`、`artifact.created`、`approval.requested` 等 callback payload 是否符合 Manager 接收规则。 + 统一 envelope： ```json diff --git a/heicode/controller/agnet_callback.go b/heicode/controller/agnet_callback.go index bd04f124..7d522ed6 100644 --- a/heicode/controller/agnet_callback.go +++ b/heicode/controller/agnet_callback.go @@ -7,6 +7,7 @@ import ( "fmt" "io" "os" + "sort" "strconv" "strings" "time" @@ -329,18 +330,90 @@ func callbackIntValue(values map[string]any, key string) int { } var agnetCallbackEventRequiredFields = map[string][]string{ - "task.created": {"task_id", "title"}, - "task.claimed": {"task_id", "agent_role"}, - "task.running": {"task_id", "agent_role"}, - "task.heartbeat": {"task_id", "agent_role"}, - "task.blocked": {"task_id", "reason"}, - "task.retried": {"task_id", "attempt"}, - "task.released": {"task_id", "agent_role"}, - "task.failed": {"task_id", "reason"}, - "task.completed": {"task_id"}, - "handoff.requested": {"task_id", "from_role", "to_role"}, - "handoff.completed": {"task_id", "from_role", "to_role"}, - "artifact.created": {"artifact_id"}, + "deployment.status_changed": {"status"}, + "phase.changed": {"stage", "checkpoint"}, + "agent.started": {"agent_role"}, + "agent.completed": {"agent_role"}, + "agent.crashed": {"agent_role", "reason"}, + "task.created": {"task_id", "title"}, + "task.claimed": {"task_id", "agent_role"}, + "task.running": {"task_id", "agent_role"}, + "task.heartbeat": {"task_id", "agent_role"}, + "task.blocked": {"task_id", "reason"}, + "task.retried": {"task_id", "attempt"}, + "task.released": {"task_id", "agent_role"}, + "task.failed": {"task_id", "reason"}, + "task.completed": {"task_id"}, + "handoff.requested": {"task_id", "from_role", "to_role"}, + "handoff.completed": {"task_id", "from_role", "to_role"}, + "approval.requested": {"approval_id", "operation", "risk_level"}, + "artifact.created": {"artifact_id"}, + "timeline.updated": {"title"}, + "sk_tool.called": {"tool_name", "tool_invocation_id"}, + "sk_tool.completed": {"tool_name", "tool_invocation_id"}, + "sk_tool.failed": {"tool_name", "tool_invocation_id", "reason"}, + "budget.alert": {"threshold_pct"}, +} + +var agnetCallbackEventCategories = map[string]string{ + "deployment.status_changed": "deployment", + "phase.changed": "ordinary_sub", + "agent.started": "ordinary_sub", + "agent.completed": "ordinary_sub", + "agent.crashed": "ordinary_sub", + "task.created": "swarm_task_flow", + "task.claimed": "swarm_task_flow", + "task.running": "swarm_task_flow", + "task.heartbeat": "swarm_task_flow", + "task.blocked": "swarm_task_flow", + "task.retried": "swarm_task_flow", + "task.released": "swarm_task_flow", + "task.failed": "swarm_task_flow", + "task.completed": "swarm_task_flow", + "handoff.requested": "swarm_task_flow", + "handoff.completed": "swarm_task_flow", + "approval.requested": "approval", + "artifact.created": "artifact", + "timeline.updated": "timeline", + "sk_tool.called": "sk", + "sk_tool.completed": "sk", + "sk_tool.failed": "sk", + "budget.alert": "budget", +} + +func AgnetGetSwarmEventCallbackSchema(c *gin.Context) { + events := make([]string, 0, len(agnetCallbackEventRequiredFields)) + for eventType := range agnetCallbackEventRequiredFields { + events = append(events, eventType) + } + sort.Strings(events) + + items := make([]gin.H, 0, len(events)) + for _, eventType := range events { + required := append([]string(nil), agnetCallbackEventRequiredFields[eventType]...) + sort.Strings(required) + items = append(items, gin.H{ + "event_type": eventType, + "category": agnetCallbackEventCategories[eventType], + "required_fields": required, + "payload_location": "top-level envelope or payload object; artifact_id may also be in artifact object", + }) + } + + common.ApiSuccess(c, gin.H{ + "callback_path": "/api/agnet/callbacks/swarm-events", + "auth": gin.H{ + "service_token_headers": []string{"X-Agnet-Service-Token", "Authorization: Bearer "}, + "hmac_headers": []string{"X-Agnet-Event-Id", "X-Agnet-Timestamp", "X-Agnet-Signature"}, + "hmac_payload": "timestamp + \".\" + event_id + \".\" + raw_body", + }, + "dedupe_keys": []string{"X-Agnet-Event-Id", "event_id", "idempotency_key"}, + "events": items, + "security": gin.H{ + "plaintext_secrets_allowed": false, + "secret_ref_scheme": "azkv:///secrets/", + }, + }) } func callbackEnvelopeFieldValue(payload agnetCallbackEnvelope, key string) string { @@ -361,9 +434,11 @@ func callbackHasFieldValue(payload agnetCallbackEnvelope, key string) bool { if callbackStringValue(payload.Payload, key) != "" { return true } - if key == "artifact_id" { - if nested := callbackMapValue(payload.Payload, "artifact"); nested != nil { - return callbackStringValue(nested, key) != "" + for _, nestedKey := range []string{"artifact", "approval"} { + if nested := callbackMapValue(payload.Payload, nestedKey); nested != nil { + if callbackStringValue(nested, key) != "" { + return true + } } } if key == "attempt" { diff --git a/heicode/controller/agnet_control_plane.go b/heicode/controller/agnet_control_plane.go index 10a8e2d5..c9183653 100644 --- a/heicode/controller/agnet_control_plane.go +++ b/heicode/controller/agnet_control_plane.go @@ -1639,6 +1639,8 @@ func AgnetListDeploymentLogs(c *gin.Context) { "timestamp": record.CreatedAt, "deployment_id": deploymentID, "stream": "control", + "source": "manager_control_plane", + "data_source": "manager_control_plane", "level": "info", "message": "deployment accepted by Manager control-plane placeholder", "phase": record.Phase, @@ -1652,6 +1654,8 @@ func AgnetListDeploymentLogs(c *gin.Context) { "timestamp": event.OccurredAt, "deployment_id": event.DeploymentID, "stream": "event", + "source": "manager_audit", + "data_source": "manager_audit", "level": "info", "message": event.Event, "phase": record.Phase, @@ -1663,11 +1667,13 @@ func AgnetListDeploymentLogs(c *gin.Context) { } common.ApiSuccess(c, gin.H{ - "deployment_id": deploymentID, - "items": items, - "next_cursor": "", - "redacted": true, - "total": len(items), + "deployment_id": deploymentID, + "items": items, + "next_cursor": "", + "redacted": true, + "data_source": "manager_control_plane", + "runtime_source": "not_connected", + "total": len(items), }) } @@ -1693,6 +1699,8 @@ func AgnetGetDeploymentMetrics(c *gin.Context) { common.ApiSuccess(c, gin.H{ "deployment_id": deploymentID, + "data_source": "manager_control_plane", + "runtime_source": "not_connected", "window": strings.TrimSpace(c.DefaultQuery("window", "15m")), "step": strings.TrimSpace(c.DefaultQuery("step", "60s")), "phase": record.Phase, diff --git a/heicode/router/agnet_runtime_http_smoke_test.go b/heicode/router/agnet_runtime_http_smoke_test.go index 097b56c2..a0fb2463 100644 --- a/heicode/router/agnet_runtime_http_smoke_test.go +++ b/heicode/router/agnet_runtime_http_smoke_test.go @@ -156,6 +156,18 @@ func TestAgnetRuntimeRealHTTPHealthAndShadowCreateSmoke(t *testing.T) { require.Contains(t, healthBody, `"success":true`) require.Contains(t, healthBody, `"status":"healthy"`) + schemaResp, err := http.Get(managerURL + "/api/agnet/callbacks/swarm-events/schema") + require.NoError(t, err) + schemaBody := readAgnetRuntimeSmokeBody(t, schemaResp) + require.Equal(t, http.StatusOK, schemaResp.StatusCode) + require.Contains(t, schemaBody, `"event_type":"task.claimed"`) + require.Contains(t, schemaBody, `"event_type":"approval.requested"`) + var schemaEnvelope map[string]any + require.NoError(t, common.Unmarshal([]byte(schemaBody), &schemaEnvelope)) + schemaData := schemaEnvelope["data"].(map[string]any) + schemaSecurity := schemaData["security"].(map[string]any) + require.Equal(t, "azkv:///secrets/", schemaSecurity["secret_ref_scheme"]) + createBody := `{ "orchestration_plan":{ "intent_id":"intent-http-smoke", @@ -184,4 +196,16 @@ func TestAgnetRuntimeRealHTTPHealthAndShadowCreateSmoke(t *testing.T) { var stored model.AgnetDeployment require.NoError(t, db.Where("runtime_swarm_id = ?", "runtime-http-swarm").First(&stored).Error) require.Equal(t, "accepted", stored.RuntimeState) + + logsResp := agnetRuntimeAdminRequest(t, http.MethodGet, managerURL+"/api/agnet/deployments/"+stored.DeploymentID+"/logs", "") + logsBody := readAgnetRuntimeSmokeBody(t, logsResp) + require.Equal(t, http.StatusOK, logsResp.StatusCode) + require.Contains(t, logsBody, `"data_source":"manager_control_plane"`) + require.Contains(t, logsBody, `"runtime_source":"not_connected"`) + + metricsResp := agnetRuntimeAdminRequest(t, http.MethodGet, managerURL+"/api/agnet/deployments/"+stored.DeploymentID+"/metrics", "") + metricsBody := readAgnetRuntimeSmokeBody(t, metricsResp) + require.Equal(t, http.StatusOK, metricsResp.StatusCode) + require.Contains(t, metricsBody, `"data_source":"manager_control_plane"`) + require.Contains(t, metricsBody, `"platform_estimated":true`) } diff --git a/heicode/router/api-router.go b/heicode/router/api-router.go index b8c09e67..65ced402 100644 --- a/heicode/router/api-router.go +++ b/heicode/router/api-router.go @@ -59,6 +59,7 @@ func SetApiRouter(router *gin.Engine) { apiRouter.POST("/stripe/webhook", controller.StripeWebhook) apiRouter.POST("/creem/webhook", controller.CreemWebhook) apiRouter.POST("/waffo/webhook", controller.WaffoWebhook) + apiRouter.GET("/agnet/callbacks/swarm-events/schema", controller.AgnetGetSwarmEventCallbackSchema) apiRouter.POST("/agnet/callbacks/swarm-events", controller.AgnetReceiveSwarmEventCallback) apiRouter.POST("/swarms", middleware.UserAuth(), controller.AgnetCreateUserSwarm) //apiRouter.POST("/waffo-pancake/webhook", controller.WaffoPancakeWebhook) diff --git a/heicode/web/default/src/features/agnet-console/pages.tsx b/heicode/web/default/src/features/agnet-console/pages.tsx index 52bbe0ff..69635cac 100644 --- a/heicode/web/default/src/features/agnet-console/pages.tsx +++ b/heicode/web/default/src/features/agnet-console/pages.tsx @@ -333,6 +333,54 @@ function taskFlowDetail(item: Record): string { return parts.join(' / ') || '—' } +type TaskFlowSummary = { + taskId: string + status: string + agentRole: string + handoff: string + source: string +} + +function buildTaskFlowSummaries( + records: Record[] +): TaskFlowSummary[] { + const byTask = new Map() + records.forEach((item, idx) => { + const event = String(item.event_type || item.event || '').toLowerCase() + const taskId = + String(recordPayloadValue(item, 'task_id') || '').trim() || + `task-${idx + 1}` + const current = + byTask.get(taskId) ?? + ({ + taskId, + status: 'observed', + agentRole: '', + handoff: '', + source: formatRecordSource(item.source), + } satisfies TaskFlowSummary) + const agentRole = String(recordPayloadValue(item, 'agent_role') || '') + const fromRole = String(recordPayloadValue(item, 'from_role') || '') + const toRole = String(recordPayloadValue(item, 'to_role') || '') + const reason = String(recordPayloadValue(item, 'reason') || '') + if (agentRole) current.agentRole = agentRole + if (event.startsWith('task.')) { + current.status = event.replace('task.', '') || current.status + } + if (event === 'handoff.requested' || event === 'handoff.completed') { + current.status = event.replace('handoff.', 'handoff:') + if (fromRole || toRole) + current.handoff = `${fromRole || '?'} -> ${toRole || '?'}` + } + if (reason && current.status === 'blocked') { + current.handoff = reason + } + current.source = formatRecordSource(item.source) + byTask.set(taskId, current) + }) + return Array.from(byTask.values()) +} + function collectResourceGrants( dep: AgnetDeployment ): Record[] { @@ -1001,11 +1049,13 @@ function RunRelatedRecordsPanel({ deploymentId }: { deploymentId: string }) { const timeline = data?.timeline ?? [] const taskFlowRecords = useMemo( () => - timeline.filter((item) => - isTaskFlowEvent(item.event_type || item.event) - ), + timeline.filter((item) => isTaskFlowEvent(item.event_type || item.event)), [timeline] ) + const taskFlowSummaries = useMemo( + () => buildTaskFlowSummaries(taskFlowRecords), + [taskFlowRecords] + ) return (
@@ -1046,12 +1096,12 @@ function RunRelatedRecordsPanel({ deploymentId }: { deploymentId: string }) { />
-
+

{t('Sub task flow')}

- + {taskFlowRecords.length}
@@ -1074,7 +1124,7 @@ function RunRelatedRecordsPanel({ deploymentId }: { deploymentId: string }) {

{event}

- + {source}
@@ -1088,6 +1138,64 @@ function RunRelatedRecordsPanel({ deploymentId }: { deploymentId: string }) { )}
+
+
+

+ {t('Agent task map')} +

+ + {taskFlowSummaries.length} + +
+ {taskFlowSummaries.length === 0 ? ( +

+ {t( + 'Waiting for Runtime task graph callbacks. Manager will show task, Agent role, handoff and source here when callbacks arrive.' + )} +

+ ) : ( +
+ + + + + + + + + + + + {taskFlowSummaries.map((item) => ( + + + + + + + + ))} + +
{t('Task')}{t('Status')}{t('Agent role')}{t('Handoff')}{t('Source')}
+ {item.taskId} + + + + {item.agentRole || '—'} + + {item.handoff || '—'} + + + {item.source} + +
+
+ )} +
+

@@ -1159,7 +1267,7 @@ function RunRelatedRecordsPanel({ deploymentId }: { deploymentId: string }) {

-
+

{t('Merged timeline')}

@@ -1179,7 +1287,7 @@ function RunRelatedRecordsPanel({ deploymentId }: { deploymentId: string }) { >

{event}

- + {source}
diff --git a/scripts/agnet_sub_mode_smoke.py b/scripts/agnet_sub_mode_smoke.py index 1736e2b3..69635202 100755 --- a/scripts/agnet_sub_mode_smoke.py +++ b/scripts/agnet_sub_mode_smoke.py @@ -11,6 +11,9 @@ authenticated deployment flow, provide: Optional: HEICODE_RUN_SIMULATE=true + HEICODE_RUN_CALLBACK_SMOKE=true + HEICODE_CALLBACK_TOKEN= + HEICODE_CALLBACK_SWARM_ID= HEICODE_MANAGER_BASE_URL=https://code.xinghanlab.com AGENT_MANAGER_HEALTH_URL=http://20.212.121.126/api/agnet/health """ @@ -31,11 +34,18 @@ AGENT_HEALTH = os.getenv( ACCESS_TOKEN = os.getenv("HEICODE_ACCESS_TOKEN", "").strip() USER_ID = os.getenv("HEICODE_USER_ID", "").strip() DEPLOYMENT_ID = os.getenv("HEICODE_DEPLOYMENT_ID", "").strip() +CALLBACK_TOKEN = os.getenv("HEICODE_CALLBACK_TOKEN", "").strip() +CALLBACK_SWARM_ID = os.getenv("HEICODE_CALLBACK_SWARM_ID", "").strip() RUN_SIMULATE = os.getenv("HEICODE_RUN_SIMULATE", "").strip().lower() in { "1", "true", "yes", } +RUN_CALLBACK_SMOKE = os.getenv("HEICODE_RUN_CALLBACK_SMOKE", "").strip().lower() in { + "1", + "true", + "yes", +} UA = "heicode-agnet-sub-mode-smoke/1.0" FAILS: list[str] = [] @@ -93,6 +103,23 @@ check("Manager /api/status success", code == 200 and status.get("success") is Tr version = status.get("data", {}).get("version") if isinstance(status.get("data"), dict) else "" check("Manager status carries version", bool(version), f"version={version!r}") +code, _headers, body = manager_request("GET", "/api/agnet/callbacks/swarm-events/schema") +schema = parse_json("callback schema JSON parseable", body) +schema_data = schema.get("data", {}) if isinstance(schema.get("data"), dict) else {} +schema_events = schema_data.get("events") or [] +schema_event_types = { + item.get("event_type") + for item in schema_events + if isinstance(item, dict) +} +check( + "callback schema lists swarm and ordinary sub events", + code == 200 + and schema.get("success") is True + and {"task.claimed", "handoff.requested", "phase.changed", "artifact.created"}.issubset(schema_event_types), + f"HTTP {code}", +) + code, _headers, body = request("GET", AGENT_HEALTH) agent = parse_json("Agent Manager health JSON parseable", body) agent_data = agent.get("data", {}) if isinstance(agent.get("data"), dict) else {} @@ -106,6 +133,91 @@ check( if not (ACCESS_TOKEN and USER_ID and DEPLOYMENT_ID): print("\nAuthenticated deployment checks skipped: set HEICODE_ACCESS_TOKEN, HEICODE_USER_ID and HEICODE_DEPLOYMENT_ID.") else: + if RUN_CALLBACK_SMOKE: + if not CALLBACK_TOKEN: + check("callback smoke has HEICODE_CALLBACK_TOKEN", False) + else: + callback_headers = { + "Content-Type": "application/json", + "X-Agnet-Service-Token": CALLBACK_TOKEN, + "X-Correlation-ID": f"smoke-corr-{DEPLOYMENT_ID}", + } + swarm_id = CALLBACK_SWARM_ID or f"smoke-swarm-{DEPLOYMENT_ID}" + callback_events = [ + { + "event_id": f"smoke-task-created-{DEPLOYMENT_ID}", + "event_type": "task.created", + "deployment_id": DEPLOYMENT_ID, + "swarm_id": swarm_id, + "task_id": f"smoke-task-{DEPLOYMENT_ID}", + "source": "smoke-runtime", + "payload": { + "title": "Smoke task created", + "agent_role": "backend", + }, + }, + { + "event_id": f"smoke-task-claimed-{DEPLOYMENT_ID}", + "event_type": "task.claimed", + "deployment_id": DEPLOYMENT_ID, + "swarm_id": swarm_id, + "task_id": f"smoke-task-{DEPLOYMENT_ID}", + "source": "smoke-runtime", + "payload": { + "agent_role": "backend", + }, + }, + { + "event_id": f"smoke-handoff-{DEPLOYMENT_ID}", + "event_type": "handoff.requested", + "deployment_id": DEPLOYMENT_ID, + "swarm_id": swarm_id, + "task_id": f"smoke-task-{DEPLOYMENT_ID}", + "source": "smoke-runtime", + "payload": { + "from_role": "backend", + "to_role": "reviewer", + "reason": "Smoke handoff validation", + }, + }, + { + "event_id": f"smoke-artifact-{DEPLOYMENT_ID}", + "event_type": "artifact.created", + "deployment_id": DEPLOYMENT_ID, + "swarm_id": swarm_id, + "task_id": f"smoke-task-{DEPLOYMENT_ID}", + "source": "smoke-runtime", + "artifact": { + "artifact_id": f"smoke-artifact-{DEPLOYMENT_ID}", + "artifact_type": "test_report", + "title": "Smoke artifact", + "summary": "Callback smoke wrote this artifact through the Manager callback endpoint.", + "uri": f"artifact://smoke/{DEPLOYMENT_ID}/report", + }, + "payload": { + "stage": "testing", + "checkpoint": "artifact_ready", + }, + }, + ] + for event in callback_events: + body_bytes = json.dumps(event).encode("utf-8") + callback_headers["X-Agnet-Event-Id"] = event["event_id"] + code, _headers, body = request( + "POST", + BASE + "/api/agnet/callbacks/swarm-events", + headers=callback_headers, + body=body_bytes, + ) + callback_response = parse_json( + f"callback {event['event_type']} JSON parseable", body + ) + check( + f"callback {event['event_type']} accepted", + code == 200 and callback_response.get("success") is True, + f"HTTP {code}", + ) + if RUN_SIMULATE: code, _headers, body = manager_request( "POST", @@ -155,6 +267,11 @@ else: for item in merged if isinstance(item, dict) } + sources = { + item.get("source") + for item in merged + if isinstance(item, dict) and item.get("source") + } check( "timeline endpoint success", code == 200 and timeline.get("success") is True and isinstance(merged, list), @@ -170,6 +287,21 @@ else: "artifact.created" in event_types or "approval.requested" in event_types, "events=" + ",".join(sorted(str(event) for event in event_types if event)[:10]), ) + if RUN_CALLBACK_SMOKE: + check( + "timeline includes smoke callback source", + "smoke-runtime" in sources, + "sources=" + ",".join(sorted(str(source) for source in sources)[:10]), + ) + check( + "callback smoke artifact is queryable", + any( + isinstance(item, dict) + and item.get("artifact_id") == f"smoke-artifact-{DEPLOYMENT_ID}" + for item in artifact_items + ), + f"artifact_count={len(artifact_items) if isinstance(artifact_items, list) else 'n/a'}", + ) if FAILS: print(f"\n{len(FAILS)} check(s) failed:")