Merge pull request #3 from xmindlab-heicode/feature/v6-benchmark-v2
Agent Swarm v6：基准 v2.1、主控 Agent、对等回复与客户端指南
This commit is contained in:
Fasthei
@@ -0,0 +1,56 @@
|
||||
name: CI
|
||||
|
||||
on:
|
||||
push:
|
||||
branches: ["**"]
|
||||
pull_request:
|
||||
branches: ["**"]
|
||||
|
||||
jobs:
|
||||
guardrails:
|
||||
runs-on: ubuntu-latest
|
||||
steps:
|
||||
- uses: actions/checkout@v4
|
||||
|
||||
# Defense in depth: fail if secrets or heavy/generated dirs were ever committed.
|
||||
- name: Block secrets & node_modules
|
||||
run: |
|
||||
if git ls-files | grep -E '(^|/)\.env($|\.)|(^|/)secrets/|\.pem$|\.key$|\.p12$|\.pfx$|(^|/)id_rsa$|(^|/)id_ed25519$'; then
|
||||
echo "::error::Secret-like files are tracked — remove them and rotate any exposed credential."; exit 1
|
||||
fi
|
||||
if git ls-files | grep -E '(^|/)node_modules/'; then
|
||||
echo "::error::node_modules is tracked — it must be gitignored."; exit 1
|
||||
fi
|
||||
|
||||
- name: Required standards files present
|
||||
run: |
|
||||
for f in CLAUDE.md PROJECT_STANDARD.md README.md; do
|
||||
test -f "$f" || { echo "::error::Missing required file: $f"; exit 1; }
|
||||
done
|
||||
|
||||
tests:
|
||||
runs-on: ubuntu-latest
|
||||
steps:
|
||||
- uses: actions/checkout@v4
|
||||
|
||||
- uses: actions/setup-python@v5
|
||||
with:
|
||||
python-version: "3.13"
|
||||
|
||||
- name: Install dependencies
|
||||
run: |
|
||||
python -m pip install --upgrade pip
|
||||
pip install -r orchestrator/requirements.txt -r agent/requirements.txt
|
||||
|
||||
# Hermetic: in-memory store, planner forced offline by the tests — no model key needed.
|
||||
- name: Manager contract test
|
||||
env: { REDIS_FAKE: "1" }
|
||||
run: python scripts/test-runtime-contract.py
|
||||
|
||||
- name: Workflow mechanism smoke test
|
||||
env: { REDIS_FAKE: "1" }
|
||||
run: python scripts/test-merge-smoke.py
|
||||
|
||||
- name: End-to-end workflow test
|
||||
env: { REDIS_FAKE: "1" }
|
||||
run: python scripts/test-workflow-e2e.py
|
||||
@@ -1,6 +1,6 @@
|
||||
# CLAUDE.md — HeiCode Swarm（执行面 / 运行时）
|
||||
|
||||
本仓库（`agent_swarm_v5`，对应 **HeiCode-Swarm**）是蜂群执行面 / 回调 / Swarm Runtime。整体介绍见 [README.md](README.md)，工程标准见 [PROJECT_STANDARD.md](PROJECT_STANDARD.md)，交付说明见 [docs/DELIVERY.md](docs/DELIVERY.md)。
|
||||
本仓库（`agent_swarm_v6`，对应 **HeiCode-Swarm**）是蜂群执行面 / 回调 / Swarm Runtime。整体介绍见 [README.md](README.md)，工程标准见 [PROJECT_STANDARD.md](PROJECT_STANDARD.md)，交付说明见 [docs/DELIVERY.md](docs/DELIVERY.md)。
|
||||
|
||||
## 改动前必读
|
||||
1. 先读本 `CLAUDE.md` 与本仓 `PROJECT_STANDARD.md`。
|
||||
|
||||
+123
@@ -0,0 +1,123 @@
|
||||
# 客户端接入指南（Client Guide）
|
||||
|
||||
本文说明**客户端如何与 Agent Swarm 运行时交互**。系统总览见 [README.md](README.md)；接口契约见 [docs/integration/runtime-contract.md](docs/integration/runtime-contract.md)；事件结构见 [docs/integration/event-schema.md](docs/integration/event-schema.md)。
|
||||
|
||||
> 架构说明：生产链路中，终端用户通过 **Heicode Manager / 桌面客户端** 提交需求，由 Manager 调用本 Swarm 运行时（见 `heicode-swarm-deferred` 裁定）。「客户端」在本文指**调用 Swarm 运行时 REST API 的一方**（Manager、联调脚本或开发者）。Swarm 不在对话回路，通过**带签名回调**回报状态。
|
||||
|
||||
---
|
||||
|
||||
## 1. 两个交互面
|
||||
|
||||
1. **REST API（客户端 → Swarm）**：创建部署、查询状态/任务/日志/事件/指标、审批、停止。
|
||||
2. **回调事件流（Swarm → 客户端/Manager）**：生命周期事件经带 HMAC 签名的回调推送（见 §6）。
|
||||
|
||||
WebSocket `/ws/{agent_id}` 仅供 **Agent 执行单元** 接入，**不是客户端接口**。
|
||||
|
||||
---
|
||||
|
||||
## 2. 鉴权
|
||||
|
||||
- 客户端调用带 `Authorization: Bearer <token>`，Swarm 校验 `AGENT_RUNTIME_SERVICE_TOKEN`（兼容 `AGNET_RUNTIME_SERVICE_TOKEN`）。
|
||||
- 未配置令牌时为**非安全开发模式**（不校验，仅本地）。
|
||||
- 常用请求头：`X-Correlation-ID`（贯穿追踪）、`X-Idempotency-Key`（创建幂等）。
|
||||
- 响应信封：成功 `{ "success": true, "data": {...} }`；失败 `{ "success": false, "error": {code,message,request_id} }`。
|
||||
|
||||
---
|
||||
|
||||
## 3. 生命周期：创建 → 观察 → （审批）→ 停止
|
||||
|
||||
### 3.1 创建部署
|
||||
`POST /api/swarms`（别名 `/api/agent/swarm/deployments`、`/api/agnet/deployments`）
|
||||
|
||||
```bash
|
||||
curl -s -X POST http://<host>:8000/api/swarms \
|
||||
-H "Content-Type: application/json" \
|
||||
-H "Authorization: Bearer $TOKEN" \
|
||||
-H "X-Correlation-ID: corr-123" \
|
||||
-H "X-Idempotency-Key: idem-123" \
|
||||
-d '{
|
||||
"mode": "swarm",
|
||||
"requirement": { "objective": "实现 add(a,b) 并补充测试与说明" },
|
||||
"callback": { "url": "https://your-manager/api/agent/callbacks/runtime-events", "subscribed_events": [] },
|
||||
"metadata": { "manager_deployment_id": "dep-1" }
|
||||
}'
|
||||
```
|
||||
必填：`requirement.objective`（或 `orchestration_plan.objective`）、`callback.url`、`metadata.manager_deployment_id`。
|
||||
|
||||
响应 `data`：`deployment_id`、`runtime_deployment_id`、`swarm_id`(=workflow_id)、`manager_deployment_id`、`mode`、`status`、`created`。**请保存 `deployment_id`** 用于后续查询。
|
||||
|
||||
### 3.2 观察
|
||||
| 接口 | 用途 |
|
||||
|---|---|
|
||||
| `GET /api/swarms/{deployment_id}` | 部署状态概要 |
|
||||
| `GET /api/swarms/{deployment_id}/workflow` | 工作流/阶段视图（phases、agents、tokens、tools、artifacts、summary） |
|
||||
| `GET /api/swarms/{deployment_id}/tasks` | 任务 DAG（task_id、agent_role、status、depends_on…） |
|
||||
| `GET /api/swarms/{deployment_id}/logs`（`/events`） | 事件流，分页 `?limit=&cursor=`（断线后用 cursor 回补） |
|
||||
| `GET /api/swarms/{deployment_id}/metrics` | 任务/Agent/预算指标 |
|
||||
| `GET /api/swarms/{deployment_id}/diagnostics` | 失败、回调尝试、审批诊断 |
|
||||
|
||||
状态机：`waiting_approval → running → (blocked ⇄ running) → completed | failed | stopped`。
|
||||
|
||||
### 3.3 审批（高危操作）
|
||||
当部署进入 `waiting_approval` 并推送 `approval.requested` 时，客户端经 Manager 审批后回执：
|
||||
```bash
|
||||
curl -s -X POST http://<host>:8000/api/swarms/<deployment_id>/approvals/<approval_id> \
|
||||
-H "Authorization: Bearer $TOKEN" -H "Content-Type: application/json" \
|
||||
-d '{ "decision": "approved" }' # 或 "rejected"，可带 credential_ref / reason
|
||||
```
|
||||
|
||||
### 3.4 停止
|
||||
```bash
|
||||
curl -s -X POST http://<host>:8000/api/swarms/<deployment_id>/stop \
|
||||
-H "Authorization: Bearer $TOKEN" -H "Content-Type: application/json" \
|
||||
-d '{ "reason": "client requested stop" }'
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 4. 健康检查
|
||||
|
||||
- `GET /` → `{status, service}`；`GET /health` → 含 Redis 与连接数；`GET /metrics` → Prometheus。
|
||||
|
||||
---
|
||||
|
||||
## 5. 启用完整工作流
|
||||
|
||||
默认仅执行 Manager 提供的编排；要启用「分解 → 评审/重做 → 汇总」主控工作流，运行 Swarm 时设置（默认关闭）：
|
||||
- `ENABLE_PLANNER_FALLBACK=1`：无 Manager 分工时由主控 Agent 自动分解。
|
||||
- `ENABLE_REVIEW_LOOP=1` / `MAX_REVIEW_CYCLES=2`：主控评审与重做循环 + 结果汇总。
|
||||
- 模型走 OpenAI 兼容网关（`OPENAI_API_KEY`/`OPENAI_API_BASE`/`OPENAI_MODEL`，经 `secret_ref` 注入）。
|
||||
|
||||
---
|
||||
|
||||
## 6. 回调事件流（Swarm → 客户端/Manager）
|
||||
|
||||
Swarm 向 `callback.url` POST 事件，**带 HMAC 签名**（客户端须验签）：
|
||||
- 头：`X-Agent-Timestamp`(ms)、`X-Agent-Signature`、`X-Agent-Event-Id`、`X-Correlation-ID`（+ `X-Agnet-` 兼容别名）。
|
||||
- 签名：`sha256=hex(HMAC_SHA256(secret, "{timestamp}.{event_id}.{raw_body}"))`，secret = `AGENT_CALLBACK_SIGNING_SECRET`，时间容差 300s。
|
||||
- 幂等：按 `X-Agent-Event-Id` → body `event_id` → `idempotency_key` 去重。
|
||||
- 事件类型与必填字段：见 [event-schema.md](docs/integration/event-schema.md)（`deployment.status_changed`、`task.*`、`handoff.*`、`approval.requested`、`artifact.created`、`timeline.updated`、`budget.alert`…）。
|
||||
|
||||
客户端可只消费 REST（轮询 `/workflow`+`/logs?cursor=`），或同时接收回调。
|
||||
|
||||
---
|
||||
|
||||
## 7. 本地联调（无需密钥）
|
||||
|
||||
```bash
|
||||
# 1) 编排器（内存存储 + 工作流开关）
|
||||
set "REDIS_FAKE=1" & set "ENABLE_PLANNER_FALLBACK=1" & set "ENABLE_REVIEW_LOOP=1"
|
||||
python -m uvicorn orchestrator.main:app --host 0.0.0.0 --port 8000
|
||||
# 2) 无密钥桩 Agent（联调用）
|
||||
set "ORCHESTRATOR_URL=ws://localhost:8000" & set "AGENT_ID=stub-1" & set "AGENT_CAPABILITIES=python,code_generation,testing,pytest,technical-writing,general"
|
||||
python scripts/stub_agent.py
|
||||
# 3) 用 §3.1 的 curl 提交需求，再用 §3.2 观察
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 8. 说明与边界
|
||||
|
||||
- **桌面/前端 UI**：当前桌面客户端为单 Agent A2A 直连模型；**蜂群图 / 协作时间线**的前端事件 API 尚未冻结（见 [frontend-event-api.md](docs/integration/frontend-event-api.md)）。
|
||||
- **不得在请求/回调/日志中出现明文密钥**；凭据一律 `secret_ref`（`azkv://`）。见 [security-boundary.md](docs/integration/security-boundary.md)。
|
||||
- 计费在 NewAPI（模型网关）侧按请求计量，Swarm 仅上报用量；见 [usage-billing-schema.md](docs/integration/usage-billing-schema.md)。
|
||||
@@ -1,9 +1,23 @@
|
||||
# Agent Swarm（HeiCode Swarm）
|
||||
|
||||
一个多智能体「蜂群」系统：用户提出需求后，由主控逻辑自动将其分解为多个子任务，分发给擅长不同领域的专家 Agent 并行完成；专家之间可就重叠领域相互协作；产出汇总后由主控评审是否达标，未达标则退回重做，循环直至生成满意的最终回答。
|
||||
一个多智能体「蜂群」系统：用户提出需求后，由**主控 Agent（Master Agent，`orchestrator/master_agent.py`）**自动将其分解为多个子任务，分发给擅长不同领域的专家 Agent 并行完成；专家之间可就重叠领域相互协作；产出汇总后由主控 Agent 评审是否达标，未达标则退回重做，循环直至生成满意的最终回答。主控 Agent 负责「分解 / 评审决策 / 汇总」的认知决策，编排器负责执行其决策（派发、重开任务、持久化、事件）。
|
||||
|
||||
> ⚠️ **能力边界（主链路接入状态）**
|
||||
> 本仓当前是一个**可运行的多 Agent 工作流运行系统**，**尚未**作为 Heicode 主链路的正式 Runtime Backend 接入。下表区分能力状态；契约见 [docs/integration/](docs/integration/)，量化标准见 [docs/benchmark/](docs/benchmark/)。
|
||||
>
|
||||
> | 能力 | 状态 |
|
||||
> |---|---|
|
||||
> | 任务分解（规划回退）、能力路由派发、专家执行（OpenAI 兼容）、评审/重做循环、结果汇总、peer 协作消息路由、WebSocket Agent 协议、Redis 持久化、Prometheus 指标 | ✅ 已实现（仓内可运行） |
|
||||
> | Manager↔Runtime 生命周期契约（对齐 `heicode-am-contract`）、HMAC 签名回调事件 envelope、稳定的 `deployment_id`/`workflow_id`/`trace_id` | 🟡 待接入（见 `docs/integration/runtime-contract.md`；事件 envelope 与签名待 Manager 对齐） |
|
||||
> | 统一 usage/计费聚合（归属 NewAPI）、审计/lineage trace、前端事件 API、统一 Agent registry/scheduling、统一 secret/workspace/tool/MCP/tenant 安全边界 | 🟡 待接入（需与 Billing / Audit / Frontend / Infra / Security Team 对齐） |
|
||||
> | Benchmark 自证（`Benchmark_Agent`、`S_swarm`、`G_E`、`G_E,c`、治理/协作/通信/鲁棒性指标、baseline 对比、telemetry 架构） | 🔴 规划中（标准见 `docs/benchmark/`，采集器尚未落地） |
|
||||
>
|
||||
> 在以上「待接入 / 规划中」项目完成并经对应 Team 验收前，本文与各子文档**不得宣称**「已接入主链路」或「已具备完整 Agent Swarm 工程能力」。
|
||||
|
||||
## 系统架构
|
||||
|
||||
> 说明：下图中 **Heicode Manager → Orchestrator** 为**目标形态**。当前 orchestrator 仅实现了占位的 Manager 面接口与回调骨架，尚未按 `heicode-am-contract` 正式注册为 Runtime Backend（见 `docs/integration/runtime-contract.md`）。
|
||||
|
||||
```
|
||||
用户
|
||||
│
|
||||
|
||||
+54
-12
@@ -74,6 +74,7 @@ class Agent:
|
||||
|
||||
MAX_CONCURRENT_TASKS = int(os.getenv("MAX_CONCURRENT_TASKS", "4"))
|
||||
TASK_TIMEOUT_SECONDS = int(os.getenv("TASK_TIMEOUT_SECONDS", "60"))
|
||||
PEER_REPLY_TIMEOUT_SECONDS = int(os.getenv("PEER_REPLY_TIMEOUT_SECONDS", "20"))
|
||||
HEARTBEAT_INTERVAL_SECONDS = 15
|
||||
|
||||
def __init__(
|
||||
@@ -102,6 +103,8 @@ class Agent:
|
||||
self.peer_waiters: dict[str, asyncio.Future] = {}
|
||||
# Summary of this agent's most recently completed task, shared when peers consult it.
|
||||
self.last_summary: Optional[str] = None
|
||||
# Lazily-created executor used to compose substantive peer replies.
|
||||
self._peer_executor: Optional[TaskExecutor] = None
|
||||
|
||||
self.workspace_git = GitOperations(str(self.workspace_dir), self.agent_id)
|
||||
|
||||
@@ -233,28 +236,66 @@ class Agent:
|
||||
summary = result.get("summary")
|
||||
return summary.strip() if isinstance(summary, str) and summary.strip() else None
|
||||
|
||||
def _get_peer_executor(self) -> TaskExecutor:
|
||||
# Lazily create a TaskExecutor for peer replies (reuses the model client / workspace).
|
||||
# Raises if no model key is configured; callers fall back to the cached summary.
|
||||
if self._peer_executor is None:
|
||||
self._peer_executor = TaskExecutor(agent_id=self.agent_id, workspace_dir=str(self.workspace_dir))
|
||||
return self._peer_executor
|
||||
|
||||
def _peer_fallback_reply(self) -> dict:
|
||||
# Cheap, no-LLM reply used when the model is unavailable or errors.
|
||||
shared = (
|
||||
f"My latest result: {self.last_summary}. "
|
||||
if self.last_summary
|
||||
else "No completed result yet. "
|
||||
)
|
||||
content = (
|
||||
f"From {self.agent_id} (capabilities: {', '.join(self.capabilities)}). {shared}"
|
||||
"Treat implementation artifacts as the source of truth for behavior and exception semantics."
|
||||
)
|
||||
return {"content": content, "stance": "info", "evidence": self.last_summary or "", "refs": []}
|
||||
|
||||
async def _build_peer_reply(self, query: str, task_id: Optional[str]) -> dict:
|
||||
# Produce a substantive, query-scoped reply grounded in this agent's own work.
|
||||
# Falls back to the cached summary if there is no query, no model key, or the call fails.
|
||||
if not query:
|
||||
return self._peer_fallback_reply()
|
||||
try:
|
||||
executor = self._get_peer_executor()
|
||||
except Exception as e:
|
||||
logger.warning(f"peer reply executor unavailable ({e}); using cached summary")
|
||||
return self._peer_fallback_reply()
|
||||
try:
|
||||
return await asyncio.wait_for(
|
||||
executor.peer_reply(
|
||||
query=query,
|
||||
capabilities=self.capabilities,
|
||||
last_summary=self.last_summary,
|
||||
),
|
||||
timeout=self.PEER_REPLY_TIMEOUT_SECONDS,
|
||||
)
|
||||
except Exception as e:
|
||||
logger.warning(f"peer reply LLM failed ({e}); using cached summary")
|
||||
return self._peer_fallback_reply()
|
||||
|
||||
async def answer_peer_query(self, message: dict):
|
||||
# Respond to an inbound peer query, sharing what this agent has actually done so far.
|
||||
# Respond to an inbound peer query with a substantive, grounded reply (LLM, with fallback).
|
||||
requester = message.get("from_agent_id") or message.get("agent_id")
|
||||
correlation_id = message.get("correlation_id")
|
||||
if not requester or not correlation_id:
|
||||
return
|
||||
shared = (
|
||||
f"My latest result: {self.last_summary}"
|
||||
if self.last_summary
|
||||
else "I have no completed result to share yet."
|
||||
)
|
||||
reply_content = (
|
||||
f"From {self.agent_id} (capabilities: {', '.join(self.capabilities)}). {shared} "
|
||||
"Treat implementation artifacts as the source of truth for behavior and exception semantics."
|
||||
)
|
||||
reply = await self._build_peer_reply((message.get("content") or "").strip(), message.get("task_id"))
|
||||
try:
|
||||
await self.safe_send({
|
||||
"type": "peer_message",
|
||||
"agent_id": self.agent_id,
|
||||
"target_agent_id": requester,
|
||||
"task_id": message.get("task_id"),
|
||||
"content": reply_content,
|
||||
"content": reply.get("content", ""),
|
||||
"stance": reply.get("stance"),
|
||||
"evidence": reply.get("evidence"),
|
||||
"refs": reply.get("refs"),
|
||||
"correlation_id": correlation_id,
|
||||
"is_reply": True,
|
||||
"timestamp": time.time(),
|
||||
@@ -489,7 +530,8 @@ class Agent:
|
||||
message.get("task_id"),
|
||||
message.get("from_agent_id") or message.get("agent_id"),
|
||||
)
|
||||
await self.answer_peer_query(message)
|
||||
# Answer in the background so a slow (LLM) reply doesn't stall the message loop.
|
||||
asyncio.create_task(self.answer_peer_query(message))
|
||||
|
||||
async def handle_message(self, message: dict):
|
||||
# Route each inbound orchestrator message to the appropriate handler.
|
||||
|
||||
@@ -292,6 +292,44 @@ Return ONLY the JSON, no other text."""
|
||||
self._record_openai_usage(response)
|
||||
return response.choices[0].message.content or ""
|
||||
|
||||
async def peer_reply(self, *, query: str, capabilities: list[str],
|
||||
last_summary: Optional[str], max_tokens: Optional[int] = None) -> dict:
|
||||
"""Compose a substantive, grounded reply to a peer agent's query (one bounded LLM call).
|
||||
|
||||
Returns {stance, content, evidence, refs}. `content` is what the requesting agent reads.
|
||||
Bounded by PEER_CONSULT_MAX_TOKENS (default 500).
|
||||
"""
|
||||
if max_tokens is None:
|
||||
max_tokens = int(os.getenv("PEER_CONSULT_MAX_TOKENS", "500"))
|
||||
workspace_files = self._summarize_workspace()
|
||||
workspace_context = self._collect_workspace_context(max_files=8)
|
||||
prompt = f"""You are a specialist agent being consulted by a peer in a collaborative swarm.
|
||||
Your capabilities: {', '.join(capabilities)}
|
||||
Your latest completed work (summary): {last_summary or 'none'}
|
||||
|
||||
A peer asks:
|
||||
{query}
|
||||
|
||||
Your workspace files:
|
||||
{json.dumps(workspace_files, indent=2)}
|
||||
Relevant workspace file contents:
|
||||
{json.dumps(workspace_context, indent=2)}
|
||||
|
||||
Answer concisely and concretely, grounded in YOUR actual work/artifacts. Treat implementation
|
||||
artifacts as the source of truth for behavior and exception semantics; if the peer's assumption
|
||||
conflicts with your work, say so.
|
||||
|
||||
Return ONLY JSON:
|
||||
{{\"stance\": \"agree|disagree|info\", \"content\": \"<concise answer to the peer>\", \"evidence\": \"<what in your work supports this>\", \"refs\": [\"relative/file/path\"]}}"""
|
||||
content = await self._complete(prompt, max_tokens=max_tokens)
|
||||
result = self._parse_json_response(content)
|
||||
if not result.get("content"):
|
||||
result["content"] = (content or "").strip()[:1000]
|
||||
result.setdefault("stance", "info")
|
||||
result.setdefault("evidence", "")
|
||||
result.setdefault("refs", [])
|
||||
return result
|
||||
|
||||
def _empty_usage(self, context: Optional[dict] = None) -> dict:
|
||||
plan = ((context or {}).get("orchestration_plan") or {})
|
||||
billing = plan.get("billing_context") or {}
|
||||
|
||||
@@ -0,0 +1,22 @@
|
||||
# benchmark/ —— 蜂群基准（骨架）
|
||||
|
||||
对应 [`docs/benchmark/`](../docs/benchmark/) 的标准实现入口。依据 **Agent 蜂群指标量化与标准 v2.0**。
|
||||
|
||||
## 实现状态
|
||||
|
||||
| 模块 | 内容 | 状态 |
|
||||
|---|---|---|
|
||||
| `metrics.py` | v2.0 `SwarmMetrics`（15 字段）+ **纯公式**（`pheromone`/`heuristic`/`p_decision`/`reward`/`completion`/`collaboration`/`communication`/`cost`/`robustness`/`governance`/`swarm_score`/`emergence_gain`/`swarm_cost`/`cost_normalized_gain`/`benchmark_agent`）+ v2.0 推荐权重常量 | ✅ 已实现、可单测（`test-benchmark-metrics.py`） |
|
||||
| `collectors/` | base `SwarmMetricsCollector` + `SwarmRunMetricsCollector`（从真实 run 计算 `s_completion`/`s_collaboration`/`s_cost`/`s_robustness`；其余标记 NaN + `coverage=False`） | 🟡 部分落地（4/15 字段真实可算） |
|
||||
| `baselines/` | Single / Chain / Sub-Agent / Strong 基线运行器 | 🔴 未落地 |
|
||||
| `replay/` | 执行回放 | 🔴 未落地 |
|
||||
| `leaderboard/` | 排行榜聚合 | 🔴 未落地 |
|
||||
|
||||
> 公式可计算 ≠ 能自证。**数据采集（collectors）与基线（baselines）未落地前，不得宣称已具备
|
||||
> Agent Swarm 量化自证能力**（不能证明 `Swarm > Single/Chain/Sub-Agent/Strong`，也无法输出完整 `Benchmark_Agent`）。
|
||||
|
||||
## 待落地（按依赖顺序）
|
||||
1. `collectors/`：接入事件流 / Prometheus / OTel / 任务与用量（telemetry-architecture）。
|
||||
2. `baselines/`：四类基线 + 统一任务集（emergence-evaluation / baseline-comparison）。
|
||||
3. `replay/`、`leaderboard/`。
|
||||
4. 权重 v2.0 已给定（`λ`/`w*`/`γ*`，Σλ=1.0）；`α/β/ρ/N_agent/ε` 用 `THETA_DEFAULTS` 推荐初值，调优口径待定。
|
||||
@@ -0,0 +1,8 @@
|
||||
"""Agent Swarm benchmark package.
|
||||
|
||||
实现状态（见 docs/benchmark/swarm-benchmark-protocol.md §6）：
|
||||
- metrics.py：指标公式（纯函数，已实现、可测）+ SwarmMetrics 数据类。
|
||||
- collectors/ baselines/ replay/ leaderboard/：骨架，数据采集与基线运行**未落地**。
|
||||
|
||||
在采集器与基线运行器落地前，不得宣称本仓具备 Agent Swarm 量化自证能力。
|
||||
"""
|
||||
@@ -0,0 +1,14 @@
|
||||
"""Benchmark baselines — Single / Chain / Sub-Agent / Strong（v2.0 §9.1）。
|
||||
|
||||
两部分：
|
||||
- **比较评估器（已落地）**：`comparison.py` 定义共享记录 `BenchmarkRunRecord` 与 `compare`/`evaluate`，
|
||||
在给定 swarm 与 baseline 记录时计算 `G_E` / `G_E,c`（纯函数、可单测）。
|
||||
- **基线运行器（未落地，🔴）**：实际运行各基线系统、在统一任务集上产出 `BenchmarkRunRecord` 的 runner，
|
||||
以及对应的 Quality 插桩（TestPass/CodeReview/UserAcceptance），尚未实现。
|
||||
|
||||
即：一旦 swarm 与 sub-agent 程序按 `BenchmarkRunRecord` 产出记录，比较即可计算。
|
||||
记录的产生（尤其 Quality）见 docs/benchmark/baseline-record-schema.md。
|
||||
"""
|
||||
from .comparison import ( # noqa: F401
|
||||
BenchmarkRunRecord, SYSTEMS, quality, cost_efficiency, unified_metrics, compare, evaluate,
|
||||
)
|
||||
@@ -0,0 +1,123 @@
|
||||
"""Baseline comparison adapter — Agent 蜂群指标量化与标准 v2.0.
|
||||
|
||||
Defines the shared `BenchmarkRunRecord` that BOTH the swarm and each baseline (Single / Chain /
|
||||
Sub-Agent / Strong) emit per benchmark run, plus the evaluator that computes G_E and G_E,c from
|
||||
two records. Pure/deterministic given records — the records themselves must be PRODUCED by the
|
||||
respective programs (instrumentation, esp. Quality, is pending; see
|
||||
docs/benchmark/baseline-record-schema.md). No record → no comparison (we never fabricate scores).
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
from dataclasses import dataclass
|
||||
|
||||
from ..metrics import (
|
||||
quality_score, completion_score, speed_score, cost_score, robustness_score,
|
||||
emergence_gain, swarm_cost, cost_normalized_gain, BASE_COEFFICIENTS,
|
||||
)
|
||||
|
||||
SYSTEMS = {"swarm", "single", "chain", "sub", "strong"}
|
||||
|
||||
|
||||
@dataclass
|
||||
class BenchmarkRunRecord:
|
||||
"""Unified per-run record (v2.0 §9 collection). Both swarm and baselines emit this."""
|
||||
system: str # one of SYSTEMS
|
||||
scenario: str # coding | refactoring | architecture | devops | bugfix
|
||||
task_set_id: str # identifies the shared task set (fairness: same id across systems)
|
||||
n_agent: int # agent count used by this system
|
||||
completed_tasks: int
|
||||
total_tasks: int
|
||||
test_pass_rate: float # 0..100 (Quality input)
|
||||
code_review_score: float # 0..100 (Quality input)
|
||||
user_acceptance: float # 0..100 (Quality input)
|
||||
budget_usd: float # planned cost
|
||||
actual_cost_usd: float # measured model cost
|
||||
model_tokens: int
|
||||
target_time_s: float # planned/target time
|
||||
actual_time_s: float # measured time
|
||||
recovered_failures: int
|
||||
total_failures: int
|
||||
|
||||
@classmethod
|
||||
def from_dict(cls, data: dict) -> "BenchmarkRunRecord":
|
||||
missing = [f for f in cls.__dataclass_fields__ if f not in data]
|
||||
if missing:
|
||||
raise ValueError(f"benchmark record missing required fields: {missing}")
|
||||
if data["system"] not in SYSTEMS:
|
||||
raise ValueError(f"unknown system '{data['system']}' (expected one of {sorted(SYSTEMS)})")
|
||||
return cls(**{f: data[f] for f in cls.__dataclass_fields__})
|
||||
|
||||
|
||||
# --- derived quantities (from a single record) ---
|
||||
def quality(rec: BenchmarkRunRecord) -> float:
|
||||
"""Q = 0.4·TestPass + 0.3·CodeReview + 0.3·UserAcceptance (v2.0 §4)."""
|
||||
return quality_score(rec.test_pass_rate, rec.code_review_score, rec.user_acceptance)
|
||||
|
||||
|
||||
def cost_efficiency(rec: BenchmarkRunRecord) -> float:
|
||||
"""CostEfficiency = 100·Budget/ActualCost (v2.0 §4 E_cost口径)."""
|
||||
return cost_score(rec.budget_usd, rec.actual_cost_usd)
|
||||
|
||||
|
||||
def unified_metrics(rec: BenchmarkRunRecord) -> dict:
|
||||
"""The protocol's unified collection (v2.0 §9): Completion / Quality / Cost / Time / Robustness."""
|
||||
return {
|
||||
"completion": completion_score(rec.completed_tasks, rec.total_tasks),
|
||||
"quality": quality(rec),
|
||||
"cost_efficiency": cost_efficiency(rec),
|
||||
"speed": speed_score(rec.target_time_s, rec.actual_time_s),
|
||||
"robustness": robustness_score(rec.recovered_failures, rec.total_failures),
|
||||
}
|
||||
|
||||
|
||||
# --- comparison (swarm vs one baseline) ---
|
||||
def compare(swarm_rec: BenchmarkRunRecord, base_rec: BenchmarkRunRecord, *,
|
||||
symmetric_cost: bool = True) -> dict:
|
||||
"""Compute G_E and G_E,c for the swarm against one baseline record.
|
||||
|
||||
G_E = Q_swarm − Q_base
|
||||
G_E,c = (Q_swarm / C_swarm) − (Q_base / C_base), C_swarm = N_agent·(CostEfficiency/100+0.5)
|
||||
|
||||
`symmetric_cost`:
|
||||
- True (DEFAULT — Benchmark Owner ratified correction, 2026-06-09): `C_base` uses the SAME
|
||||
formula as C_swarm with the baseline's own N_agent + CostEfficiency → both terms are
|
||||
quality-per-cost (comparable, non-degenerate).
|
||||
- False: literal v2.0 text — `C_base = 1.0` (scale-degenerate; kept for reference only).
|
||||
See docs/benchmark/cost-normalized-gain.md and OWNER-NOTE-cost-normalized-gain.md.
|
||||
"""
|
||||
if swarm_rec.system != "swarm":
|
||||
raise ValueError("swarm_rec.system must be 'swarm'")
|
||||
if swarm_rec.task_set_id != base_rec.task_set_id:
|
||||
raise ValueError("records must share the same task_set_id (fair comparison)")
|
||||
|
||||
q_swarm, q_base = quality(swarm_rec), quality(base_rec)
|
||||
g_e = emergence_gain(q_swarm, q_base)
|
||||
c_swarm = swarm_cost(swarm_rec.n_agent, cost_efficiency(swarm_rec))
|
||||
c_base = swarm_cost(base_rec.n_agent, cost_efficiency(base_rec)) if symmetric_cost else 1.0
|
||||
g_e_c = cost_normalized_gain(q_swarm, c_swarm, q_base, c_base=c_base)
|
||||
return {
|
||||
"base_system": base_rec.system,
|
||||
"base_coefficient": BASE_COEFFICIENTS.get(base_rec.system),
|
||||
"cost_mode": "symmetric" if symmetric_cost else "literal_v2.0",
|
||||
"q_swarm": q_swarm,
|
||||
"q_base": q_base,
|
||||
"c_swarm": c_swarm,
|
||||
"c_base": c_base,
|
||||
"g_e": g_e,
|
||||
"g_e_cost": g_e_c,
|
||||
"raw_gain_positive": g_e > 0,
|
||||
"cost_normalized_positive": g_e_c > 0,
|
||||
}
|
||||
|
||||
|
||||
def evaluate(swarm_rec: BenchmarkRunRecord, baseline_recs: list[BenchmarkRunRecord], *,
|
||||
symmetric_cost: bool = True) -> dict:
|
||||
"""Compare the swarm against all baselines. swarm_valid requires beating ALL on G_E and G_E,c."""
|
||||
results = [compare(swarm_rec, b, symmetric_cost=symmetric_cost) for b in baseline_recs]
|
||||
swarm_valid = bool(results) and all(r["raw_gain_positive"] and r["cost_normalized_positive"] for r in results)
|
||||
return {
|
||||
"scenario": swarm_rec.scenario,
|
||||
"cost_mode": "symmetric" if symmetric_cost else "literal_v2.0",
|
||||
"comparisons": results,
|
||||
"swarm_valid": swarm_valid,
|
||||
}
|
||||
@@ -0,0 +1,20 @@
|
||||
"""Benchmark collectors — 把运行时信号采集成 SwarmMetrics。
|
||||
|
||||
状态：**部分落地**。base `SwarmMetricsCollector.collect()` 故意抛 NotImplementedError（避免占位冒充完成，
|
||||
组织规则 #9）；具体实现 `run_collector.SwarmRunMetricsCollector` 从真实 run 计算
|
||||
`completion`/`collaboration`/`cost`/`robustness`，其余指标返回 NaN 并在 `coverage` 标记 False。
|
||||
未落地部分（gain/communication/p_decision/reward + 基线）见 docs/benchmark/telemetry-architecture.md 与 baseline-comparison.md。
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
from ..metrics import SwarmMetrics
|
||||
|
||||
|
||||
class SwarmMetricsCollector:
|
||||
"""标准 §9 采集接口。子类应从运行时数据源装配 SwarmMetrics。"""
|
||||
|
||||
async def collect(self) -> SwarmMetrics:
|
||||
raise NotImplementedError(
|
||||
"数据采集未落地：需接入 telemetry-architecture 的数据源后，"
|
||||
"用 benchmark.metrics 的公式装配 SwarmMetrics。"
|
||||
)
|
||||
@@ -0,0 +1,137 @@
|
||||
"""SwarmRunMetricsCollector — compute SwarmMetrics from a real swarm run.
|
||||
|
||||
Wires benchmark.metrics formulas to a run's tasks + event stream (orchestrator state).
|
||||
Only metrics with real data sources are computed; the rest are returned as NaN and flagged
|
||||
in `coverage` (False) — we do NOT fake a 0/100 score for uncollected metrics (rule #9).
|
||||
|
||||
Uncollected today (need work flagged in docs/benchmark/swarm-metrics-schema.md):
|
||||
- gain → needs baselines (emergence-evaluation)
|
||||
- communication → needs agent message telemetry (not counted yet)
|
||||
- p_decision → needs τ/η decision scoring (not implemented)
|
||||
- reward → needs weights (standard gives no numeric w*)
|
||||
- governance → only derivable from approvals; NaN when a run has no governed ops
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
import json
|
||||
import math
|
||||
from collections import Counter
|
||||
|
||||
from . import SwarmMetricsCollector
|
||||
from ..metrics import (
|
||||
SwarmMetrics, completion_score, collaboration_score, cost_score, robustness_score,
|
||||
governance_score,
|
||||
)
|
||||
|
||||
|
||||
def _task_cost(task) -> float:
|
||||
"""Extract model_cost_usd from a task's stored result, 0.0 if absent."""
|
||||
result = getattr(task, "result", None)
|
||||
if not result:
|
||||
return 0.0
|
||||
try:
|
||||
data = json.loads(result) if isinstance(result, str) else result
|
||||
return float((data.get("usage") or {}).get("model_cost_usd") or 0.0)
|
||||
except Exception:
|
||||
return 0.0
|
||||
|
||||
|
||||
class SwarmRunMetricsCollector(SwarmMetricsCollector):
|
||||
"""Collect SwarmMetrics for one swarm run (by swarm_id)."""
|
||||
|
||||
def __init__(self, swarm_id: str):
|
||||
self.swarm_id = swarm_id
|
||||
self.coverage: dict[str, bool] = {}
|
||||
|
||||
async def collect(self) -> SwarmMetrics:
|
||||
# Lazy import: the collector reads live orchestrator state.
|
||||
from orchestrator.swarm_runtime import swarm_runtime
|
||||
from orchestrator.task_queue import task_queue, TaskStatus
|
||||
|
||||
run = await swarm_runtime.get_run(self.swarm_id)
|
||||
if not run:
|
||||
raise ValueError(f"run not found: {self.swarm_id}")
|
||||
|
||||
tasks = [t for t in [await task_queue.get_task(tid) for tid in run.task_ids] if t]
|
||||
events = (await swarm_runtime.list_events(self.swarm_id, limit=10000)).get("events", [])
|
||||
event_types = Counter(e.get("event_type") for e in events)
|
||||
|
||||
def status(t):
|
||||
return t.status.value if hasattr(t.status, "value") else t.status
|
||||
|
||||
# --- s_completion (real) ---
|
||||
total = len(tasks)
|
||||
completed = sum(1 for t in tasks if status(t) == TaskStatus.COMPLETED.value)
|
||||
s_completion = completion_score(completed, total)
|
||||
self.coverage["s_completion"] = total > 0
|
||||
|
||||
# --- s_collaboration (real) ---
|
||||
req = event_types.get("handoff.requested", 0)
|
||||
comp = event_types.get("handoff.completed", 0)
|
||||
handoff_success = (100.0 * comp / req) if req else 100.0
|
||||
dep_tasks = [t for t in tasks if t.depends_on]
|
||||
done_ids = {t.task_id for t in tasks if status(t) == TaskStatus.COMPLETED.value}
|
||||
resolved = [t for t in dep_tasks if all(d in done_ids for d in t.depends_on)]
|
||||
dep_resolution = (100.0 * len(resolved) / len(dep_tasks)) if dep_tasks else 100.0
|
||||
per_agent = Counter(t.assigned_agent_id for t in tasks if t.assigned_agent_id)
|
||||
if per_agent:
|
||||
counts = list(per_agent.values())
|
||||
workload_balance = 100.0 * (min(counts) / max(counts))
|
||||
else:
|
||||
workload_balance = 100.0
|
||||
s_collaboration = collaboration_score(handoff_success, dep_resolution, workload_balance)
|
||||
self.coverage["s_collaboration"] = total > 0
|
||||
|
||||
# --- s_robustness (real) ---
|
||||
failures = [t for t in tasks if t.retry_count > 0 or status(t) == TaskStatus.FAILED.value]
|
||||
recovered = [t for t in failures if status(t) == TaskStatus.COMPLETED.value]
|
||||
s_robustness = robustness_score(len(recovered), len(failures))
|
||||
self.coverage["s_robustness"] = True
|
||||
|
||||
# --- s_cost (real if budget + usage present) ---
|
||||
plan = (run.request_body or {}).get("orchestration_plan") or {}
|
||||
budget = plan.get("budget") or {}
|
||||
max_cost = budget.get("max_cost_usd")
|
||||
actual = sum(_task_cost(t) for t in tasks)
|
||||
if isinstance(max_cost, (int, float)) and actual > 0:
|
||||
s_cost = cost_score(float(max_cost), actual)
|
||||
self.coverage["s_cost"] = True
|
||||
else:
|
||||
s_cost = math.nan
|
||||
self.coverage["s_cost"] = False
|
||||
|
||||
# --- s_governance (real only if the run had governed ops / approvals) ---
|
||||
approvals = list((run.approvals or {}).values())
|
||||
if approvals:
|
||||
compliant = sum(1 for a in approvals if a.get("decision") in ("approved", "rejected"))
|
||||
s_governance = governance_score(compliant, len(approvals))
|
||||
self.coverage["s_governance"] = True
|
||||
else:
|
||||
s_governance = math.nan
|
||||
self.coverage["s_governance"] = False
|
||||
|
||||
# --- not yet collectable (see docs/benchmark/metric-coverage-gaps.md) ---
|
||||
# s_gain needs baselines; s_communication needs message telemetry; tau/eta/p_decision need
|
||||
# the decision-layer signals; reward needs quality/risk/rework inputs; s_swarm/g_e/g_e_cost/
|
||||
# benchmark depend on the above (any NaN component → NaN aggregate).
|
||||
for k in ("tau", "eta", "p_decision", "reward", "s_gain", "s_communication",
|
||||
"s_swarm", "g_e", "g_e_cost", "benchmark"):
|
||||
self.coverage[k] = False
|
||||
|
||||
return SwarmMetrics(
|
||||
tau=math.nan,
|
||||
eta=math.nan,
|
||||
p_decision=math.nan,
|
||||
reward=math.nan,
|
||||
s_completion=s_completion,
|
||||
s_gain=math.nan,
|
||||
s_collaboration=s_collaboration,
|
||||
s_communication=math.nan,
|
||||
s_cost=s_cost,
|
||||
s_robustness=s_robustness,
|
||||
s_governance=s_governance,
|
||||
s_swarm=math.nan,
|
||||
g_e=math.nan,
|
||||
g_e_cost=math.nan,
|
||||
benchmark=math.nan,
|
||||
)
|
||||
@@ -0,0 +1,6 @@
|
||||
"""Benchmark leaderboard — 排行榜聚合与展示字段。
|
||||
|
||||
状态：**未落地（骨架）**。展示字段（标准 §11）：Benchmark_Agent / S_swarm / G_E / G_E,c /
|
||||
Reward / Cost Efficiency；场景：Coding / Refactoring / Architecture / DevOps / Bug Fix。
|
||||
需定义 leaderboard schema 与持久化。
|
||||
"""
|
||||
@@ -0,0 +1,177 @@
|
||||
"""Swarm benchmark metric formulas + SwarmMetrics schema.
|
||||
|
||||
Aligned to **Agent 蜂群指标量化与标准 v2.0**. Pure formulas only (given inputs → value, unit-testable);
|
||||
data collection (turning runtime signals into inputs) is partial — see collectors/run_collector.py
|
||||
and docs/benchmark/metric-coverage-gaps.md. Recommended weights/hyperparameters from v2.0 are
|
||||
provided as defaults below.
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
from dataclasses import dataclass
|
||||
|
||||
# --- v2.0 recommended weights / hyperparameters ---
|
||||
TAU_WEIGHTS = {"success": 0.25, "quality": 0.20, "acceptance": 0.20,
|
||||
"cost": 0.10, "time": 0.10, "risk": 0.08, "rollback": 0.07}
|
||||
ETA_WEIGHTS = {"match": 0.25, "urgency": 0.15, "dependency": 0.15, "resource": 0.15,
|
||||
"confidence": 0.10, "risk": 0.10, "budget_pressure": 0.10}
|
||||
REWARD_WEIGHTS = {"s_task": 0.20, "q_quality": 0.20, "v_speed": 0.12, "e_cost": 0.13,
|
||||
"r_robust": 0.13, "g_gov": 0.10, "p_risk": 0.07, "p_rework": 0.05}
|
||||
SWARM_WEIGHTS = {"completion": 0.25, "gain": 0.20, "collaboration": 0.15, "communication": 0.10,
|
||||
"cost": 0.10, "robustness": 0.10, "governance": 0.10}
|
||||
LAMBDA_WEIGHTS = {"lambda1": 0.30, "lambda2": 0.20, "lambda3": 0.25, "lambda4": 0.15, "lambda5": 0.10} # Σ = 1.0
|
||||
THETA_DEFAULTS = {"alpha": 1.0, "beta": 2.0, "rho": 0.10, "n_agent": 5, "epsilon": 0.10}
|
||||
# Q_base reference coefficients (standard §6.1), by baseline type.
|
||||
# ⚠️ PROVISIONAL / METADATA ONLY: these are assumed constants — the standard gives NO derivation.
|
||||
# They are reported as `base_coefficient` but used in NO formula (G_E/G_E,c use raw Q_base).
|
||||
# TODO(benchmark): quantify empirically later, e.g. coefficient = Q_baseline / Q_reference on the
|
||||
# shared task set (reference = strongest baseline or an oracle/acceptance ceiling). Until then,
|
||||
# do not wire into scoring. See docs/benchmark/baseline-comparison.md §1.1.
|
||||
BASE_COEFFICIENTS = {"single": 0.75, "chain": 0.85, "sub": 0.90, "strong": 0.95}
|
||||
|
||||
|
||||
@dataclass
|
||||
class SwarmMetrics:
|
||||
"""v2.0 §8.1 collection interface."""
|
||||
tau: float # 信息素得分
|
||||
eta: float # 启发式得分
|
||||
p_decision: float # 决策概率
|
||||
reward: float # 执行奖励
|
||||
s_completion: float
|
||||
s_gain: float
|
||||
s_collaboration: float
|
||||
s_communication: float
|
||||
s_cost: float
|
||||
s_robustness: float
|
||||
s_governance: float
|
||||
s_swarm: float # 蜂群总分
|
||||
g_e: float # 涌现增益
|
||||
g_e_cost: float # 成本归一化增益
|
||||
benchmark: float # 综合评分
|
||||
|
||||
|
||||
def _ratio_pct(numerator: float, denominator: float, *, empty: float = 0.0) -> float:
|
||||
if not denominator:
|
||||
return empty
|
||||
return 100.0 * numerator / denominator
|
||||
|
||||
|
||||
# --- 决策层（v2.0 §3） ---
|
||||
def pheromone(*, success: float, quality: float, acceptance: float, cost: float, time: float,
|
||||
risk: float, rollback: float, weights: dict = TAU_WEIGHTS) -> float:
|
||||
w = weights
|
||||
return (w["success"] * success + w["quality"] * quality + w["acceptance"] * acceptance
|
||||
- w["cost"] * cost - w["time"] * time - w["risk"] * risk - w["rollback"] * rollback)
|
||||
|
||||
|
||||
def heuristic(*, match: float, urgency: float, dependency: float, resource: float,
|
||||
confidence: float, risk: float, budget_pressure: float, weights: dict = ETA_WEIGHTS) -> float:
|
||||
g = weights
|
||||
return (g["match"] * match + g["urgency"] * urgency + g["dependency"] * dependency
|
||||
+ g["resource"] * resource + g["confidence"] * confidence
|
||||
- g["risk"] * risk - g["budget_pressure"] * budget_pressure)
|
||||
|
||||
|
||||
def action_probability(tau: float, eta: float, alpha: float, beta: float,
|
||||
candidates: list[tuple[float, float]]) -> float:
|
||||
# P(s,a,r) = τ^α·η^β / Σ τ_i^α·η_i^β
|
||||
denom = sum((t ** alpha) * (e ** beta) for t, e in candidates)
|
||||
if not denom:
|
||||
return 0.0
|
||||
return ((tau ** alpha) * (eta ** beta)) / denom
|
||||
|
||||
|
||||
def p_decision(tau: float, eta: float, alpha: float = THETA_DEFAULTS["alpha"],
|
||||
beta: float = THETA_DEFAULTS["beta"]) -> float:
|
||||
# v2.0 §3.3: P_decision = τ^α · η^β · 100
|
||||
return (tau ** alpha) * (eta ** beta) * 100.0
|
||||
|
||||
|
||||
# --- 执行层（v2.0 §4） ---
|
||||
def quality_score(test_pass_rate: float, code_review_score: float, user_acceptance: float) -> float:
|
||||
return 0.4 * test_pass_rate + 0.3 * code_review_score + 0.3 * user_acceptance
|
||||
|
||||
|
||||
def speed_score(target_time: float, actual_time: float) -> float:
|
||||
return _ratio_pct(target_time, actual_time)
|
||||
|
||||
|
||||
def cost_efficiency_score(expected_cost: float, actual_cost: float) -> float:
|
||||
# Owner ruling (v2.1): E_cost (§4.1) == S_cost (§5.1) == CostEfficiency (§6.2) — one quantity,
|
||||
# 100×Budget/ActualCost (Budget≡ExpectedCost, ActualUsage≡ActualCost). Same math as cost_score().
|
||||
return _ratio_pct(expected_cost, actual_cost)
|
||||
|
||||
|
||||
def rework_penalty(rework_count: int, total_tasks: int) -> float:
|
||||
return _ratio_pct(rework_count, total_tasks)
|
||||
|
||||
|
||||
def reward(*, s_task: float, q_quality: float, v_speed: float, e_cost: float, r_robust: float,
|
||||
g_gov: float, p_risk: float, p_rework: float, weights: dict = REWARD_WEIGHTS) -> float:
|
||||
w = weights
|
||||
return (w["s_task"] * s_task + w["q_quality"] * q_quality + w["v_speed"] * v_speed
|
||||
+ w["e_cost"] * e_cost + w["r_robust"] * r_robust + w["g_gov"] * g_gov
|
||||
- w["p_risk"] * p_risk - w["p_rework"] * p_rework)
|
||||
|
||||
|
||||
# --- 蜂群层（v2.0 §5） ---
|
||||
def completion_score(completed_tasks: int, total_tasks: int) -> float:
|
||||
return _ratio_pct(completed_tasks, total_tasks)
|
||||
|
||||
|
||||
def collaboration_score(handoff_success_rate: float, dependency_resolution_rate: float,
|
||||
workload_balance_score: float) -> float:
|
||||
return (0.5 * handoff_success_rate + 0.3 * dependency_resolution_rate
|
||||
+ 0.2 * workload_balance_score)
|
||||
|
||||
|
||||
def communication_score(successful_messages: int, total_messages: int) -> float:
|
||||
return _ratio_pct(successful_messages, total_messages)
|
||||
|
||||
|
||||
def cost_score(budget: float, actual_usage: float) -> float:
|
||||
return _ratio_pct(budget, actual_usage)
|
||||
|
||||
|
||||
def robustness_score(recovered_failures: int, total_failures: int) -> float:
|
||||
return _ratio_pct(recovered_failures, total_failures, empty=100.0)
|
||||
|
||||
|
||||
def governance_score(compliant_operations: int, total_operations: int) -> float:
|
||||
return _ratio_pct(compliant_operations, total_operations, empty=100.0)
|
||||
|
||||
|
||||
def swarm_score(*, completion: float, gain: float, collaboration: float, communication: float,
|
||||
cost: float, robustness: float, governance: float, weights: dict = SWARM_WEIGHTS) -> float:
|
||||
w = weights
|
||||
return (w["completion"] * completion + w["gain"] * gain + w["collaboration"] * collaboration
|
||||
+ w["communication"] * communication + w["cost"] * cost
|
||||
+ w["robustness"] * robustness + w["governance"] * governance)
|
||||
|
||||
|
||||
# --- 涌现增益（v2.0 §6） ---
|
||||
def emergence_gain(q_swarm: float, q_base: float) -> float:
|
||||
# G_E = Q_swarm − Q_base
|
||||
return q_swarm - q_base
|
||||
|
||||
|
||||
def swarm_cost(n_agent: int, cost_efficiency: float) -> float:
|
||||
# v2.0 §6.2: C_swarm = N_agent × (CostEfficiency/100 + 0.5)
|
||||
return n_agent * (cost_efficiency / 100.0 + 0.5)
|
||||
|
||||
|
||||
def cost_normalized_gain(q_swarm: float, c_swarm: float, q_base: float, c_base: float = 1.0) -> float:
|
||||
# v2.0 §6.2 (CHANGED from v1 ratio-of-ratios to a DIFFERENCE):
|
||||
# G_E,c = (Q_swarm / C_swarm) − (Q_base / C_base)
|
||||
if not c_swarm or not c_base:
|
||||
raise ValueError("cost_normalized_gain needs non-zero C_swarm / C_base")
|
||||
return (q_swarm / c_swarm) - (q_base / c_base)
|
||||
|
||||
|
||||
# --- 综合 Benchmark（v2.0 §7） ---
|
||||
def benchmark_agent(*, s_swarm: float, g_e: float, reward: float, observability: float,
|
||||
governance: float, weights: dict = LAMBDA_WEIGHTS, validate: bool = True) -> float:
|
||||
"""Benchmark_Agent = λ1·S_swarm + λ2·G_E + λ3·R + λ4·O + λ5·Gov, with Σλ = 1.0 (v2.0 §7.1)."""
|
||||
if validate and abs(sum(weights.values()) - 1.0) > 1e-6:
|
||||
raise ValueError(f"λ weights must sum to 1.0 (got {sum(weights.values())})")
|
||||
return (weights["lambda1"] * s_swarm + weights["lambda2"] * g_e + weights["lambda3"] * reward
|
||||
+ weights["lambda4"] * observability + weights["lambda5"] * governance)
|
||||
@@ -0,0 +1,5 @@
|
||||
"""Benchmark replay — 回放一次 swarm/benchmark 执行。
|
||||
|
||||
状态：**未落地（骨架）**。事件流已按 swarm_id 顺序持久化，可作为回放数据源；
|
||||
需定义快照格式与回放接口。见 docs/integration/audit-trace-schema.md §4。
|
||||
"""
|
||||
@@ -0,0 +1,57 @@
|
||||
# 指标采集覆盖与缺口根因分析（Metric Coverage Gaps）
|
||||
|
||||
> 状态：**现状分析**。说明 `SwarmRunMetricsCollector`（`benchmark/collectors/run_collector.py`）当前能真实计算哪些指标、哪些返回 `NaN + coverage=False`，以及**为什么**。
|
||||
>
|
||||
> 配套：[`swarm-metrics-schema.md`](./swarm-metrics-schema.md)、[`telemetry-architecture.md`](./telemetry-architecture.md)、[`emergence-evaluation.md`](./emergence-evaluation.md)、[`baseline-comparison.md`](./baseline-comparison.md)、[`governance-score.md`](./governance-score.md)。
|
||||
|
||||
## 1. 一句话根因
|
||||
|
||||
本仓最初是**多 Agent 工作流执行器**，不是**被插桩的基准测量目标**。能算出来的指标是「执行过程本就会产生」的副产物；其余指标各自缺少执行器从不需要产生的东西（基线 / 计数器 / 决策机制 / 输入）。**v2.0 已给定全部权重（τ/η/reward/λ）**，但这不改变「输入/机制缺失」的根因。
|
||||
|
||||
## 2. 覆盖现状（v2.0 `SwarmMetrics` 共 15 字段，4 个真实可算）
|
||||
|
||||
| 字段 | 状态 | 数据来源 / 缺口 |
|
||||
|---|---|---|
|
||||
| `s_completion` | ✅ 真实 | 任务状态统计 |
|
||||
| `s_collaboration` | ✅ 真实 | `handoff.*` 事件 + `depends_on` + `assigned_agent_id` |
|
||||
| `s_cost` | ✅ 真实 | 每任务 `usage.model_cost_usd` + 请求 `budget.max_cost_usd` |
|
||||
| `s_robustness` | ✅ 真实 | `retry_count` + 任务状态 |
|
||||
| `s_governance` | 🟡 部分 | 仅审批可派生；无审批 → NaN |
|
||||
| `s_gain` | 🔴 NaN | 需基线 |
|
||||
| `s_communication` | 🔴 NaN | 无消息计数 |
|
||||
| `tau` / `eta` / `p_decision` | 🔴 NaN | 无 τ/η 决策引擎 |
|
||||
| `reward` | 🔴 NaN | 权重已给定，**输入**未采集 |
|
||||
| `s_swarm` / `g_e` / `g_e_cost` / `benchmark` | 🔴 NaN | 依赖上述（任一 NaN → 聚合 NaN） |
|
||||
|
||||
> 不可算的指标返回 `NaN` 并在 `coverage` 标记 `False`，**不伪造 0/100 分值**（组织规则 #9）。
|
||||
|
||||
## 3. 为什么这 4 个能算
|
||||
|
||||
它们都是执行器正常运行的副产物，已存在于 orchestrator 状态：
|
||||
- `completion` ← 队列本就跟踪任务状态。
|
||||
- `collaboration` ← 移交事件、依赖、被分配 Agent 都是派发所需。
|
||||
- `cost` ← 计费归因本就记录每任务 `model_cost_usd`；预算在请求里。
|
||||
- `robustness` ← 重试逻辑本就维护 `retry_count` 与状态。
|
||||
|
||||
## 4. 为什么这 5 个算不出（逐项根因）
|
||||
|
||||
| 指标 | 根因（缺什么） | 类别 | 关闭成本 |
|
||||
|---|---|---|---|
|
||||
| `gain`（`G_E = Q_swarm − Q_base`） | **本质是对比指标**，单次 swarm run 无法自算；缺 baseline 运行器（Single/Strong/Chain/Sub-Agent）、对比 harness 与数据集 | 缺**基线** | 高（跨团队/基础设施/设计决策） |
|
||||
| `communication`（`Successful/Total messages`） | peer/WS 消息只**路由**、从不**计数**；无成功/总数计数器，数据流过但未插桩 | 缺**计数器** | 低（加计数器） |
|
||||
| `governance`（`Compliant/Total ops`） | 有审批机制，但无「受治理/敏感操作 vs 合规」计数；更广的治理面（tool/MCP 权限、allowed_paths 强制）未实现，**没有受治理操作记录可计数**；无审批的 run → 0 操作 → 空集 → NaN | 缺**计数器 + 策略强制点** | 中（计数器 + 部分强制点） |
|
||||
| `p_decision`（`τ^α·η^β·100`） | v2.0 已给公式，但派发仍是**确定性贪心能力匹配**（`can_agent_run_task`）；ACO 决策模型**未实现**——无信息素 `τ`（历史有效性追踪）、无启发式 `η` 评分；没有可测的概率决策 | 缺**整套机制** | 高（新建决策引擎 + 历史库） |
|
||||
| `reward`（`w₁·S_task + … − w₈·P_rework`） | v2.0 **已给定 `w₁..w₈`**；但**输入**未采集（`Q_quality` 需 TestPass/CodeReview/UserAcceptance、`P_risk`、`P_rework`）→ 仍不可算 | 缺**输入**（权重已定） | 中（质量/CI/风险/返工接入） |
|
||||
|
||||
## 5. 关闭路径（按成本排序）
|
||||
|
||||
1. **低成本（本仓可做）**：`s_communication`、`s_governance` — 在 peer/WS 发送处与受治理操作处加计数器，按 `swarm_id` 聚合。可把真实可算项从 4 提到 6。
|
||||
2. **中成本**：`reward` — 权重 v2.0 已定；接入 `Q_quality`（CI/评审/验收）与 risk/rework 输入即可算。
|
||||
3. **高成本**：
|
||||
- `p_decision` — 设计并实现 `τ/η/P` 决策引擎与历史有效性追踪（改变派发机制）。
|
||||
- `gain`（及由其驱动的 `Benchmark_Agent`、`G_E,c`、「Swarm > baselines」验收）— 实现 4 类基线运行器 + 统一数据集 + 对比/显著性，属跨团队与基础设施工作（见 baseline-comparison）。
|
||||
|
||||
## 6. 影响
|
||||
|
||||
- 在 `gain` 落地前，**无法输出完整 `Benchmark_Agent`，也无法证明 Swarm 优于任一基线** —— 这正是工单「当前更接近多 Agent Workflow Demo」的判断依据。
|
||||
- `NaN + coverage=False` 是这一缺口的**诚实证据**；不得以占位分值对外宣称已具备量化自证能力。
|
||||
@@ -0,0 +1,53 @@
|
||||
# Owner Note：`G_E,c` 成本归一化增益公式退化 —— ✅ 已裁定（对称 C_base）
|
||||
|
||||
> 提交：Swarm 仓（HeiCode-Swarm）· 面向：Benchmark Owner（Agent 蜂群指标量化与标准 v2.0 §6.2）
|
||||
> 状态：**✅ 已裁定（2026-06-09）**。Benchmark Owner 采纳「对称 `C_base`」修正：`C_base` 用基线自身
|
||||
> `N_agent` 与 `CostEfficiency` 同式计算。已设为 `benchmark/baselines.compare` 的**默认**（`symmetric_cost=True`）；
|
||||
> 字面 v2.0（`C_base=1.0`）仅作参考（`symmetric_cost=False`）。下文保留问题背景与依据。
|
||||
|
||||
## 1. 问题
|
||||
|
||||
v2.0 §6.2：
|
||||
```
|
||||
G_E,c = (Q_swarm / C_swarm) − (Q_base / C_base)
|
||||
C_swarm = N_agent × (CostEfficiency/100 + 0.5)
|
||||
C_base = 1.0
|
||||
```
|
||||
`C_base` 固定为 `1.0`，而 `C_swarm ≈ O(N_agent)`（数量级 ~数倍）。两项尺度不一致：
|
||||
- `Q_base / C_base = Q_base`（满量级，如 65.5）
|
||||
- `Q_swarm / C_swarm`（被压缩，如 85.5/12.5 ≈ 6.84）
|
||||
|
||||
→ 基线项恒压倒实验项，**几乎任何多 Agent 蜂群的 `G_E,c` 都为负**，与质量无关。该指标**退化**，无法区分「高性价比蜂群」与「堆 Agent 蜂群」——恰与其设计意图相反。
|
||||
|
||||
**复现**（`scripts/test-baseline-comparison.py`）：Q_swarm=85.5 > Q_sub=65.5（`G_E=+20`），但字面公式 `G_E,c = 6.84 − 65.5 = −58.7`。
|
||||
|
||||
## 2. 根因
|
||||
|
||||
`C_base = 1.0` 疑为简化/笔误：它让基线「零成本」，使差值失去「单位成本质量」的可比性。
|
||||
|
||||
## 3. 建议修复（推荐：对称成本归一化）
|
||||
|
||||
让 `C_base` 与 `C_swarm` 同式计算（用基线自身的 `N_agent` 与 `CostEfficiency`）：
|
||||
```
|
||||
C_base = N_agent_base × (CostEfficiency_base/100 + 0.5)
|
||||
```
|
||||
则两项均为「单位成本质量」，可比、非退化，且保留 v2.0「差值 + 动态成本」的设计意图。
|
||||
|
||||
- 复现示例（对称）：`C_base = 3×(166.7/100+0.5) = 6.5` → `G_E,c = 6.84 − 10.08 = −3.24`。这是**有意义**的结果：蜂群质量更高，但用 5 个 Agent 的代价不成比例；当质量增益足够大时 `G_E,c` 可转正。
|
||||
|
||||
备选：恢复 v1 的**比值形式** `G_E,c = (Q_swarm/C_swarm)/(Q_base/C_base)`（阈值 `>1`），天然尺度无关。
|
||||
|
||||
## 4. 本仓现状（已就绪，等裁定）
|
||||
|
||||
- `benchmark/metrics.py:cost_normalized_gain` 接受 `c_base` 参数。
|
||||
- `benchmark/baselines/compare(..., symmetric_cost=False)`：**默认 = 字面 v2.0（C_base=1.0）**；`symmetric_cost=True` = 上述对称修复。
|
||||
- 由 `scripts/test-baseline-comparison.py` 覆盖两种模式。
|
||||
- 一旦 Owner 确认，将对称模式设为默认并更新 v2.0 文档与本仓默认值。
|
||||
|
||||
## 5. 裁定结果（2026-06-09）
|
||||
|
||||
- [x] **采用对称 `C_base`**（默认 `symmetric_cost=True`）。
|
||||
- [ ] ~~改回比值形式（阈值 >1）~~ 未采用。
|
||||
- [ ] ~~维持 `C_base=1.0`~~ 仅作参考保留（`symmetric_cost=False`）。
|
||||
- [x] `CostEfficiency` 口径：统一为 `100·Budget/ActualCost`（= `S_cost` = `E_cost`，Owner 裁定 v2.1）。
|
||||
- [ ] **参考系数 ×0.75/0.85/0.90/0.95**：Owner 决定**暂留为元数据、暂不接入公式**，**标记待后续实测量化**（如 `系数 = Q_baseline/Q_reference`）。其含义见 `baseline-comparison.md §1.1`。
|
||||
@@ -0,0 +1,67 @@
|
||||
# Baseline Comparison（基线对比）
|
||||
|
||||
> 状态：**规划中（方法已定义，基线运行器未落地）**。
|
||||
>
|
||||
> 依据：**Agent 蜂群指标量化与标准 v2.0 §9.1**。配套：[`emergence-evaluation.md`](./emergence-evaluation.md)、[`cost-normalized-gain.md`](./cost-normalized-gain.md)、[`swarm-benchmark-protocol.md`](./swarm-benchmark-protocol.md)。
|
||||
|
||||
## 1. 基线与实验组（标准 §9.1）
|
||||
|
||||
| 组 | 系统 | 说明 | 参考系数 |
|
||||
|---|---|---|---|
|
||||
| Baseline A | Single Agent | 单 Agent 直接完成（最小基准） | ×0.75 |
|
||||
| Baseline B | Chain Agent | 串行链式编排（无蜂群协作/评审） | ×0.85 |
|
||||
| Baseline C | Sub-Agent | 主从结构委派（无对等协作/评审循环） | ×0.90 |
|
||||
| Baseline D | Strong Agent | 高能力单体（更大模型/更长上下文） | ×0.95 |
|
||||
| Experimental | **Swarm Agent**（本仓） | 分解→派发→协作→评审/重做→汇总 | — |
|
||||
|
||||
### 1.1 参考系数的含义（暂定值 · 待量化 · 暂不接入公式）
|
||||
|
||||
> ⚠️ **状态：暂定 / 仅元数据**。×0.75/0.85/0.90/0.95 为人为设定的强度先验，标准**未给计算方法**；**不参与任何公式**（仅在 `compare(...)` 结果中作 `base_coefficient` 上报）。**待后续实测量化**（如 `系数 = Q_baseline/Q_reference`，按场景标定）后再决定是否接入。
|
||||
|
||||
参考系数（Single 0.75 < Chain 0.85 < Sub 0.90 < Strong 0.95）表示**各基线的相对能力强度**——把四类基线按「越接近满能力」排序：Single 最弱（约 0.75），Strong 最强（约 0.95）。其用意是：
|
||||
|
||||
- **打败越强的基线越有价值**。对 Strong（0.95）取得的质量增益，比对 Single（0.75）取得同样增益更能证明涌现——因为强基线的「上行空间」更小（headroom ≈ `1 − 系数`）。
|
||||
- 因此它本质是一个**难度 / 可信度因子**：用于在跨基线汇总时给「战胜强基线」的增益更高权重，防止「我们赢了单 Agent」被当作强涌现证据。
|
||||
|
||||
可能的数学接入方式（**标准 v2.x 未指定，待裁定**）：
|
||||
- 难度加权增益：`weighted_G_E = (Q_swarm − Q_base) × 系数`（战胜强基线权重更高）；或
|
||||
- headroom 归一化：`G_E_norm = (Q_swarm − Q_base) / (1 − 系数)`（放大对强基线的增益）；或
|
||||
- 期望基线缩放：`Q_base_expected = 系数 × Q_reference`。
|
||||
|
||||
**当前实现**：系数仅作为 `compare(...)` 结果里的 `base_coefficient` **元数据上报，不参与任何公式**（`G_E`/`G_E,c` 用原始 `Q_base`）。代码常量见 `benchmark/metrics.py:BASE_COEFFICIENTS`。
|
||||
|
||||
## 2. 统一采集指标（标准 §10）
|
||||
|
||||
所有组在**同一任务集**上运行，统一采集：`Completion`、`Quality`、`Cost`、`Time`、`Robustness`。
|
||||
由此得到每组 `Q_*`（质量，口径见 emergence-evaluation §3）与 `C_*`（成本，见 cost-normalized-gain §3）。
|
||||
|
||||
## 3. 对比方法
|
||||
|
||||
1. **同一任务集 / 同一场景**（Coding / Refactoring / Architecture / DevOps / Bug Fix）。
|
||||
2. **同等约束**：相同模型网关、相同预算上限口径（避免实验组单独放宽）。
|
||||
3. 每组多次运行，报告均值与方差。
|
||||
4. 输出：
|
||||
- `G_E = Q_swarm − Q_base`（对 A/B/C/D 各算一次）。
|
||||
- `G_E,c = (Q_swarm/C_swarm)/(Q_base/C_base)`。
|
||||
- 成立判定：对**全部** A/B/C/D 满足 `G_E > 0` 且 `G_E,c > 1`。
|
||||
|
||||
## 4. 公平性约束（防止虚高）
|
||||
|
||||
- 实验组不得使用基线没有的额外资源/预算/工具（除「蜂群协作与评审」本身）。
|
||||
- 成本必须计入**多 Agent + 评审重做**的累计（天然反映在按 `swarm_id` 聚合的 `usage`，见 cost-normalized-gain §3）。
|
||||
- 报告需同时给出 `G_E` 与 `G_E,c`；只报 `G_E` 不足以验收。
|
||||
|
||||
## 5. 实现状态
|
||||
|
||||
| 组件 | 位置 | 状态 |
|
||||
|---|---|---|
|
||||
| 基线运行器 A–D | `benchmark/baselines/` | 🔴 未实现 |
|
||||
| 统一任务集 / 数据集 | `test-data/` + `benchmark/baselines/` | 🔴 未实现 |
|
||||
| 指标采集 | `benchmark/collectors/` | 🔴 未实现 |
|
||||
| 对比与显著性 | `benchmark/` | 🔴 未实现 |
|
||||
|
||||
## 6. 待对齐
|
||||
|
||||
- 四类基线的标准实现边界（尤其 Strong / Chain / Sub-Agent）。
|
||||
- 统一任务集与场景数据集。
|
||||
- 运行次数、方差/显著性门槛与公平性约束的强制方式。
|
||||
@@ -0,0 +1,71 @@
|
||||
# Baseline Run Record Schema（基线运行记录 · 共享契约）
|
||||
|
||||
> 状态：**Schema + 比较评估器已落地；记录生产（尤其 Quality）未落地**。
|
||||
>
|
||||
> 依据：**Agent 蜂群指标量化与标准 v2.0 §6, §9**。实现：`benchmark/baselines/comparison.py`（`BenchmarkRunRecord` / `compare` / `evaluate`）。配套：[`baseline-comparison.md`](./baseline-comparison.md)、[`emergence-evaluation.md`](./emergence-evaluation.md)、[`cost-normalized-gain.md`](./cost-normalized-gain.md)、[`IMPORTANT-metric-coverage-gaps.md`](./IMPORTANT-metric-coverage-gaps.md)。
|
||||
|
||||
## 1. 目的
|
||||
|
||||
蜂群验收的核心是 **swarm vs baseline 对比**（尤其 sub-agent，Baseline C ×0.90）。要算 `G_E`/`G_E,c`，**swarm 与每个基线必须在同一任务集、同等约束下，产出同一份记录**。本文定义这份共享记录 `BenchmarkRunRecord`：**两侧程序各自产出**，由本仓评估器统一比较。
|
||||
|
||||
## 2. 共享记录 `BenchmarkRunRecord`
|
||||
|
||||
| 字段 | 类型 | 含义 |
|
||||
|---|---|---|
|
||||
| `system` | str | `swarm` / `single` / `chain` / `sub` / `strong` |
|
||||
| `scenario` | str | `coding` / `refactoring` / `architecture` / `devops` / `bugfix` |
|
||||
| `task_set_id` | str | 共享任务集 ID（**同一 id 才可比**） |
|
||||
| `n_agent` | int | 该系统使用的 Agent 数 |
|
||||
| `completed_tasks` / `total_tasks` | int | 完成度（Completion） |
|
||||
| `test_pass_rate` / `code_review_score` / `user_acceptance` | float(0–100) | 质量三输入（Quality） |
|
||||
| `budget_usd` / `actual_cost_usd` | float | 计划/实测成本（Cost、CostEfficiency） |
|
||||
| `model_tokens` | int | token 用量 |
|
||||
| `target_time_s` / `actual_time_s` | float | 计划/实测耗时（Time、Speed） |
|
||||
| `recovered_failures` / `total_failures` | int | 恢复/总失败（Robustness） |
|
||||
|
||||
JSON 示例：
|
||||
```json
|
||||
{
|
||||
"system": "sub", "scenario": "coding", "task_set_id": "coding-set-1", "n_agent": 3,
|
||||
"completed_tasks": 9, "total_tasks": 10,
|
||||
"test_pass_rate": 70, "code_review_score": 60, "user_acceptance": 65,
|
||||
"budget_usd": 10, "actual_cost_usd": 6, "model_tokens": 42000,
|
||||
"target_time_s": 600, "actual_time_s": 720, "recovered_failures": 1, "total_failures": 2
|
||||
}
|
||||
```
|
||||
|
||||
## 3. 派生量（评估器，已实现）
|
||||
|
||||
- `Q = quality(rec)` = `0.4·TestPass + 0.3·CodeReview + 0.3·UserAcceptance`
|
||||
- `CostEfficiency = 100·Budget/ActualCost`
|
||||
- 统一采集 `unified_metrics(rec)`：Completion / Quality / CostEfficiency / Speed / Robustness（v2.0 §9）
|
||||
- `compare(swarm_rec, base_rec)`（默认 `symmetric_cost=True`，Owner 修正 2026-06-09）：
|
||||
- `G_E = Q_swarm − Q_base`
|
||||
- `C_swarm = N_agent_swarm·(CostEfficiency_swarm/100 + 0.5)`；`C_base = N_agent_base·(CostEfficiency_base/100 + 0.5)`（对称）
|
||||
- `G_E,c = (Q_swarm/C_swarm) − (Q_base/C_base)`
|
||||
- 字面 v2.0（`C_base=1.0`）仅 `symmetric_cost=False` 保留参考；背景见 [`OWNER-NOTE-cost-normalized-gain.md`](./OWNER-NOTE-cost-normalized-gain.md)。
|
||||
- `evaluate(swarm_rec, [baselines])`：对全部基线比较；`swarm_valid` 要求对每个基线 `G_E>0` 且 `G_E,c>0`。
|
||||
|
||||
## 4. 谁产出什么
|
||||
|
||||
| 字段族 | swarm 侧现状 | sub-agent 侧需产出 |
|
||||
|---|---|---|
|
||||
| Completion (`completed/total`) | ✅ 任务状态 | ✅ 需提供 |
|
||||
| Cost (`actual_cost_usd`/`model_tokens`)、`n_agent` | ✅ usage 已采集 | ✅ 需提供 |
|
||||
| Time (`actual_time_s`) | ✅ 时间戳 | ✅ 需提供 |
|
||||
| Robustness (`recovered/total_failures`) | 🟡 需计数 | ✅ 需提供 |
|
||||
| **Quality (`test_pass_rate`/`code_review_score`/`user_acceptance`)** | 🔴 **未插桩** | 🔴 **需插桩** |
|
||||
|
||||
> **关键阻塞**：Quality 在 **swarm 与 sub-agent 两侧都未插桩**（无 CI/测试通过率、评审分、验收）。没有 Quality 就无 `Q` → 无 `G_E`。这是「记录已可比较、但记录尚不可生产」的根本原因。
|
||||
|
||||
## 5. 接入路径
|
||||
|
||||
1. 双方各实现一个「跑统一任务集 → 产出 `BenchmarkRunRecord`」的 runner（sub-agent 侧在其程序内；swarm 侧可由 `SwarmRunMetricsCollector` 扩展 + Quality 插桩）。
|
||||
2. 统一任务集与 `task_set_id`、同等约束（同模型网关、同预算口径）。
|
||||
3. 把两侧记录交给 `benchmark/baselines/evaluate(...)` → 得 `G_E`/`G_E,c` 与 `swarm_valid`。
|
||||
|
||||
## 6. 待对齐
|
||||
|
||||
- Quality 三输入的采集口径与 CI/评审/验收来源。
|
||||
- `CostEfficiency` 口径（`S_cost` vs `E_cost`）与基线参考系数（×0.75/0.85/0.90/0.95）在公式中的确切作用。
|
||||
- 统一任务集与数据集（各场景）。
|
||||
@@ -0,0 +1,63 @@
|
||||
# Cost-Normalized Gain（成本归一化增益）
|
||||
|
||||
> 状态：**规划中（公式已对齐，ROI 评估器未落地）**。
|
||||
>
|
||||
> 依据：**Agent 蜂群指标量化与标准 v2.0 §6.2**。配套：[`emergence-evaluation.md`](./emergence-evaluation.md)、[`swarm-metrics-schema.md`](./swarm-metrics-schema.md)、[`usage-billing` 用量](../integration/usage-billing-schema.md)。实现：`benchmark/metrics.py` 的 `cost_normalized_gain` / `swarm_cost`。
|
||||
|
||||
## 1. 公式（v2.0 + Owner 修正 2026-06-09：对称成本归一化）
|
||||
|
||||
```
|
||||
G_E,c = (Q_swarm / C_swarm) − (Q_base / C_base)
|
||||
|
||||
C_swarm = N_agent_swarm × (CostEfficiency_swarm / 100 + 0.5)
|
||||
C_base = N_agent_base × (CostEfficiency_base / 100 + 0.5)
|
||||
```
|
||||
|
||||
- `Q_*`：质量（同 emergence-evaluation §3）。`CostEfficiency`：成本效率 `100·Budget/ActualCost`（**= §5.1 `S_cost` = §4.1 `E_cost`，同一量**，Owner 裁定 v2.1）。
|
||||
- 判读：`G_E,c > 0` 表示**单位成本质量**优于基线；`< 0` 表明蜂群以更高的 Agent 成本换取质量、性价比不及基线，应缩减 `N_agent`。
|
||||
|
||||
> ✅ **Owner 修正（2026-06-09）**：原始 v2.0 §6.2 取 `C_base = 1.0`，与 `C_swarm ≈ O(N_agent)` 尺度不一致，
|
||||
> 致使几乎任何多 Agent 蜂群 `G_E,c < 0`（与质量无关，指标退化）。经 Benchmark Owner 裁定，改为**对称归一化**
|
||||
> （`C_base` 用基线自身 `N_agent` 与 `CostEfficiency` 同式计算），使两项均为单位成本质量、可比。
|
||||
> 实现为 `benchmark/baselines.compare` 的**默认**（`symmetric_cost=True`）；字面 v2.0（`C_base=1.0`）仅 `symmetric_cost=False` 保留参考。背景见 [`OWNER-NOTE-cost-normalized-gain.md`](./OWNER-NOTE-cost-normalized-gain.md)。
|
||||
|
||||
## 2. 目的
|
||||
|
||||
防止通过**无限扩张 Agent 数量 / Token** 获得虚假涌现增益：`C_swarm` 随 `N_agent` 线性增长，堆 Agent 会拉低 `Q_swarm/C_swarm`。蜂群验收**不只**看 `G_E > 0`，还要 `G_E,c > 0`。
|
||||
|
||||
## 3. 成本 C 口径
|
||||
|
||||
`C` 建议为单次任务集的总成本，至少包含（数据源见 usage-billing-schema）：
|
||||
```
|
||||
C = model_cost_usd（所有 task / 所有 review 重做累计）
|
||||
[+ runtime / 基础设施成本，待接入]
|
||||
```
|
||||
|
||||
| 成本项 | 本仓可采集 |
|
||||
|---|---|
|
||||
| `model_cost_usd`（含多 Agent、多 review 重做累计） | 🟡 usage 已采集，按 `swarm_id` 聚合即可 |
|
||||
| `model_tokens`（token 效率视角） | 🟡 已采集 |
|
||||
| 基础设施成本（cpu/memory/Pod） | 🔴 未接入账本（见 usage-billing §6） |
|
||||
|
||||
> 多 Agent / review 重做的成本**已隐含累计**在按 `swarm_id` 聚合的 `usage` 中，因此 `C_swarm` 天然反映「堆 Agent / 重做」的代价——这正是 `G_E,c` 要惩罚的。
|
||||
|
||||
## 4. 衍生视角（Leaderboard）
|
||||
|
||||
- **Token 效率**：`Q / total_tokens`。
|
||||
- **Cost Efficiency**：`Q / model_cost_usd`。
|
||||
- **Swarm ROI**：`G_E,c` 本身。
|
||||
|
||||
## 5. 实现状态
|
||||
|
||||
| 组件 | 状态 |
|
||||
|---|---|
|
||||
| `usage` 成本聚合（按 swarm_id） | 🟡 数据具备，无聚合器 |
|
||||
| 成本归一化评估器（G_E,c） | 🔴 未实现 |
|
||||
| 多 Agent ROI / token 效率 / 成本效率 | 🔴 未实现 |
|
||||
| 基础设施成本纳入 C | 🔴 未接入 |
|
||||
|
||||
## 6. 待对齐
|
||||
|
||||
- `C` 是否纳入基础设施成本（与 Billing Team / usage-billing §7 对齐）。
|
||||
- `Q` 口径与 emergence-evaluation 一致化。
|
||||
- `CostEfficiency` 的确切口径（`S_cost` vs `E_cost`）与 `G_E,c` 的验收阈值（`>0` 之外是否设场景化下限）。
|
||||
@@ -0,0 +1,68 @@
|
||||
# Emergence Evaluation（涌现增益评估）
|
||||
|
||||
> 状态：**规划中（公式已对齐，基线运行器/评估器未落地）**。
|
||||
>
|
||||
> 依据：**Agent 蜂群指标量化与标准 v2.0 §6.1, §9**。配套：[`baseline-comparison.md`](./baseline-comparison.md)、[`cost-normalized-gain.md`](./cost-normalized-gain.md)、[`swarm-benchmark-protocol.md`](./swarm-benchmark-protocol.md)。
|
||||
|
||||
## 1. 原始增益
|
||||
|
||||
```
|
||||
G_E = Q_swarm − Q_base
|
||||
```
|
||||
|
||||
`Q_base` 取自四类基线之一（标准 §6.1，附**参考系数**，由低到高）：
|
||||
|
||||
| 基准 | 类型 | 参考系数 |
|
||||
|---|---|---|
|
||||
| Baseline A | Single Agent | ×0.75 |
|
||||
| Baseline B | Chain Agent | ×0.85 |
|
||||
| Baseline C | Sub-Agent | ×0.90 |
|
||||
| Baseline D | Strong Agent | ×0.95 |
|
||||
|
||||
> ⚠️ **参考系数为暂定值（待量化）**：×0.75/0.85/0.90/0.95 是人为设定的强度先验，标准未给计算方法；**当前不参与公式**（`G_E` 用原始 `Q_base`），仅作元数据。后续应实测量化（见 [`baseline-comparison.md §1.1`](./baseline-comparison.md)）。代码常量见 `benchmark/metrics.py:BASE_COEFFICIENTS`。
|
||||
|
||||
## 2. 成立条件（蜂群是否真正优于基线）
|
||||
|
||||
蜂群成立要求对**全部**基线为正增益：
|
||||
```
|
||||
Q_swarm > Q_single ∧ Q_swarm > Q_strong ∧ Q_swarm > Q_chain ∧ Q_swarm > Q_sub
|
||||
```
|
||||
否则蜂群不成立（仅是「多 Agent 工作流」而非有涌现的蜂群）。
|
||||
|
||||
> 仅有原始增益不足以验收：还须经成本归一化（见 [`cost-normalized-gain.md`](./cost-normalized-gain.md)），证明增益非「堆 Agent / Token」虚高。
|
||||
|
||||
## 3. Q（质量）口径
|
||||
|
||||
`Q` 采用统一质量度量。建议复用标准 §5.3：
|
||||
```
|
||||
Q = Q_quality = 0.4·TestPassRate + 0.3·CodeReviewScore + 0.3·UserAcceptance
|
||||
```
|
||||
并在每组统一采集 `Completion / Quality / Cost / Time / Robustness`（标准 §10），按场景（Coding / Refactoring / Architecture / DevOps / Bug Fix）分别计算 `G_E`。
|
||||
|
||||
> ⚠️ `Q` 的精确口径（是否等于 `Q_quality`，或综合 Completion/Robustness）v2.0 未唯一指定 → 待对齐。
|
||||
|
||||
## 4. 评测流程（草案）
|
||||
|
||||
1. 固定任务集（按场景）。
|
||||
2. 在 A/B/C/D 与 Swarm 上**同一任务集**运行，统一采集指标（见 swarm-metrics-schema §0）。
|
||||
3. 计算每场景 `Q_*`，得 `G_E = Q_swarm − Q_base`。
|
||||
4. 进入成本归一化（`G_E,c`）。
|
||||
5. 多次运行取均值并报告方差/显著性。
|
||||
|
||||
## 5. 实现状态
|
||||
|
||||
| 组件 | 位置（规划） | 状态 |
|
||||
|---|---|---|
|
||||
| 基线运行器 A–D | `benchmark/baselines/` | 🔴 未实现 |
|
||||
| 基线 benchmark 套件 / 数据集 | `benchmark/baselines/` + `test-data/` | 🔴 未实现 |
|
||||
| 基线对比 | baseline-comparison | 🔴 未实现 |
|
||||
| 涌现评估器（G_E） | `benchmark/` | 🔴 未实现 |
|
||||
| 归一化增益评估器（G_E,c） | cost-normalized-gain | 🔴 未实现 |
|
||||
|
||||
> 在以上落地前，**不得宣称已验证蜂群涌现能力**（重大能力缺口）。
|
||||
|
||||
## 6. 待对齐
|
||||
|
||||
- `Q` 的唯一口径。
|
||||
- 四类基线的标准实现（尤其 Strong Agent / Chain Agent / Sub-Agent 的定义边界）与统一数据集。
|
||||
- 运行次数、方差/显著性门槛。
|
||||
@@ -0,0 +1,53 @@
|
||||
# Governance Score（治理评分）
|
||||
|
||||
> 状态：**规划中（公式已对齐，治理计数器未落地）**。
|
||||
>
|
||||
> 依据：**Agent 蜂群指标量化与标准 v2.0 §4.1, §5.1**（公式未变）。配套：[`security-boundary`](../integration/security-boundary.md)、[`audit-trace`](../integration/audit-trace-schema.md)、[`swarm-metrics-schema.md`](./swarm-metrics-schema.md)。
|
||||
|
||||
## 1. 公式
|
||||
|
||||
```
|
||||
G_gov = CompliantActions / SensitiveActions × 100 # 执行层（奖励 R 中 w₆=0.10）
|
||||
S_governance = CompliantOperations / TotalOperations × 100 # 蜂群层（S_swarm 中 0.10）
|
||||
```
|
||||
|
||||
`Gov`（综合治理能力，进入 `Benchmark_Agent`）由上述构成。
|
||||
|
||||
## 2. 何为「需治理 / 合规」操作
|
||||
|
||||
需纳入治理计数的操作类别（对齐 security-boundary）：
|
||||
- **审批门**：高危操作是否走客户端审批（`approval.requested` → 审批回执）。
|
||||
- **权限**：资源访问是否在 `allowed_actions`/`constraints` 范围内。
|
||||
- **密钥**：是否仅用 `secret_ref`、无明文泄露。
|
||||
- **MCP / 工具边界**：工具调用是否在允许集合内（`sk_tool.*`）。
|
||||
- **敏感操作**：生产部署、DB 写、外部 API 等是否满足策略与 TTL。
|
||||
- **审计**：是否产生可追溯记录（见 audit-trace）。
|
||||
|
||||
「合规」= 上述操作满足策略；`S_governance` = 合规操作数 / 总（敏感/受治理）操作数。
|
||||
|
||||
## 3. 本仓可采集状态
|
||||
|
||||
| 治理维度 | 现状 |
|
||||
|---|---|
|
||||
| 审批门（waiting_approval + approvals 回执） | 🟡 状态机已实现，但未计数「合规/总」 |
|
||||
| 密钥（`azkv://` 强校验、明文拒绝、脱敏） | 🟡 已强制，但未作为治理计数项 |
|
||||
| 权限（allowed_actions/constraints 运行时强制） | 🔴 未运行时强制（见 security-boundary §3） |
|
||||
| MCP / 工具边界 | 🔴 未实现（无工具权限引擎） |
|
||||
| 敏感操作策略 + TTL 校验 | 🔴 未实现（审批逐项校验待加强） |
|
||||
| 审计可追溯 | 🟡 事件流可追溯，未量化为治理分 |
|
||||
|
||||
> 结论：治理当前仍**偏 README/机制级**，缺少 `CompliantOperations / TotalOperations` 的统一计数与导出 → `S_governance` / `Gov` 暂不可量化输出。
|
||||
|
||||
## 4. 实现状态
|
||||
|
||||
| 组件 | 状态 |
|
||||
|---|---|
|
||||
| 治理操作计数器（governed/compliant/total） | 🔴 未实现 |
|
||||
| 治理评分导出（S_governance / Gov） | 🔴 未实现 |
|
||||
| 与 audit 事件的绑定 | 🔴 未实现 |
|
||||
|
||||
## 5. 待对齐
|
||||
|
||||
- 「敏感 / 受治理操作」的精确清单与判定（Security/Governance Team）。
|
||||
- 「合规」判定规则（策略命中、审批命中、范围内、TTL 内）。
|
||||
- 计数与导出口径（按 `swarm_id` 聚合，进入 `SwarmMetrics.governance`）。
|
||||
@@ -0,0 +1,88 @@
|
||||
# Swarm Benchmark Protocol
|
||||
|
||||
> 状态：**规划中（协议已成文，采集器/基线/回放未落地）**。
|
||||
>
|
||||
> 依据：**Agent 蜂群指标量化与标准 v2.0**。本目录其余文档对各分量展开：[`swarm-metrics-schema.md`](./swarm-metrics-schema.md)、[`emergence-evaluation.md`](./emergence-evaluation.md)、[`baseline-comparison.md`](./baseline-comparison.md)、[`cost-normalized-gain.md`](./cost-normalized-gain.md)、[`governance-score.md`](./governance-score.md)、[`telemetry-architecture.md`](./telemetry-architecture.md)。
|
||||
|
||||
## 1. 目标
|
||||
|
||||
统一蜂群评测体系：支持不同 Agent Framework 横向比较、持续优化与回归测试、Benchmark 排行榜。**最终验收不是「能运行」或「能协作」，而是通过本协议自证**（尤其涌现增益 `G_E` 与成本归一化增益 `G_E,c`）。
|
||||
|
||||
## 2. 综合评分 Benchmark_Agent
|
||||
|
||||
```
|
||||
Benchmark_Agent = λ1·S_swarm + λ2·G_E + λ3·R + λ4·O + λ5·Gov
|
||||
```
|
||||
|
||||
| 分量 | 含义 | 展开文档 |
|
||||
|---|---|---|
|
||||
| `S_swarm` | 蜂群能力 | swarm-metrics-schema §蜂群层 |
|
||||
| `G_E` | 涌现增益（vs baseline） | emergence-evaluation |
|
||||
| `R` | 奖励函数 | swarm-metrics-schema §执行层 |
|
||||
| `O` | 可观测性 | telemetry-architecture |
|
||||
| `Gov` | 治理能力 | governance-score |
|
||||
|
||||
**λ 权重（v2.0，强制 `Σλᵢ = 1.0`）**：`λ1=0.30`（S_swarm）· `λ2=0.20`（G_E）· `λ3=0.25`（R）· `λ4=0.15`（O）· `λ5=0.10`（Gov）。`benchmark/metrics.py:benchmark_agent` 会校验 `Σλ=1.0`。
|
||||
|
||||
`S_swarm` 子权重（v2.0，未变）：
|
||||
```
|
||||
S_swarm = 0.25·S_completion + 0.20·S_gain + 0.15·S_collaboration
|
||||
+ 0.10·S_communication + 0.10·S_cost + 0.10·S_robustness + 0.10·S_governance
|
||||
```
|
||||
|
||||
## 3. Benchmark Protocol（基线与实验组）
|
||||
|
||||
| 组 | 系统 |
|
||||
|---|---|
|
||||
| Baseline A | Single Agent |
|
||||
| Baseline B | Chain Agent |
|
||||
| Baseline C | Sub-Agent |
|
||||
| Baseline D | Strong Agent |
|
||||
| Experimental | **Swarm Agent**（本仓） |
|
||||
|
||||
所有组**统一采集**：`Completion`、`Quality`、`Cost`、`Time`、`Robustness`。
|
||||
蜂群成立的硬条件：`Swarm > Single`、`Swarm > Chain`、`Swarm > Sub-Agent`、`Swarm > Strong`（见 emergence-evaluation），且经成本归一化后增益非「堆 Agent/Token」虚高（见 cost-normalized-gain）。
|
||||
|
||||
## 3.1 超参数向量 Θ（v2.0 §7.2）
|
||||
|
||||
```
|
||||
Θ = [α, β, ρ, N_agent, ε] Θ* = argmax(Benchmark_Agent)
|
||||
```
|
||||
推荐初值：`α=1.0`、`β=2.0`、`ρ=0.10`（信息素挥发率）、`N_agent=5`、`ε=0.10`（探索率）。见 `benchmark/metrics.py:THETA_DEFAULTS`。
|
||||
|
||||
## 4. 场景（Leaderboard 维度）
|
||||
|
||||
Coding · Refactoring · Architecture · DevOps · Bug Fix。
|
||||
|
||||
## 5. Leaderboard 展示字段
|
||||
|
||||
`Benchmark_Agent`、`S_swarm`、`G_E`、`G_E,c`、`Reward`、`Cost Efficiency`。
|
||||
|
||||
## 6. 实现状态（本仓）
|
||||
|
||||
| 组件 | 位置（规划） | 状态 |
|
||||
|---|---|---|
|
||||
| 指标定义 / schema | `docs/benchmark/*`、`SwarmMetrics` | ✅ 已成文 |
|
||||
| 采集器 `SwarmMetricsCollector` | `benchmark/collectors/` | 🟡 部分落地（`SwarmRunMetricsCollector` 真实计算 completion/collaboration/cost/robustness；其余 NaN+coverage） |
|
||||
| 基线运行器 A–D | `benchmark/baselines/` | 🔴 未落地 |
|
||||
| 回放 replay | `benchmark/replay/` | 🔴 未落地 |
|
||||
| 排行榜 leaderboard | `benchmark/leaderboard/` | 🔴 未落地 |
|
||||
| 数据采集架构 | telemetry-architecture | 🔴 未接入（OTel/Prometheus/ClickHouse/ES） |
|
||||
|
||||
> 在采集器与基线运行器落地前，**不得宣称本仓已具备 Agent Swarm 量化自证能力**。
|
||||
|
||||
## 7. 验收标准（对齐工单）
|
||||
|
||||
- [ ] 能输出完整 `Benchmark_Agent`。
|
||||
- [ ] 能量化 `S_swarm`、`G_E`、`G_E,c`、`Gov`。
|
||||
- [ ] 能回放一次 benchmark execution。
|
||||
- [ ] 能证明 `Swarm > Single / Chain / Sub-Agent`。
|
||||
- [ ] 能证明增益不依赖无限 token/GPU 堆叠（成本归一化）。
|
||||
|
||||
## 8. 待对齐
|
||||
|
||||
- ~~`λ1…λ5` 数值权重~~ ✅ v2.0 已给定（Σλ=1.0）。
|
||||
- `Θ*` 调优口径与搜索方法。
|
||||
- 各基线（Single/Chain/Sub-Agent/Strong）的标准实现与数据集。
|
||||
- `Quality` 的统一口径（TestPassRate / CodeReviewScore / UserAcceptance 来源）。
|
||||
- 数据源与平台（telemetry-architecture）。
|
||||
@@ -0,0 +1,103 @@
|
||||
# Swarm Metrics Schema（v2.0）
|
||||
|
||||
> 状态：**公式已对齐标准 v2.0；采集器部分落地**。
|
||||
>
|
||||
> 依据：**Agent 蜂群指标量化与标准 v2.0**（§3–§8）。配套：[`swarm-benchmark-protocol.md`](./swarm-benchmark-protocol.md)、[`metric-coverage-gaps.md`](./metric-coverage-gaps.md)、[`telemetry-architecture.md`](./telemetry-architecture.md)。
|
||||
>
|
||||
> 实现：公式见 `benchmark/metrics.py`（纯函数，含 v2.0 推荐权重常量 `TAU_WEIGHTS`/`ETA_WEIGHTS`/`REWARD_WEIGHTS`/`SWARM_WEIGHTS`/`LAMBDA_WEIGHTS`/`THETA_DEFAULTS`）；真实采集见 `benchmark/collectors/run_collector.py`（`SwarmRunMetricsCollector`）。
|
||||
|
||||
## 0. 采集接口（标准 §8.1）
|
||||
|
||||
```python
|
||||
@dataclass
|
||||
class SwarmMetrics:
|
||||
tau: float # 信息素得分
|
||||
eta: float # 启发式得分
|
||||
p_decision: float # 决策概率
|
||||
reward: float # 执行奖励
|
||||
s_completion: float
|
||||
s_gain: float
|
||||
s_collaboration: float
|
||||
s_communication: float
|
||||
s_cost: float
|
||||
s_robustness: float
|
||||
s_governance: float
|
||||
s_swarm: float # 蜂群总分
|
||||
g_e: float # 涌现增益
|
||||
g_e_cost: float # 成本归一化增益
|
||||
benchmark: float # 综合评分
|
||||
```
|
||||
`SwarmRunMetricsCollector` 真实计算 `s_completion`/`s_collaboration`/`s_cost`/`s_robustness`，其余返回 `NaN` 并在 `coverage` 标记 `False`（不伪造分值）。
|
||||
|
||||
## 1. 决策层（标准 §3）
|
||||
|
||||
```
|
||||
τ(s,a,r) = w₁·Success + w₂·Quality + w₃·Acceptance − w₄·Cost − w₅·Time − w₆·Risk − w₇·Rollback
|
||||
η(s,a,r) = γ₁·Match + γ₂·Urgency + γ₃·Dependency + γ₄·Resource + γ₅·Confidence − γ₆·Risk − γ₇·BudgetPressure
|
||||
P(s,a,r) = τ^α·η^β / Σ τ_i^α·η_i^β
|
||||
P_decision = τ^α · η^β · 100 # v2.0 §3.3（变更）
|
||||
```
|
||||
|
||||
**τ 权重（v2.0）**：Success 0.25 · Quality 0.20 · **Acceptance 0.20（提升为一级因子）** · Cost 0.10 · Time 0.10 · Risk 0.08 · Rollback 0.07。
|
||||
**η 权重（v2.0）**：Match 0.25 · Urgency 0.15 · Dependency 0.15 · Resource 0.15 · **Confidence 0.10（新增）** · Risk 0.10 · BudgetPressure 0.10。
|
||||
|
||||
| 因子 | 数据来源（标准） | 本仓可采集 |
|
||||
|---|---|---|
|
||||
| τ.Success | Task 完成记录 | ✅ |
|
||||
| τ.Quality | 测试 / CI/CD | 🔴 无 CI 接入 |
|
||||
| τ.Acceptance | 人工验收日志 | 🟡 评审 accepted |
|
||||
| τ.Cost / Time | 资源监控 / Runtime 日志 | 🟡 usage / 时间戳 |
|
||||
| τ.Risk / Rollback | 安全审计 / Git·部署 | 🔴 |
|
||||
| η.Match / Dependency | 能力画像 / DAG | ✅ |
|
||||
| η.Confidence | Agent 自评接口 | 🔴 未实现 |
|
||||
| η.Urgency / Resource / Risk / BudgetPressure | 优先级 / 授权 / 风险引擎 / 预算 | 🔴 / 🟡 / 🔴 / 🟡 |
|
||||
| P_decision | 上述综合 | 🔴 无 τ/η 引擎 |
|
||||
|
||||
## 2. 执行层（标准 §4）
|
||||
|
||||
```
|
||||
R = w₁·S_task + w₂·Q_quality + w₃·V_speed + w₄·E_cost + w₅·R_robust + w₆·G_gov − w₇·P_risk − w₈·P_rework
|
||||
S_task = CompletedTasks/TotalTasks×100 Q_quality = 0.4·TestPass + 0.3·CodeReview + 0.3·UserAcceptance
|
||||
V_speed = 100·TargetTime/ActualTime E_cost = 100·ExpectedCost/ActualCost
|
||||
R_robust = RecoveredFailures/TotalFailures×100 G_gov = CompliantActions/SensitiveActions×100
|
||||
P_rework = ReworkCount/TotalTasks×100
|
||||
```
|
||||
|
||||
**R 权重（v2.0，已给定）**：S_task **0.20** · Q_quality **0.20（与完成度对齐）** · V_speed 0.12 · E_cost 0.13 · R_robust 0.13 · G_gov 0.10 · P_risk 0.07 · P_rework 0.05。
|
||||
|
||||
| 指标 | 本仓可采集 |
|
||||
|---|---|
|
||||
| `S_task` | ✅ |
|
||||
| `Q_quality` | 🔴 需 TestPass/CodeReview/UserAcceptance（无 CI/评分） |
|
||||
| `V_speed` / `E_cost` | 🟡 Actual 有；Target/Expected 需基线 |
|
||||
| `R_robust` | 🟡 重试/恢复未计数 |
|
||||
| `G_gov` / `P_risk` / `P_rework` | 🔴 无敏感操作/风险/返工计数 |
|
||||
|
||||
> v2.0 已给定全部 `w*`/`γ*` 权重，但多数**输入**仍未采集 → `τ`/`η`/`reward`/`p_decision` 暂不可算。
|
||||
|
||||
## 3. 蜂群层（标准 §5，权重未变）
|
||||
|
||||
```
|
||||
S_swarm = 0.25·S_completion + 0.20·S_gain + 0.15·S_collaboration
|
||||
+ 0.10·S_communication + 0.10·S_cost + 0.10·S_robustness + 0.10·S_governance
|
||||
S_completion = CompletedTasks/TotalTasks×100
|
||||
S_collaboration= 0.5·HandoffSuccessRate + 0.3·DependencyResolutionRate + 0.2·WorkloadBalanceScore
|
||||
S_communication= SuccessfulMessages/TotalMessages×100
|
||||
S_cost = 100·Budget/ActualUsage
|
||||
S_robustness = RecoveredFailures/TotalFailures×100
|
||||
S_governance = CompliantOperations/TotalOperations×100
|
||||
```
|
||||
|
||||
| 指标 | 状态 |
|
||||
|---|---|
|
||||
| `s_completion` / `s_collaboration` / `s_cost` / `s_robustness` | ✅ 真实可算（采集器） |
|
||||
| `s_gain` | 🔴 需基线（见 emergence-evaluation） |
|
||||
| `s_communication` | 🔴 未计数（无消息 telemetry） |
|
||||
| `s_governance` | 🟡 仅审批可派生 |
|
||||
| `s_swarm` | 🔴 含 gain/communication NaN → 暂为 NaN |
|
||||
|
||||
## 4. 说明与待对齐
|
||||
|
||||
- **成本口径统一（v2.1）**：`S_cost`（§3）、`E_cost`（§2）、`CostEfficiency`（见 cost-normalized-gain）为**同一量** `100×Budget/ActualCost`。
|
||||
- `α/β/ρ/N_agent/ε` 取值：标准 §7.2 给推荐初值（1.0 / 2.0 / 0.10 / 5 / 0.10），调优口径待定。
|
||||
- `Q_quality` 三项来源、`SuccessfulMessages/TotalMessages`、`Recovered/Total`、`Compliant/Total`、`Rework/Total` 的精确计数定义。
|
||||
@@ -0,0 +1,53 @@
|
||||
# Telemetry Architecture（数据采集架构）
|
||||
|
||||
> 状态：**规划中（部分信号已发出，统一采集管道未接入）**。
|
||||
>
|
||||
> 依据：**Agent 蜂群指标量化与标准 v2.0 §8.2**。配套：[`swarm-metrics-schema.md`](./swarm-metrics-schema.md)、[`event-schema`](../integration/event-schema.md)、[`audit-trace`](../integration/audit-trace-schema.md)。`O`（可观测性）分量来源于本文。
|
||||
|
||||
## 1. 数据来源矩阵（标准 §8.2）
|
||||
|
||||
| 指标类型 | 数据来源 | 推荐平台 | 本仓现状 |
|
||||
|---|---|---|---|
|
||||
| 任务完成 | Task Logs、Handoff Logs | Elasticsearch | ✅ 任务状态 + 事件流（按 `swarm_id`）；未汇入 ES |
|
||||
| 成本 / Token | Runtime Metrics | Prometheus | 🟡 usage 已采集；Prometheus 指标可抓取 |
|
||||
| 质量 | CI/CD Results、Code Review | 各 CI 平台 | 🔴 未接入 |
|
||||
| 治理合规 | Audit Logs | OpenTelemetry | 🟡 事件流可作审计源，未独立留存 |
|
||||
| 通信 | WebSocket Logs | ClickHouse | 🟡 连接事件有日志；消息成功率未计数 |
|
||||
| 基础设施 | Infrastructure Metrics | Prometheus | 🔴 未接入（cpu/memory/Pod） |
|
||||
|
||||
## 3. 本仓已发出的信号
|
||||
|
||||
| 信号 | 实现 | 去向 |
|
||||
|---|---|---|
|
||||
| Prometheus 指标 | ✅ 编排器 `GET /metrics`；Agent `METRICS_PORT` | 可被 Prometheus 抓取（`k8s/prometheus-config.yaml`） |
|
||||
| OpenTelemetry tracing | 🟡 `orchestrator/tracing.py`（`OTEL_EXPORTER_OTLP_ENDPOINT`） | 需配置 collector |
|
||||
| 事件流（lifecycle/DAG/handoff/usage） | ✅ 按 `swarm_id` 持久化 + 回调 | Redis + HM 回调；可导出 ClickHouse/ES（未接） |
|
||||
| 任务/用量/预算 | ✅ `/tasks`、`/metrics`、`budget.alert` | REST + 事件 |
|
||||
|
||||
## 4. 目标采集管道（草案）
|
||||
|
||||
```
|
||||
Agent / Orchestrator
|
||||
├─ Prometheus ← /metrics（指标：任务、时长、并发、被拒/重复、reconnect…）
|
||||
├─ OpenTelemetry → OTLP collector → trace 存储（task/agent/handoff span）
|
||||
└─ 事件流（event-schema） → 导出器 → ClickHouse / Elasticsearch（用于 benchmark 聚合与回放）
|
||||
↑
|
||||
SwarmMetricsCollector 读取并产出 SwarmMetrics
|
||||
```
|
||||
|
||||
## 5. 缺口
|
||||
|
||||
| 项 | 状态 |
|
||||
|---|---|
|
||||
| Prometheus 指标暴露 | ✅ 已有（指标项可再补：消息成功率、恢复率、治理计数） |
|
||||
| OTel trace 实际导出 | 🟡 模块在，collector/span 覆盖待完善 |
|
||||
| 事件 → ClickHouse/ES 导出器 | 🔴 未实现 |
|
||||
| CI/CD、Git、基础设施指标接入 | 🔴 未实现 |
|
||||
| 统一 `SwarmMetricsCollector` 落地 | 🔴 未实现（见 swarm-metrics-schema §0） |
|
||||
| 刷新可恢复 / 断线回补（前端） | 🟡 `/logs?cursor=`（见 frontend-event-api） |
|
||||
|
||||
## 6. 待对齐
|
||||
|
||||
- 选定后端（ClickHouse / ES）与 schema、留存期（与 Audit Team）。
|
||||
- 指标命名规范与标签（`swarm_id`/`agent_role`/`scenario`）。
|
||||
- 导出器归属（本仓 exporter vs 平台统一采集）。
|
||||
@@ -0,0 +1,65 @@
|
||||
# Agent 能力与调度 Schema（Capability / Registry / Routing）
|
||||
|
||||
> 状态：**DRAFT / 待对齐 Infra & Scheduling Team**。
|
||||
>
|
||||
> 依据：`heicode-mananger/docs/heicode.md §五`、`docs/integration/heicode-am-contract.md §2`。配套：[`runtime-contract.md`](./runtime-contract.md)、[`security-boundary.md`](./security-boundary.md)。
|
||||
|
||||
## 1. 现状
|
||||
|
||||
- 当前能力调度为 **swarm 私有实现**：Agent 以 `AGENT_CAPABILITIES` 注册到编排器（Redis 注册表），编排器按「`required_capabilities ⊆ agent.capabilities` + 剩余容量」派发。
|
||||
- **尚未接入统一 agent registry / capability center / quota / tenant / region·GPU 调度 / provider routing / model policy**。这些归 Infra/Scheduling，未在本仓实现（避免 Runtime Scheduling 双实现）。
|
||||
|
||||
## 2. 已实现的能力模型（本仓）
|
||||
|
||||
### 2.1 Agent 注册（`AgentMetadata`）
|
||||
```jsonc
|
||||
{ "agent_id": "...", "status": "idle|busy|handoff-pending|failed",
|
||||
"capabilities": ["python","testing", ...], "current_task_id": "...|null",
|
||||
"last_heartbeat": 0.0 }
|
||||
```
|
||||
另：register/heartbeat 上报 `available_slots`（剩余容量），编排器据此避免向满载 Agent 派发。
|
||||
|
||||
### 2.2 任务侧能力需求
|
||||
任务携带 `agent_role` 与 `required_capabilities`；派发用 `can_agent_run_task`（`required ⊆ capabilities`，空需求视为通用可执行）。
|
||||
|
||||
### 2.3 能力别名（`handoff_logic.CAPABILITY_ALIASES`）
|
||||
`code_generation` / `python` / `general` 等扩展为本地等价能力集合，用于移交决策与匹配宽松化。
|
||||
|
||||
### 2.4 角色（role）
|
||||
- Manager/产品口径角色（`heicode.md §五`）：`product`、`frontend`、`backend`、`reviewer`、`ops`。
|
||||
- 本仓规划回退默认角色：`implementation`、`testing`、`documentation`（`agent_role` 字段，注入 `specialist_role`）。
|
||||
|
||||
## 3. 角色 ↔ 资源 ↔ env（对齐 AM）
|
||||
|
||||
资源授权（Resource Grant，Manager 侧）按角色绑定资源，HM 解析为固定 env 名注入 Agent（`heicode-am-contract §2`）：
|
||||
|
||||
| 资源类型 | provider | 非密 env | 密钥 env（经 secret_ref） |
|
||||
|---|---|---|---|
|
||||
| git | github/gitea/gitlab | `GIT_PROVIDER` `GIT_REPO_URL` `GIT_DEFAULT_BRANCH` | `GIT_TOKEN` |
|
||||
| database | mysql / postgres | `MYSQL_*` / `POSTGRES_*`（host/port/db/user） | `MYSQL_PASSWORD` / `POSTGRES_PASSWORD` |
|
||||
| storage | azure blob | `AZURE_BLOB_ACCOUNT_NAME` `AZURE_BLOB_CONTAINER` | `AZURE_BLOB_ACCOUNT_KEY` |
|
||||
|
||||
角色/指令经 `AGENT_ROLE_NAME` + `AGENT_INSTRUCTION_TEXT` 注入（AM 约定）。
|
||||
|
||||
## 4. 目标统一调度 Schema（待 Infra 定义）
|
||||
|
||||
| 维度 | 字段（建议） | 状态 |
|
||||
|---|---|---|
|
||||
| Registry | `agent_id`、`capabilities`、`role`、`framework`、`version`、`health` | 🟡 本仓有私有注册表，未接统一中心 |
|
||||
| Capability center | 能力字典、能力版本、能力↔工具映射 | 🔴 未接入 |
|
||||
| Quota | `max_concurrent`、`token_quota`、`cost_quota`（按 user/channelId） | 🟡 仅 Agent 侧 `MAX_CONCURRENT_TASKS`/容量 |
|
||||
| Tenant isolation | —（按标准用 user/channelId，不引入 tenant） | ⛔ 不在本仓 |
|
||||
| Region / GPU scheduling | `region`、`gpu_class`、`node_pool` | 🔴 未接入（AKS/Infra） |
|
||||
| Provider routing | `provider`、`model_policy`、`fallback` | 🔴 未接入（NewAPI/网关侧） |
|
||||
| Model policy | `model_ref`、`profile`、`budget` | 🟡 Agent 用 `OPENAI_MODEL`；统一 policy 未接 |
|
||||
|
||||
## 5. 缺口
|
||||
|
||||
- 🔴 统一 agent registry / capability center / provider routing / region·GPU 调度：未实现，归 Infra/Scheduling。
|
||||
- 🟡 配额仅 Agent 并发/容量级；无 user/channelId 维度配额中心。
|
||||
- ⛔ tenant 维度：按标准不引入。
|
||||
- ⚠️ 风险：若 Infra 落地统一调度，需收敛本仓私有派发，避免 Runtime Scheduling 双实现。
|
||||
|
||||
## 6. 待对齐对象
|
||||
|
||||
Infra & Scheduling Team：统一 registry/capability center 字段、quota（按 user/channelId）、region·GPU·provider routing、model policy，以及本仓私有派发与统一调度的收敛路径。
|
||||
@@ -0,0 +1,75 @@
|
||||
# 审计与链路追踪 Schema（Audit / Lineage / Trace）
|
||||
|
||||
> 状态：**DRAFT / 待对齐 Audit & Compliance Team**。
|
||||
>
|
||||
> 依据：`heicode-mananger/docs/heicode.md §五/§九`、`docs/heicode-runtime-auth-newapi-secret-design.md §三`。配套：[`event-schema.md`](./event-schema.md)、[`security-boundary.md`](./security-boundary.md)。
|
||||
|
||||
## 1. 原则
|
||||
|
||||
- 审计要能回答：**谁、在什么时候、让哪个子 Agent、使用了什么资源、做了什么、结果如何**。
|
||||
- AGENT.md / resource context 面向模型理解；**permission manifest 面向系统强制执行**；审计以结构化记录为准，不以 Markdown 为准。
|
||||
- 审计记录**不得含明文密钥**：只记 `secret_ref`（`azkv://`）、审批 `approval_id`、范围与 TTL。
|
||||
|
||||
## 2. 追踪域（Lineage）
|
||||
|
||||
### 2.1 部署 / 运行链路
|
||||
`manager_deployment_id ↔ deployment_id ↔ swarm_id(workflow_id) ↔ correlation_id(trace_id)`，四者贯穿一次交付，事件按 `swarm_id` 持久化。
|
||||
|
||||
### 2.2 任务链路（已实现）
|
||||
每个任务携带：`task_id`、`parent_task_id`、`root_task_id`、`child_task_ids`、`depends_on`、`source`（`runtime_bridge` / `planner` / `dynamic_handoff`）、`agent_role`、`assigned_agent_id`、`retry_count`、时间戳。可由此重建任务 DAG 与重做/移交谱系。
|
||||
|
||||
### 2.3 执行链路（已实现，基于事件流）
|
||||
按 `swarm_id` 顺序持久化的事件即执行轨迹：`task.created/claimed/running/heartbeat/blocked/retried/failed/completed`、`handoff.requested/completed`、`artifact.created`、`timeline.updated`、`deployment.status_changed`，每条含 `event_id`、`occurred_at`、`agent_instance_id`、`correlation_id`。
|
||||
|
||||
### 2.4 审批链路（已实现）
|
||||
`run.approvals[approval_id]`：`operation`、`risk_level`、`decision`（approved/rejected）、`credential_ref`、`lease_id`、`reason`、决策时间。客户端审批，运行时仅记录与校验。
|
||||
|
||||
### 2.5 资源 / 凭证访问链路
|
||||
`secret_refs`（`azkv://`，仅引用）、resource grant 的 `allowed_actions`/`constraints`/`status`/`created_by`/`revoked_by`/`created_at`/`revoked_at`（字段定义在 Manager 侧 `heicode.md §五`）。本仓只透传与引用，不落明文。
|
||||
|
||||
### 2.6 回调投递链路（已实现）
|
||||
`run.metadata.callback_attempts`：每次回调的 `event_id`、`event_type`、`url`、`status`、`attempted_at`，用于回调可靠性审计（`/diagnostics` 暴露）。
|
||||
|
||||
## 3. 已实现（本仓）
|
||||
|
||||
| 能力 | 实现 |
|
||||
|---|---|
|
||||
| 事件流持久化 + 分页查询 | `_store_event`（按 `swarm_id` 追加）、`list_events`（`cursor`/`limit`）；`/logs`、`/events` 暴露 |
|
||||
| 任务谱系字段 | `parent/root/child_task_ids`、`depends_on`、`source`、`retry_count` |
|
||||
| 审批记录 | `run.approvals`、`record_approval_decision` |
|
||||
| 回调投递审计 | `callback_attempts`、`/diagnostics` |
|
||||
| 用量归因 | `usage` + `X-Agent-*` 归因头（见 usage-billing） |
|
||||
| 脱敏 | `_redact_sensitive`（保留 `secret_ref`，脱敏明文密钥） |
|
||||
|
||||
## 4. 缺口
|
||||
|
||||
| 追踪项 | 状态 |
|
||||
|---|---|
|
||||
| Task lineage / execution trace | ✅ 已实现（事件流 + 任务字段） |
|
||||
| Approval trace | ✅ 已实现 |
|
||||
| Prompt trace（每次模型输入提示留痕） | 🔴 未实现（仅记 `model_id`/用量，不留 prompt 原文） |
|
||||
| Model trace（请求/响应、参数、provider） | 🟡 部分（`model_id`/tokens；无完整请求响应留痕） |
|
||||
| Tool trace（工具调用谱系） | 🔴 未实现（无 SK/MCP 工具计量，`sk_tool.*` 仅在 schema 预留） |
|
||||
| 统一审计查询字段 / 留存策略 | 🟡 事件可查；标准查询字段与留存期未定义 |
|
||||
| 交付回放（replay） | 🟡 事件流可顺序回放生命周期；无独立 replay 接口/快照格式 |
|
||||
|
||||
## 5. 建议审计记录 Schema（草案，待 Audit Team 冻结）
|
||||
|
||||
```jsonc
|
||||
{
|
||||
"audit_id": "aud_...",
|
||||
"occurred_at": "2026-06-08T...Z",
|
||||
"actor": { "user_id": "...", "channel_id": "..." }, // 谁（按 user/channelId，非 tenant）
|
||||
"subject": { "agent_role": "...", "agent_instance_id": "...", "task_id": "..." }, // 哪个子 Agent / 任务
|
||||
"action": "task.completed | approval.decided | resource.accessed | ...",
|
||||
"resource": { "type": "git|database|storage|model", "ref": "non-secret-id", "secret_ref": "azkv://..." },
|
||||
"approval_id": "approval_... | null",
|
||||
"lineage": { "manager_deployment_id": "...", "deployment_id": "...", "swarm_id": "...", "correlation_id": "..." },
|
||||
"result": "success | failed | blocked",
|
||||
"redacted": true
|
||||
}
|
||||
```
|
||||
|
||||
## 6. 待对齐对象
|
||||
|
||||
Audit & Compliance Team：prompt/model/tool lineage 是否强制留痕及留存期、统一审计查询字段、交付回放快照格式、日志留存与合规要求；与 `telemetry-architecture`（benchmark 目录）数据源对齐。
|
||||
@@ -0,0 +1,98 @@
|
||||
# Swarm 运行时事件 Schema（回调 / 事件流）
|
||||
|
||||
> 状态：**已对齐 HM 实现**（依据 `heicode-mananger/heicode/controller/agent_callback.go`，handler `POST /api/agent/callbacks/runtime-events`）。本文是 Swarm → HM 回调与 `GET …/{id}/events` 事件流的统一 schema。
|
||||
>
|
||||
> 配套：生命周期与签名见 [`runtime-contract.md`](./runtime-contract.md)。
|
||||
|
||||
## 1. 传输与鉴权
|
||||
|
||||
- Swarm → HM：`POST {callback.url}`（create 时下发）。
|
||||
- 鉴权二选一：服务令牌 `X-Agent-Service-Token`（或 `Authorization: Bearer`），或 HMAC 签名。
|
||||
- **HMAC 签名（已实现，与 HM 一致）**：
|
||||
- 头：`X-Agent-Timestamp`（Unix 毫秒）、`X-Agent-Signature`、`X-Agent-Event-Id`、`X-Correlation-ID`（均含 `X-Agnet-` 兼容别名）。
|
||||
- 规范串：`canonical = f"{timestamp}.{event_id}.{raw_body}"`。
|
||||
- 签名：`X-Agent-Signature = "sha256=" + hex(HMAC_SHA256(secret, canonical))`。
|
||||
- secret：`AGENT_CALLBACK_SIGNING_SECRET`（兼容 `AGNET_…`，支持 `…_REF` 的 `azkv://` 解析）。
|
||||
- 时间容差：HM 默认 300s（`AGENT_CALLBACK_SIGNATURE_TOLERANCE_SECONDS`）。
|
||||
- **幂等去重**：HM 依次按 `X-Agent-Event-Id` 头 → body `event_id` → `idempotency_key`。重复返回 `{inserted:false, idempotent:true, deduplicated:true}`。
|
||||
|
||||
## 2. 事件 Envelope
|
||||
|
||||
```jsonc
|
||||
{
|
||||
"event_id": "evt-...", // 必填；去重主键
|
||||
"idempotency_key": "...", // 可选；缺省回退 event_id
|
||||
"event_type": "task.completed", // 必填；取值见 §4
|
||||
"deployment_id": "runtime-dep-...",// 运行时部署 ID
|
||||
"swarm_id": "swarm-...", // 工作流 ID（事件按此持久化）
|
||||
"agent_instance_id": "...", // Agent 实例/角色实例 ID（可选）
|
||||
"task_id": "...", // 任务相关事件填
|
||||
"occurred_at": "2026-06-08T...Z", // 事件时间
|
||||
"correlation_id": "corr-...", // 追踪 ID（X-Correlation-ID）
|
||||
"source": "heicode-swarm-runtime", // 事件来源标识
|
||||
"metadata": { }, // 自定义元数据（不得含明文密钥）
|
||||
"payload": { }, // 事件专属字段（见 §4）
|
||||
"artifact": { } // 可选；见 §3
|
||||
}
|
||||
```
|
||||
|
||||
## 3. Artifact 子对象（`artifact.created` 及完成事件可带）
|
||||
|
||||
```jsonc
|
||||
{
|
||||
"artifact_id": "art_...",
|
||||
"artifact_type": "code_patch | document | deployment_manifest",
|
||||
"title": "...",
|
||||
"summary": "...",
|
||||
"uri": "git://repo#branch | runtime://...",
|
||||
"checksum": "<commit_sha>",
|
||||
"metadata": { "redacted": true, "agent_role": "...", "files_modified": [] }
|
||||
}
|
||||
```
|
||||
|
||||
## 4. 事件类型注册表（与 HM 一致）
|
||||
|
||||
| event_type | payload 必填字段（HM 校验） | 分类 | Swarm 是否发出 |
|
||||
|---|---|---|---|
|
||||
| `deployment.status_changed` | `status` | deployment | ✅ |
|
||||
| `phase.changed` | `stage` 或 `checkpoint` | ordinary_sub | ❌（暂未用） |
|
||||
| `agent.started` | `agent_role` | ordinary_sub | ❌ |
|
||||
| `agent.completed` | `agent_role` | ordinary_sub | ❌ |
|
||||
| `agent.crashed` | `agent_role`, `reason` | ordinary_sub | ❌ |
|
||||
| `task.created` | `task_id`, `title` | swarm_task_flow | ✅ |
|
||||
| `task.claimed` | `task_id`, `agent_role` | swarm_task_flow | ✅ |
|
||||
| `task.running` | `task_id`, `agent_role` | swarm_task_flow | ✅ |
|
||||
| `task.heartbeat` | `task_id`, `agent_role` | swarm_task_flow | ✅ |
|
||||
| `task.blocked` | `task_id`, `reason` | swarm_task_flow | ✅ |
|
||||
| `task.retried` | `task_id`, `attempt` | swarm_task_flow | ✅ |
|
||||
| `task.released` | `task_id`, `agent_role` | swarm_task_flow | ⚠️ 未发出（见 §6） |
|
||||
| `task.failed` | `task_id`, `reason` | swarm_task_flow | ✅ |
|
||||
| `task.completed` | `task_id` | swarm_task_flow | ✅ |
|
||||
| `handoff.requested` | `task_id`, `from_role`, `to_role` | swarm_task_flow | ✅ |
|
||||
| `handoff.completed` | `task_id`, `from_role`, `to_role` | swarm_task_flow | ⚠️ 缺 `from_role`/`to_role`（见 §6） |
|
||||
| `approval.requested` | `approval_id`, `operation`, `risk_level` | approval | ✅ |
|
||||
| `artifact.created` | `artifact_id` | artifact | ✅ |
|
||||
| `timeline.updated` | `title` | timeline | ⚠️ 发出 `summary`，缺 `title`（见 §6） |
|
||||
| `sk_tool.called` | `tool_name`, `tool_invocation_id` | sk | ❌（无 SK 工具） |
|
||||
| `sk_tool.completed` | `tool_name`, `tool_invocation_id` | sk | ❌ |
|
||||
| `sk_tool.failed` | `tool_name`, `tool_invocation_id`, `reason` | sk | ❌ |
|
||||
| `budget.alert` | `threshold_pct` | budget | ⚠️ 发出 `threshold`，缺 `threshold_pct`（见 §6） |
|
||||
|
||||
> 说明：评审/重做循环复用 `task.retried` + `timeline.updated`（`Review cycle N`）表达，无单独 review 事件类型；如 HM 前端需要独立 review 事件，列为待对齐项。
|
||||
|
||||
## 5. HM 响应
|
||||
|
||||
```json
|
||||
{ "success": true, "event_id": "evt-...", "inserted": true, "idempotent": false, "deduplicated": false, "deployment_id": "dep_..." }
|
||||
```
|
||||
|
||||
## 6. 已知对齐缺口（需在 Swarm 代码修正）
|
||||
|
||||
下列为本仓 `orchestrator/` 当前发出字段与 HM 校验的差异，应在对应 emit 处修正后再宣称完全对齐：
|
||||
|
||||
1. **`timeline.updated`**：HM 要求 `title`；当前发 `summary`。修正：payload 增加 `title`（可复用 `summary`）。
|
||||
2. **`budget.alert`**：HM 要求 `threshold_pct`；当前发 `threshold`（如 `0.8`）。修正：增加 `threshold_pct`（百分比，如 `80`）。
|
||||
3. **`handoff.completed`**：HM 要求 `from_role`/`to_role`；当前缺。修正：在 `finalize_parent_after_child` / handoff 完成处补 `from_role`/`to_role`。
|
||||
4. **`task.released`**：HM 注册表含此事件；`task_queue.release_task` 当前静默。修正：释放任务时发 `task.released`（`task_id`, `agent_role`）。
|
||||
|
||||
> 这些是确定性的小修，建议配 contract test（按 HM 必填字段断言每类事件 payload）。
|
||||
@@ -0,0 +1,58 @@
|
||||
# 前端事件 API（Frontend Event API）
|
||||
|
||||
> 状态：**DRAFT / 待对齐 Frontend & Product Team**。
|
||||
>
|
||||
> 依据：`heicode-mananger/docs/integration/heicode-desktop-client-api.md`、`docs/integration/manager-side-contract-patches.md §6`。配套：[`event-schema.md`](./event-schema.md)、[`runtime-contract.md`](./runtime-contract.md)。
|
||||
|
||||
## 1. 现状（重要）
|
||||
|
||||
- 当前**桌面端是单 Agent 模型**：用户部署「模板 Agent」，客户端从 HM 拿 `subdomain`+`access_token` 后**直连 Agent**走 A2A：`POST {subdomain}/message/stream`（SSE）/ `message/send`、`GET /.well-known/agent.json`、`/health`。HM 不在回路。
|
||||
- **HM 尚未定义「蜂群图 / agent 拓扑 / review loop / retry 时间线」的前端事件 API**。蜂群前端契约**归 Swarm + Frontend/Product Team 共同定义**，目前不存在。
|
||||
- 因此本文档区分：①HM 既有的任务卡片事件通道；②Swarm 现已提供、前端可消费的 REST；③待定义的蜂群前端事件 API。
|
||||
|
||||
## 2. HM 既有前端事件通道（任务卡片，非蜂群图）
|
||||
|
||||
来自 `manager-side-contract-patches.md §6`：
|
||||
- `GET /api/user/tasks/{id}/events`（SSE）；HM 当前以**轮询兜底**（active 3s、其它状态 15s）。
|
||||
- 建议 SSE 事件名与载荷：
|
||||
- `status_changed` → 新状态
|
||||
- `followup_added` → followup 载荷
|
||||
- `card_updated` → 完整 `HeicodeTaskCard`（`{goal, scope, generated_artifacts, manager_actions:[{label, deeplink}]}`）
|
||||
- 这是**任务卡片**视角，不是蜂群 DAG/拓扑视角。
|
||||
|
||||
## 3. Swarm 现已提供的可消费 REST（已实现）
|
||||
|
||||
前端（经 HM 或直连受控读接口）可消费：
|
||||
|
||||
| 接口 | 用途 | 关键字段 |
|
||||
|---|---|---|
|
||||
| `GET …/{deployment_id}/workflow` | 工作流/阶段视图 | `status`、`summary`、`agent_count`、`tokens`、`tools`、`elapsed_seconds`、`phases[]`、`artifacts[]` |
|
||||
| `GET …/{deployment_id}/tasks` | 任务 DAG | 每任务 `task_id`、`agent_role`、`status`、`depends_on`、`parent_task_id`、`root_task_id`、`attempt`、`blocked_reason` |
|
||||
| `GET …/{deployment_id}/logs`（`/events`） | 事件流 | 事件 envelope（见 event-schema），分页 `cursor`/`limit` |
|
||||
| `GET …/{deployment_id}/metrics` | 指标 | `tasks_by_status`、`agents_connected`、`average_task_duration_seconds`、`budget` |
|
||||
| `GET …/{deployment_id}/diagnostics` | 诊断 | 失败任务、回调尝试、审批 |
|
||||
|
||||
`phases[]` 形如：`{phase_id, name, status, agents[]}`，`name ∈ {Plan, Dispatch, Execute, Handoff, Review, Deliver}`；`agents[]` 形如 `{agent_id, name, role, status, tokens, tools, elapsed_seconds, artifact_ids}`。
|
||||
|
||||
> 注意：当前**仅 REST 拉取**，Swarm 未向前端推送 SSE/WS。前端可轮询 `/workflow` + `/logs`（用 `cursor` 增量）。
|
||||
|
||||
## 4. 目标：蜂群前端事件 API（待定义）
|
||||
|
||||
为支撑工单要求的 swarm graph / agent topology / task DAG / review loop / retry timeline / token-cost monitor / collaboration timeline，建议（待 Frontend/Product Team 冻结）：
|
||||
|
||||
| UI 视图 | 数据来源（现有） | 缺口 |
|
||||
|---|---|---|
|
||||
| Task DAG / topology | `/tasks`（`depends_on`）+ `/workflow.phases` | 稳定 graph schema、增量更新事件 |
|
||||
| Review loop / retry timeline | `task.retried` + `timeline.updated`（`Review cycle N`） | 独立 review/retry 事件类型与时间线 schema |
|
||||
| Collaboration timeline | `handoff.requested/completed` + peer 路由 | peer 消息事件未对前端暴露 |
|
||||
| Token / cost monitor | `/metrics` + `budget.alert`（usage） | 统一 usage 推送（见 usage-billing §7） |
|
||||
| 实时推送 | `/logs` 轮询 | 统一 SSE/WS event stream（事件名、断线重连、刷新可恢复） |
|
||||
|
||||
建议事件流形态（草案）：SSE `GET …/{deployment_id}/stream`，事件名复用 `event-schema` 的 `event_type`，载荷为对应 envelope；前端按 `swarm_id` 渲染、按 `event_id` 去重、断线后用 `/logs?cursor=` 回补。
|
||||
|
||||
## 5. 缺口与待对齐
|
||||
|
||||
- 🔴 Swarm 未提供前端 SSE/WS 推送（仅 REST 轮询）。
|
||||
- 🔴 蜂群 graph/timeline/cost 的统一前端 schema 未定义。
|
||||
- 🟡 review/retry/collaboration 缺独立事件类型（当前借 `task.retried`/`timeline.updated`）。
|
||||
- 待对齐：Frontend / Product Team 确认蜂群三视图（graph / timeline / cost）入口与事件流形态；与 HM 的 `/api/user/tasks/{id}/events` 通道如何衔接。
|
||||
@@ -0,0 +1,96 @@
|
||||
# Manager ↔ Swarm Runtime Contract（Swarm 侧拥有）
|
||||
|
||||
> 状态：**草案（Swarm 侧拥有，待 HM 对齐）** · 对齐对象：Heicode Manager Runtime Team
|
||||
>
|
||||
> 依据：
|
||||
> - HM 侧裁定 `heicode-mananger/docs/integration/heicode-swarm-deferred.md`：**HM 当前不实现 swarm runtime**；多 Agent 蜂群归 `agent_swarm` / `HeiCode-Swarm`（本仓）；HM 侧待本仓给出正式 `/api/agent/swarm/*` 接口后，在 HM 的 `docs/integration/` 另立契约跟踪。
|
||||
> - 既有运行时集成范式 `heicode-mananger/docs/integration/heicode-am-contract.md`（单 Agent 模板 Agent，经 AM 启动）。本契约沿用其鉴权、回调、env、路径覆盖等约定。
|
||||
>
|
||||
> 本文是 HM 侧 deferred 锚点所要求的「Swarm 侧正式接口」起草。冻结需 Manager Runtime Team 评审。
|
||||
|
||||
## 1. 角色与边界
|
||||
|
||||
- **HM（Heicode Manager）**：控制面。负责用户输入、资源/权限/计费/审计、模型网关（`/v1/*`）。HM **不**承载蜂群编排。
|
||||
- **Swarm Runtime（本仓）**：执行面。接收 HM 下发的编排请求，分解→派发→执行→评审→汇总，并通过**带签名回调**回报生命周期事件。
|
||||
- HM 不在对话/执行回路；Swarm 通过回调把状态推回 HM，HM 也可主动拉取（见 §4）。
|
||||
|
||||
## 2. 鉴权（沿用 AM 约定）
|
||||
|
||||
- HM → Swarm：`Authorization: Bearer <service_token>`。Swarm 侧校验 `AGENT_RUNTIME_SERVICE_TOKEN`（兼容 `AGNET_RUNTIME_SERVICE_TOKEN`）。未配置时为**非安全开发模式**（仅本地）。
|
||||
- 统一响应信封：成功 `{ "success": true, "data": {...} }`；失败 `{ "success": false, "error": { "code", "message", "request_id" } }`。
|
||||
- 常用请求头：`X-Correlation-ID`、`X-Idempotency-Key`、`Authorization`。
|
||||
|
||||
## 3. 生命周期接口（已实现 / 待对齐）
|
||||
|
||||
Swarm 暴露以下接口（三组别名等价，便于 HM 路径覆盖）：
|
||||
`/api/swarms`、`/api/agent/swarm/deployments`、`/api/agnet/deployments`。
|
||||
|
||||
| 动作 | 方法 + 路径 | 状态 | 说明 |
|
||||
|---|---|---|---|
|
||||
| **create** | `POST /api/agent/swarm/deployments` | ✅ 已实现 | 创建蜂群部署；幂等键 `X-Idempotency-Key` |
|
||||
| **status** | `GET …/{deployment_id}` | ✅ 已实现 | 返回部署概要与状态 |
|
||||
| task graph | `GET …/{deployment_id}/tasks` | ✅ 已实现 | 任务 DAG |
|
||||
| logs / events | `GET …/{deployment_id}/logs`、`/events` | ✅ 已实现 | 事件流（分页 `cursor`/`limit`） |
|
||||
| metrics / workflow / diagnostics | `GET …/{deployment_id}/metrics`、`/workflow`、`/diagnostics` | ✅ 已实现 | 指标与编排视图 |
|
||||
| **cancel** | `POST …/{deployment_id}/stop` | ✅ 已实现 | 停止并取消非终态任务，向 Agent 下发 `cancel_task` |
|
||||
| **approve** | `POST …/{deployment_id}/approvals/{approval_id}` | ✅ 已实现 | 接收 Manager 审批决定（approved/rejected） |
|
||||
| **resume** | — | 🟡 待对齐 | 当前仅「审批通过」隐式恢复（approvals）；无独立 resume 端点 |
|
||||
| **retry** | — | 🟡 待对齐 | 任务级重试 / 评审重做为内部机制；无外部 retry 端点 |
|
||||
|
||||
### 3.1 create 请求（必填校验）
|
||||
必填：`orchestration_plan.objective`、`callback.url`、`metadata.manager_deployment_id`。
|
||||
约束：`mode` 必须为 `swarm`；`billing_context.secret_ref`（若有）必须 `azkv://` 前缀；`resource_grants`/`metadata`/`callback` 不得含明文密钥。
|
||||
|
||||
create 响应 `data`：`deployment_id`、`runtime_deployment_id`、`manager_deployment_id`、`swarm_id`、`mode`、`status`、`runtime_execution_status`、`created`。
|
||||
|
||||
### 3.2 ID 语义（回应 HM 标记的 `deployment_id ↔ swarm_id` 缺口）
|
||||
- `deployment_id` / `runtime_deployment_id`：Swarm 侧运行时部署 ID（`runtime-dep-…`）。
|
||||
- `swarm_id`：工作流 ID（`swarm-…`），作为 `workflow_id`；事件按 `swarm_id` 持久化与查询。
|
||||
- `manager_deployment_id`：HM 侧部署 ID（请求 `metadata.manager_deployment_id` 透传）。
|
||||
- `correlation_id`：贯穿一次交付的追踪 ID（`X-Correlation-ID`），作为 `trace_id`。
|
||||
- 三者映射在 Swarm 持久化中维护：`deployment_id ↔ swarm_id ↔ manager_deployment_id`，`get_run_by_identifier` 支持三者任一查询。
|
||||
|
||||
> HM 侧 `heicode-swarm-deferred.md` 记录的「Swarm 仅暴露 `/tasks`、缺 `deployment_id↔swarm_id`」为旧状态；本仓 v5/v6 已实现上述映射与 `/api/agent/swarm/*` 接口，需 HM 复核更新该锚点。
|
||||
|
||||
## 4. 状态机
|
||||
|
||||
部署状态：`waiting_approval` → `running` →（`blocked` ⇄ `running`）→ 终态 `completed` / `failed` / `stopped`。
|
||||
|
||||
| 状态 | 含义 | 进入方式 |
|
||||
|---|---|---|
|
||||
| `waiting_approval` | 高危/需审批，等待 Manager 审批 | create 时命中审批条件 |
|
||||
| `running` | 任务派发与执行中 | 审批通过 / 有在途任务 |
|
||||
| `blocked` | 任务因移交/依赖阻塞 | 子任务 `blocked_on_handoff` |
|
||||
| `completed` | 所有任务终态且评审通过 | 全部完成（评审循环可选） |
|
||||
| `failed` | 存在失败且不可恢复 | 任务终态含 failed |
|
||||
| `stopped` | Manager 主动停止 | `…/stop` |
|
||||
|
||||
每次状态变更通过回调 `deployment.status_changed` 推送（见 §5）。
|
||||
|
||||
## 5. 回调（已与 HM 处理器对齐）
|
||||
|
||||
Swarm → HM：`POST {callback.url}`（create 时下发，默认 HM 的 `/api/agent/callbacks/runtime-events`）。
|
||||
|
||||
**鉴权与签名（与 HM `agent_callback.go` 一致，已实现）：**
|
||||
- 头：`X-Agent-Service-Token`（或 `Authorization: Bearer`）、`X-Agent-Timestamp`（Unix 毫秒）、`X-Agent-Signature`、`X-Agent-Event-Id`、`X-Correlation-ID`（均含 `X-Agnet-` 兼容别名）。
|
||||
- 签名：`signature = "sha256=" + hex(HMAC_SHA256(secret, f"{timestamp}.{event_id}.{raw_body}"))`。
|
||||
- secret：`AGENT_CALLBACK_SIGNING_SECRET`（兼容 `AGNET_…`）；HM 侧容差默认 300s。
|
||||
- 幂等：HM 按 `X-Agent-Event-Id` → body `event_id` → `idempotency_key` 去重。
|
||||
|
||||
事件 envelope 字段与 event_type 取值见 [`event-schema.md`](./event-schema.md)（与 HM 注册表一致）。
|
||||
|
||||
## 6. 路径与超时覆盖（沿用 AM 约定）
|
||||
|
||||
HM 侧可用 env 覆盖：`AGENT_RUNTIME_BASE_URL`、`AGENT_RUNTIME_SERVICE_TOKEN`、`AGENT_RUNTIME_AGENT_START_PATH`、`AGENT_RUNTIME_AGENT_PATH`、`AGENT_RUNTIME_AGENT_STOP_PATH`、`AGENT_RUNTIME_START_TIMEOUT_SECONDS`、`AGENT_RUNTIME_CALLBACK_URL`。
|
||||
|
||||
## 7. 安全
|
||||
|
||||
- `env`/请求体不得含明文密钥；凭据经 `secret_ref`（`azkv://`）注入。详见 [`security-boundary.md`](./security-boundary.md)。
|
||||
- Swarm 启动接口必须 HTTPS / 私网；回调走 HTTPS。
|
||||
|
||||
## 8. 待对齐项（冻结前需 Manager Runtime Team 确认）
|
||||
|
||||
- [ ] resume / retry 是否需要独立外部端点，还是沿用 approvals + 内部重做。
|
||||
- [ ] HM 是否以 AM 同款 `/agents` 生命周期（而非 `/api/agent/swarm/*`）调用 Swarm；若是，需路径映射。
|
||||
- [ ] `event-schema` 中各 event_type 的必填字段与 HM 注册表逐项核对（见 event-schema.md）。
|
||||
- [ ] 更新 HM 侧 `heicode-swarm-deferred.md` 锚点，登记本契约。
|
||||
@@ -0,0 +1,85 @@
|
||||
# 安全边界（Secret / Workspace / Tool / Approval / Tenant / Sandbox）
|
||||
|
||||
> 状态：**部分已实现，部分待接入**（依据 `heicode-mananger/docs/heicode.md §六/七`、`docs/heicode-runtime-auth-newapi-secret-design.md §三/五`、`docs/integration/heicode-am-contract.md §3.1/§4`）。
|
||||
>
|
||||
> 配套：[`runtime-contract.md`](./runtime-contract.md)、[`usage-billing-schema.md`](./usage-billing-schema.md)。
|
||||
|
||||
## 1. 不可破坏原则
|
||||
|
||||
- 密钥/Token/云凭据/SSH 私钥/数据库密码/NewAPI key **不得**进入代码、日志、Markdown、前端响应或 Git。
|
||||
- 凭据基线 = **Azure Key Vault**；引用一律 `azkv://<vault>/secrets/<name>`，**不向后兼容 `vault://`**。
|
||||
- Manager DB 与 Swarm 只保存 `secret_ref`，真实凭证由 Secret Broker 写入 Key Vault。
|
||||
- 子 Agent 不持有长期密钥；只接收角色、资源元数据、AGENT.md 与**短期、最小权限、可审计**凭证。
|
||||
- 高危操作审批**只在客户端完成**；运行时只校验审批结果，不发起审批。
|
||||
|
||||
## 2. Secret 注入边界
|
||||
|
||||
| 项 | 标准 | 本仓状态 |
|
||||
|---|---|---|
|
||||
| `secret_ref` 前缀 `azkv://` 强校验 | 必须 | ✅ `swarm_runtime.validate_create_request` 校验 `billing_context.secret_ref` 等为 `azkv://` |
|
||||
| 拒绝明文密钥进入请求 | 必须 | ✅ `_reject_plaintext_secrets`（`metadata`/`resource_grants`/`callback`） |
|
||||
| 响应/事件/持久化脱敏 | 必须 | ✅ `_redact_sensitive` 将明文密钥键（`password`/`token`/`secret`/`private_key`/`access_key` 及 `*_token`/`*_secret`/`*_password`/`*_key`）脱敏为 `[redacted]`；**保留** `secret_ref`/`credential_ref`/`signing_secret_ref`（安全引用） |
|
||||
| `.env`/密钥不入库 | 必须 | ✅ `.gitignore` 忽略 `.env`、`secrets/`、`*.pem/key/p12/pfx`、`id_rsa/ed25519` |
|
||||
| 真实凭证写入 Key Vault（Secret Broker） | Manager 侧 | ⛔ 非本仓（Manager Secret Broker 负责） |
|
||||
| 短期凭证派生与注入 | 客户端审批后 | 🟡 Swarm 接收 `secret_context`；K8s 派生/注入由 Agent 平台（AM）实现 |
|
||||
|
||||
`secret_context`（HM 下发，仅引用与审批结果）：
|
||||
```jsonc
|
||||
"secret_context": {
|
||||
"secret_refs": ["azkv://heicode-kv.vault.azure.net/secrets/res_git_1"],
|
||||
"inject_short_lived_credentials": true,
|
||||
"approval_id": "approval_123"
|
||||
}
|
||||
```
|
||||
|
||||
## 3. Workspace 隔离
|
||||
|
||||
- 每个任务在**独立按任务工作目录**执行（agent `task_workspace`）。
|
||||
- 文件写入有**路径越界校验**（`task_executor._resolve_workspace_path`，拒绝绝对路径与逃逸 workspace）。
|
||||
- Git 操作在仓库根（`repo_root`）执行，限定结果分支。
|
||||
- 🟡 待接入：跨任务/跨租户的强隔离、只读挂载、allowed_paths 强制（当前由模型提示约束，未做运行时强制）。
|
||||
|
||||
## 4. Tool / MCP 权限边界
|
||||
|
||||
- 当前 Agent 工具能力 = 工作区内文件读写 + Git；**无** SK/MCP 工具权限引擎。
|
||||
- 🟡 待接入：统一 tool/MCP permission boundary、allowed/denied 工具策略、敏感工具审批联动（`sk_tool.*` 事件已在 schema 预留）。
|
||||
|
||||
## 5. 审批门（Approval Gate）
|
||||
|
||||
- 高危操作审批**只在客户端**；Swarm 不发起审批。
|
||||
- Swarm 实现：create 命中高危（`risk_level=high` 或 `requires_user_approval`）→ 进入 `waiting_approval`，发 `approval.requested`（`approval_id`/`operation`/`risk_level`）；客户端经 Manager 审批后回 `POST …/approvals/{approval_id}`（approved/rejected）→ 恢复或阻断。
|
||||
- 🟡 待接入：审批主体/范围/TTL/`credential_ref`/`lease_id` 的逐项校验与到期失效，需与 Manager 审批链对齐。
|
||||
|
||||
## 6. Agent 鉴权与租户隔离
|
||||
|
||||
- **Swarm 模型**：Agent 主动出站连编排器 WebSocket（`/ws/{agent_id}`），**不**对公网暴露每 Agent 子域名。AM 单 Agent 模型里的「客户端↔agent 直连 + `AGENT_ACCESS_TOKEN` 本地校验」**不适用于** swarm（无直连回路）。
|
||||
- 服务间鉴权：HM→Swarm 用 `AGENT_RUNTIME_SERVICE_TOKEN`（Bearer）；回调 HMAC 签名。
|
||||
- 🟡 待接入：多租户运行时隔离（命名空间/网络/配额）由 Agent 平台（AKS Workload Identity）承载，非本仓编排器；归因主轴为 `user.id`/`channelId`（见 `usage-billing-schema.md`），不引入 tenant 概念。
|
||||
|
||||
## 7. 外部 API 与传输
|
||||
|
||||
- 模型调用统一走 HM `/v1`（OpenAI 兼容），用 HM 现签 `OPENAI_API_KEY`。
|
||||
- 传输：编排器/Agent 接口与回调走 **HTTPS / 私网**；`env` 含明文密钥时启动接口必须 HTTPS（`heicode-am-contract §4`）。
|
||||
- 🟡 待接入：统一 external API egress 策略（白名单/出网控制）。
|
||||
|
||||
## 8. 执行沙箱
|
||||
|
||||
- 当前执行单元为进程 / K8s Pod，隔离强度依赖部署（namespace/资源限额）。
|
||||
- 🟡 待接入：强化沙箱（seccomp/只读根/网络策略/能力裁剪），由 Infra/Security Team 定义。
|
||||
|
||||
## 9. 覆盖与缺口
|
||||
|
||||
| 边界 | 状态 |
|
||||
|---|---|
|
||||
| `azkv://` `secret_ref` 强校验 / 明文拒绝 / 脱敏 / `.gitignore` | ✅ 已实现（本仓） |
|
||||
| Workspace 路径越界校验 | ✅ 已实现 |
|
||||
| 审批状态机（waiting_approval + approvals 回执） | ✅ 已实现（逐项校验待加强） |
|
||||
| 短期凭证派生注入、Workload Identity | 🟡 AM/K8s 侧 |
|
||||
| Tool/MCP 权限引擎 | 🔴 未实现 |
|
||||
| allowed_paths 运行时强制、强隔离 | 🔴 未实现 |
|
||||
| 强化执行沙箱 | 🔴 未实现 |
|
||||
| 租户隔离 | ⛔ 不在本仓（归因按 user/channelId） |
|
||||
|
||||
## 10. 待对齐对象
|
||||
|
||||
Security / Governance Team（tool/MCP 边界、沙箱、审批逐项校验）、Infra Team（Workload Identity、租户隔离、egress 策略）。
|
||||
@@ -0,0 +1,108 @@
|
||||
# 用量与计费 Schema（Usage / Billing）
|
||||
|
||||
> 状态：**已对齐既有边界**（依据 `heicode-mananger/docs/heicode.md §八`、`docs/heicode-runtime-auth-newapi-secret-design.md §四/五`、`docs/integration/Heicode-Manager-PayPal支付接入与计费关系说明.md §3/§5`）。
|
||||
>
|
||||
> 配套：事件载体见 [`event-schema.md`](./event-schema.md)（`budget.alert`）；安全见 [`security-boundary.md`](./security-boundary.md)。
|
||||
|
||||
## 1. 计费主线原则（不可破坏）
|
||||
|
||||
- **NewAPI 是计费账本**：模型调用费用在 HM 模型网关 `/v1/*` 处**按请求计量并扣费**（NewAPI 的 user / token / group / quota / usage）。**Swarm 不是账本**，也不重算扣费。
|
||||
- **Swarm 的职责 = 用量上报与预算约束**，用于归因、观测与预算告警，不作为扣费依据。
|
||||
- **归因主轴 = `user.id` / `channelId` / NewAPI user/token/group**，**不是** `tenant`/`project`。`heicode-runtime-auth §六` 明确：当前不引入 tenant/project 作为计费主轴。
|
||||
- Agent 基础设施成本（CPU/内存/Pod）当前**未形成用户账本**（PayPal 说明 §5）。
|
||||
|
||||
> ⚠️ 与工单口径的差异：工单要求「tenant attribution」；按 Manager 标准应改为 **user/channelId 归因**。如未来确需租户账本，需作为独立产品决策，不在本仓暗自添加。
|
||||
|
||||
## 2. 计费归因上下文（HM 下发，Swarm 透传）
|
||||
|
||||
来自 `heicode-runtime-auth §五`：
|
||||
|
||||
```jsonc
|
||||
"billing_context": {
|
||||
"provider": "newapi",
|
||||
"newapi_user_ref": "newapi_user_123",
|
||||
"newapi_group": "development",
|
||||
"quota_ref": "newapi_token_or_group_quota_ref"
|
||||
}
|
||||
```
|
||||
|
||||
- Agent 调模型走 HM `/v1`，用 **HM 为该用户现签的 `OPENAI_API_KEY`（sk-）**，扣费即记在该用户名下；删 agent 时吊销。
|
||||
- Swarm 的 task_executor 随模型请求带 **`X-Agent-*` / `X-Agnet-*` 归因头**（`manager_deployment_id`、`swarm_id`、`task_id`、`agent_role`、`correlation_id`、`model_id`），供 HM/NewAPI 关联归因。
|
||||
|
||||
## 3. 预算约束（orchestration_plan.budget）
|
||||
|
||||
来自 PayPal 说明 §5：
|
||||
|
||||
| 字段 | 含义 |
|
||||
|---|---|
|
||||
| `budget.max_tokens` / `token_limit` | 本次部署 token 上限 |
|
||||
| `budget.max_cost_usd` | 本次部署美元成本上限 |
|
||||
| `budget.duration_seconds` / `max_duration_seconds` | 运行时长上限 |
|
||||
|
||||
Swarm 在运行中按时长/成本比例发 `budget.alert`（默认 80% 阈值），并在 `metrics` 返回预算占比。**这是约束与告警，不是扣费。**
|
||||
|
||||
## 4. 用量上报 Schema
|
||||
|
||||
### 4.1 Agent 单任务用量（task 结果 `usage`，已实现）
|
||||
```jsonc
|
||||
"usage": {
|
||||
"model_id": "gpt-4o-mini",
|
||||
"model_tokens": 0,
|
||||
"prompt_tokens": 0,
|
||||
"completion_tokens": 0,
|
||||
"model_cost_usd": 0.0,
|
||||
"runtime_seconds": 0.0,
|
||||
"billing_source": "newapi | unknown"
|
||||
}
|
||||
```
|
||||
|
||||
### 4.2 运行级用量事件（orchestrator，已实现）
|
||||
通过 `budget.alert` 事件上报（payload 含）：
|
||||
```jsonc
|
||||
{
|
||||
"model_id": "...", "model_tokens": 0, "prompt_tokens": 0, "completion_tokens": 0,
|
||||
"model_cost_usd": 0.0, "runtime_seconds": 0.0, "billing_source": "...",
|
||||
"manager_deployment_id": "...", "swarm_id": "...", "task_id": "...",
|
||||
"agent_role": "...", "correlation_id": "...",
|
||||
"budget": { "max_tokens": null, "max_cost_usd": null, "consumed_usd": 0.0, "remaining_usd": null }
|
||||
}
|
||||
```
|
||||
|
||||
### 4.3 目标完整回传形态（对齐 AM，部分待实现）
|
||||
PayPal 说明 §5 给出的运行时用量回传目标：
|
||||
```jsonc
|
||||
{
|
||||
"deployment_id": "dep_xxx", "agent_instance_id": "agi_backend_001",
|
||||
"usage": {
|
||||
"model_tokens": 32000, "model_cost_usd": 4.21, "runtime_seconds": 930,
|
||||
"cpu_core_seconds": 1200, "memory_mb_seconds": 2048000
|
||||
},
|
||||
"billing_source": "newapi", "correlation_id": "corr_xxx"
|
||||
}
|
||||
```
|
||||
|
||||
## 5. 多 Agent / 评审重做 成本聚合
|
||||
|
||||
- 一次 swarm 请求拆成多 task / 多 agent / 多评审轮；**每个 task 的 `usage` 可按 `swarm_id` 聚合**得到运行级合计（观测用途）。
|
||||
- **评审重做成本**：每轮重做都会重新执行任务并累计 `usage`，因此**已隐含计入**运行级合计；但当前**未单独打标**「review_retry 成本」。
|
||||
|
||||
## 6. 字段覆盖与缺口
|
||||
|
||||
| 字段 | 状态 |
|
||||
|---|---|
|
||||
| `model_tokens` / `prompt_tokens` / `completion_tokens` | ✅ 已采集 |
|
||||
| `model_cost_usd`、`runtime_seconds`、`billing_source` | ✅ 已采集 |
|
||||
| 归因：`manager_deployment_id`/`swarm_id`/`task_id`/`agent_role`/`correlation_id` | ✅ 已采集 |
|
||||
| `reasoning_tokens` / `cache_tokens` | 🔴 未采集（取决于 provider usage 返回） |
|
||||
| `tool_cost`（工具调用成本） | 🔴 未采集（无 SK 工具计量） |
|
||||
| `cpu_core_seconds` / `memory_mb_seconds`（基础设施） | 🔴 未采集（K8s 指标未接入账本） |
|
||||
| `review_retry` 成本单独打标 | 🟡 隐含累计，未单独标注 |
|
||||
| provider 成本拆分 | 🟡 由 NewAPI/账本侧负责，非本仓 |
|
||||
| tenant 归因 | ❌ 按标准不使用（见 §1） |
|
||||
|
||||
## 7. 待对齐项
|
||||
|
||||
- [ ] 运行级用量是否需独立 `usage.report` 事件类型（当前借 `budget.alert` 载体）；与 HM/Billing Team 确认（HM 注册表暂无 usage 事件）。
|
||||
- [ ] `budget.alert` 需补 `threshold_pct`（见 `event-schema.md §6`）。
|
||||
- [ ] 若需基础设施计费，接入 K8s 用量（cpu/memory）并由 Billing Team 定义资源计价。
|
||||
- [ ] `review_retry` 成本是否需单独打标，由 Billing Team 决定。
|
||||
@@ -0,0 +1,117 @@
|
||||
# Peer Communication Logistics（对等通信机制与深化设计）
|
||||
|
||||
> 状态：**现状 + 深化设计**。描述专家 Agent 之间「就重叠领域相互沟通」的消息机制、路由逻辑、当前局限，以及深化方案与分阶段计划。
|
||||
>
|
||||
> 相关代码：`agent/main.py`（`request_peer_collaboration` / `handle_peer_message` / `answer_peer_query`）、`orchestrator/main.py`（`peer_message` 路由 / `build_dispatch_context`）、`agent/task_executor.py`（`_execute_subtask` 触发）。相关文档：[`event-schema`](./integration/event-schema.md)、[`frontend-event-api`](./integration/frontend-event-api.md)、[`benchmark/swarm-metrics-schema`](./benchmark/swarm-metrics-schema.md)（`S_communication`）。
|
||||
|
||||
---
|
||||
|
||||
## 1. 当前实现（mechanics）
|
||||
|
||||
### 1.1 消息信封
|
||||
```jsonc
|
||||
{
|
||||
"type": "peer_message",
|
||||
"agent_id": "发送方",
|
||||
"target_agent_id": "接收方",
|
||||
"from_agent_id": "由编排器转发时补盖（接收方据此回复）",
|
||||
"task_id": "...",
|
||||
"content": "文本内容",
|
||||
"correlation_id": "peer-<task>-<rand>", // 关联请求与回复
|
||||
"is_reply": false, // true = 回复
|
||||
"timestamp": 0.0
|
||||
}
|
||||
```
|
||||
|
||||
### 1.2 路由（编排器为 broker）
|
||||
Agent 出站连编排器 WebSocket，彼此不直连。编排器收到 `peer_message` → 按 `target_agent_id` 转发，并**补盖 `from_agent_id`**（发送方身份）。无 `target_agent_id` 则记告警丢弃。
|
||||
|
||||
### 1.3 请求/回复（一问一答）
|
||||
- 发起方 `request_peer_collaboration(task_id, target_agent_id, content, timeout)`：生成 `correlation_id`，挂一个 `asyncio.Future` 到 `peer_waiters[correlation_id]`，发出 query，`await asyncio.wait_for(waiter, timeout)`。
|
||||
- 接收方 `handle_peer_message`：若 `correlation_id` 命中自身 waiter（或 `is_reply`）→ 解析为「回复」并 resolve waiter；否则视为「入站查询」→ `answer_peer_query`。
|
||||
- `answer_peer_query`：**已实现实质性回复**——对查询做一次受限 LLM 推理（`TaskExecutor.peer_reply`，受 `PEER_CONSULT_MAX_TOKENS`，默认 500 约束），结合自身 workspace 产物，返回结构化 `{stance, content, evidence, refs}`；无模型/超时/失败时回退到 `last_summary` 摘要。回复在后台任务中完成，不阻塞消息循环（`PEER_REPLY_TIMEOUT_SECONDS`，默认 20s）。
|
||||
|
||||
### 1.4 触发与编排（logistics）
|
||||
- 派发时 `build_dispatch_context` 向任务上下文注入：`peer_agents`（本次 run 中**已连接**的其它 Agent：`agent_id`/`role`/`capabilities`）与 `dependency_artifacts`（已完成依赖产物）。
|
||||
- `task_executor._execute_subtask` 仅当 `specialist_role ∈ {testing, documentation}` 且存在 `peer_agents` 且提供了回调时发起咨询；按 `prefer implementation` 排序，取前 `max_peer_consults`（默认 2）个，`peer_timeout_seconds`（默认 10s）。
|
||||
- 对齐原则（提示词内）：实现产物为 API/异常语义的 source of truth；peer 输入与实现冲突时以实现为准并在变更摘要中说明。
|
||||
|
||||
---
|
||||
|
||||
## 2. 当前局限（为何"浅"）
|
||||
|
||||
| 局限 | 说明 |
|
||||
|---|---|
|
||||
| ~~回复浅~~ ✅ 已深化 | `answer_peer_query` 现做受限 LLM 推理 + 引用 workspace 产物 + 结构化字段（`stance/content/evidence/refs`）；无模型时回退摘要（见 §3.3） |
|
||||
| 触发窄 | 仅 testing/documentation 角色发起；实现角色不主动协作 |
|
||||
| 单轮 | 一问一答，无多轮/澄清/协商线程 |
|
||||
| 无广播 | 只能点对点 `target_agent_id`，无按角色/能力的群发或发现 |
|
||||
| 无遥测 | 消息未计数（成功/失败/总数）→ `S_communication` 不可算（见 metric-coverage-gaps） |
|
||||
| 弱冲突处理 | 仅"以实现为准"的提示约定，无结构化协商/升级协议 |
|
||||
| 无前端可见 | peer 消息未作为事件暴露，前端协作时间线无数据 |
|
||||
|
||||
---
|
||||
|
||||
## 3. 深化设计（target）
|
||||
|
||||
### 3.1 消息类型（扩展 `kind`）
|
||||
`query`（提问）·`reply`（回答）·`clarify`（追问）·`proposal`（提案）·`critique`（质疑）·`broadcast`（群发）·`ack`（确认）。
|
||||
|
||||
### 3.2 多轮线程
|
||||
增加 `conversation_id` + `turn`；`peer_waiters` 升级为按会话的队列，支持 `clarify` 往返与超时续约，受 `max_rounds` 约束。
|
||||
|
||||
### 3.3 实质性回复 ✅ 已实现
|
||||
`answer_peer_query` 已由「回 last_summary」升级为：**针对 query 生成有据回复**（`TaskExecutor.peer_reply` 做一次受限推理，引用自身 workspace 产物），返回结构化 `{stance, content, evidence, refs}`，受 `PEER_CONSULT_MAX_TOKENS`（默认 500）约束；无模型/失败回退摘要；后台任务回复不阻塞消息循环。由 `scripts/test-merge-smoke.py` 覆盖（fallback + 实质回复两条断言）。
|
||||
|
||||
### 3.4 寻址与发起
|
||||
- 点对点（`target_agent_id`）+ **按角色/能力寻址**（编排器按注册表解析）+ 受控**广播**。
|
||||
- 任何角色均可发起咨询（不限 testing/doc），由预算与策略限制频次。
|
||||
|
||||
### 3.5 冲突解决协议
|
||||
`critique`/`proposal` 往返；默认「实现为 source of truth」；**僵局上交 Master Agent**（`master_agent` 仲裁，纳入评审决策），而非各 Agent 私下定夺。
|
||||
|
||||
### 3.6 遥测（接入 benchmark）
|
||||
编排器对每条 `peer_message` 计数：`total` / `delivered` / `failed`（接收方离线/超时）。
|
||||
- 产出 `S_communication = SuccessfulMessages / TotalMessages`（见 swarm-metrics-schema）。
|
||||
- 作为协作事件暴露（见 §4），供前端协作时间线与审计。
|
||||
|
||||
### 3.7 预算与可靠性
|
||||
`max_peer_consults`、`peer_timeout_seconds`、`max_rounds`、`peer_consult_max_tokens`；接收方离线 → 立即回 `failed` 而非干等；`correlation_id` 幂等去重。
|
||||
|
||||
---
|
||||
|
||||
## 4. 事件与可见性
|
||||
|
||||
为满足前端与审计，建议把对等协作纳入事件流（与 [`event-schema`](./integration/event-schema.md) 对齐）：
|
||||
- 新增/复用事件：`collaboration.message`（含 `from`/`to`/`kind`/`conversation_id`/`correlation_id`，内容脱敏）、`collaboration.resolved`/`collaboration.escalated`。
|
||||
- 前端按 `conversation_id` 渲染协作时间线（见 frontend-event-api §4）。
|
||||
- HM 注册表暂无 `collaboration.*` 事件 → 列为与 Workflow/Frontend Team 的待对齐项。
|
||||
|
||||
---
|
||||
|
||||
## 5. 与其它机制的关系
|
||||
|
||||
| 机制 | 作用 | 区别 |
|
||||
|---|---|---|
|
||||
| `dependency_artifacts` | 顺序依赖：下游看到上游产物 | 单向、派发时注入，非交互 |
|
||||
| handoff | 把子任务委派给更合适的专家 | 转移所有权，非对等沟通 |
|
||||
| peer communication | 重叠领域的对等交流 | 双向、运行中、不转移所有权 |
|
||||
| Master Agent 评审 | 全局裁决是否达标 | 中心化决策，非点对点 |
|
||||
|
||||
---
|
||||
|
||||
## 6. 分阶段计划
|
||||
|
||||
1. **遥测先行（低成本）**：编排器对 `peer_message` 计数 → 点亮 `S_communication`；把消息作为 `collaboration.message` 事件暴露。
|
||||
2. **实质回复** ✅ 已实现：`answer_peer_query` 改为受限 LLM 推理 + 引用产物（替换 last_summary）。
|
||||
3. **多轮 + 寻址**：`conversation_id`/`clarify`、角色/能力寻址、广播。
|
||||
4. **冲突协议 + 升级 Master**：`critique`/`proposal`，僵局上交 `master_agent`。
|
||||
5. 与 Frontend/Workflow Team 冻结 `collaboration.*` 事件与协作时间线契约。
|
||||
|
||||
---
|
||||
|
||||
## 7. 待对齐
|
||||
|
||||
- `collaboration.*` 事件类型与载荷（Workflow/Frontend Team；当前 HM 注册表未含）。
|
||||
- 对等咨询的预算口径（`max_rounds`/token）与频次策略（Governance）。
|
||||
- 实质回复是否计入用量/计费（每次 peer consult 是一次模型调用 → 见 usage-billing §5）。
|
||||
+26
-4
@@ -24,6 +24,7 @@ from .handoff_manager import handoff_manager, HandoffRequest
|
||||
from .task_queue import task_queue, TaskStatus
|
||||
from .swarm_runtime import RuntimeValidationError, swarm_runtime
|
||||
from .planner import planner
|
||||
from .master_agent import master_agent
|
||||
|
||||
# Configure logging
|
||||
logging.basicConfig(
|
||||
@@ -217,7 +218,7 @@ async def maybe_run_review_cycle(run, tasks) -> bool:
|
||||
return False
|
||||
|
||||
results = {t.task_id: {"result": parse_task_result(t) or {}} for t in completed}
|
||||
verdict = await planner.review(
|
||||
verdict = await master_agent.review_and_decide(
|
||||
run.objective,
|
||||
[task_payload(t) for t in completed],
|
||||
results,
|
||||
@@ -382,7 +383,7 @@ async def refresh_swarm_run_status(run):
|
||||
# Synthesize the specialist results into one coherent, user-facing answer.
|
||||
completed = [t for t in tasks if t.status == TaskStatus.COMPLETED]
|
||||
results = {t.task_id: {"result": parse_task_result(t) or {}} for t in completed}
|
||||
final_summary = await planner.synthesize(run.objective, results)
|
||||
final_summary = await master_agent.synthesize(run.objective, results)
|
||||
run.metadata["final_summary"] = final_summary
|
||||
await swarm_runtime.save_run(run)
|
||||
|
||||
@@ -538,7 +539,7 @@ async def build_planner_task_specs(run, body: Dict[str, Any], base_specs: List[D
|
||||
"""
|
||||
objective = run.objective or "Complete swarm objective"
|
||||
base_context = (base_specs[0].get("context") if base_specs else {}) or {}
|
||||
subtasks = await planner.build_plan(run.swarm_id, objective)
|
||||
subtasks = await master_agent.plan(run.swarm_id, objective)
|
||||
|
||||
prefix = f"{run.swarm_id}-"
|
||||
|
||||
@@ -1017,6 +1018,8 @@ async def finalize_parent_after_child(run, child_task, agent_id: str, success: b
|
||||
**payload,
|
||||
"child_task_id": child_task.task_id,
|
||||
"parent_task_id": parent_task.task_id,
|
||||
"from_role": parent_task.agent_role,
|
||||
"to_role": child_task.agent_role,
|
||||
},
|
||||
)
|
||||
await swarm_runtime.emit_event(
|
||||
@@ -1896,6 +1899,10 @@ async def websocket_endpoint(websocket: WebSocket, agent_id: str):
|
||||
"source_agent_id": handoff.source_agent_id,
|
||||
"target_agent_id": handoff.target_agent_id,
|
||||
"task_id": task_id,
|
||||
# HM requires from_role/to_role; this legacy accept path
|
||||
# lacks role context, so fall back to agent ids.
|
||||
"from_role": handoff.source_agent_id,
|
||||
"to_role": handoff.target_agent_id,
|
||||
},
|
||||
)
|
||||
|
||||
@@ -2130,7 +2137,22 @@ async def websocket_endpoint(websocket: WebSocket, agent_id: str):
|
||||
logger.info(
|
||||
f"Agent {agent_id} rejected task {rejected_task_id} ({reject_reason}); requeuing"
|
||||
)
|
||||
await task_queue.release_task(rejected_task_id, agent_id=agent_id)
|
||||
released = await task_queue.release_task(rejected_task_id, agent_id=agent_id)
|
||||
if released:
|
||||
released_task = await task_queue.get_task(rejected_task_id)
|
||||
released_run = await swarm_runtime.get_run_for_task(rejected_task_id)
|
||||
if released_task and released_run:
|
||||
await swarm_runtime.emit_event(
|
||||
released_run,
|
||||
"task.released",
|
||||
task_id=rejected_task_id,
|
||||
agent_instance_id=agent_id,
|
||||
payload={
|
||||
"task_id": rejected_task_id,
|
||||
"agent_role": released_task.agent_role,
|
||||
"reason": reject_reason,
|
||||
},
|
||||
)
|
||||
|
||||
elif message_type == "peer_message":
|
||||
# Route peer collaboration messages between agents; stamp the sender so the
|
||||
|
||||
@@ -0,0 +1,69 @@
|
||||
"""Master Agent — the decision-making entity of a swarm run.
|
||||
|
||||
This makes the "master" a first-class entity rather than scattered orchestrator functions.
|
||||
The master agent owns the *cognitive* loop:
|
||||
- plan() : decompose the objective into specialist subtasks
|
||||
- review_and_decide(): judge whether the combined result is good enough, and which tasks to redo
|
||||
- synthesize() : compose the specialists' outputs into one user-facing answer
|
||||
|
||||
It uses an LLM as its brain (via `planner`, with deterministic fallbacks). The orchestrator
|
||||
remains the "hands": it executes the master's decisions (dispatch, reopen tasks, persist state,
|
||||
emit events). This separation keeps decisions in one named entity while leaving runtime
|
||||
mechanics (and the Manager contract) in the orchestrator.
|
||||
|
||||
NOTE: agent→task dispatch is still capability-matched in the orchestrator loop; `select_agent`
|
||||
below is the seam where the master can later own assignment (LLM-driven), without changing the
|
||||
loop's contract today.
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
import logging
|
||||
from typing import Any, Dict, List, Optional
|
||||
|
||||
from .planner import planner
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
|
||||
class MasterAgent:
|
||||
"""The master agent: decompose → decide → synthesize for a swarm run."""
|
||||
|
||||
def __init__(self, brain=planner):
|
||||
# `brain` is the LLM-backed planner (build_plan / review / synthesize), swappable for tests.
|
||||
self.brain = brain
|
||||
|
||||
async def plan(self, run_id: str, objective: str) -> List[Dict[str, Any]]:
|
||||
"""Decompose the user objective into specialist subtasks."""
|
||||
subtasks = await self.brain.build_plan(run_id, objective)
|
||||
logger.info(f"[master] planned {len(subtasks)} subtask(s) for run {run_id}")
|
||||
return subtasks
|
||||
|
||||
async def review_and_decide(self, objective: str, tasks: List[Dict[str, Any]],
|
||||
results: Dict[str, Any]) -> Dict[str, Any]:
|
||||
"""Decide whether the combined result is good enough.
|
||||
|
||||
Returns the master's verdict: {accepted: bool, summary: str, retry_tasks: [task_id,...]}.
|
||||
The orchestrator acts on it (reopen retry_tasks or finalize).
|
||||
"""
|
||||
verdict = await self.brain.review(objective, tasks, results)
|
||||
logger.info(
|
||||
f"[master] review decision: accepted={verdict.get('accepted')} "
|
||||
f"retry={verdict.get('retry_tasks')}"
|
||||
)
|
||||
return verdict
|
||||
|
||||
async def synthesize(self, objective: str, results: Dict[str, Any]) -> str:
|
||||
"""Compose the specialists' outputs into one user-facing answer."""
|
||||
return await self.brain.synthesize(objective, results)
|
||||
|
||||
def select_agent(self, task, candidates: List[Any]) -> Optional[Any]:
|
||||
"""Choose which agent runs a task. Seam for future LLM-driven assignment.
|
||||
|
||||
Default: first capability-eligible candidate (capability matching is enforced upstream
|
||||
by task_queue.get_ready_pending_task), preserving current dispatch behavior.
|
||||
"""
|
||||
return candidates[0] if candidates else None
|
||||
|
||||
|
||||
# Singleton master agent for the runtime.
|
||||
master_agent = MasterAgent()
|
||||
@@ -427,6 +427,15 @@ class SwarmRuntime:
|
||||
**redacted_payload,
|
||||
}
|
||||
|
||||
# Conform to HM event contract required fields (heicode agent_callback.go):
|
||||
# timeline.updated requires `title`; budget.alert requires `threshold_pct`.
|
||||
if event_type == "timeline.updated" and "title" not in redacted_payload:
|
||||
redacted_payload["title"] = redacted_payload.get("summary") or "timeline"
|
||||
if event_type == "budget.alert" and "threshold_pct" not in redacted_payload:
|
||||
threshold = redacted_payload.get("threshold")
|
||||
if isinstance(threshold, (int, float)):
|
||||
redacted_payload["threshold_pct"] = threshold * 100 if threshold <= 1 else threshold
|
||||
|
||||
event_id = f"evt_{uuid.uuid4().hex}"
|
||||
body: Dict[str, Any] = {
|
||||
"event_id": event_id,
|
||||
|
||||
@@ -0,0 +1,86 @@
|
||||
"""Test the baseline comparison adapter (benchmark/baselines/comparison.py) — v2.0.
|
||||
|
||||
Run from agent_swarm_v6: python scripts/test-baseline-comparison.py
|
||||
"""
|
||||
import sys
|
||||
from pathlib import Path
|
||||
|
||||
sys.path.insert(0, str(Path(__file__).resolve().parents[1]))
|
||||
|
||||
from benchmark.baselines import BenchmarkRunRecord, quality, unified_metrics, compare, evaluate
|
||||
|
||||
failures = []
|
||||
|
||||
|
||||
def check(name, cond):
|
||||
print(("PASS" if cond else "FAIL"), "-", name)
|
||||
if not cond:
|
||||
failures.append(name)
|
||||
|
||||
|
||||
swarm = BenchmarkRunRecord(
|
||||
system="swarm", scenario="coding", task_set_id="coding-set-1", n_agent=5,
|
||||
completed_tasks=10, total_tasks=10,
|
||||
test_pass_rate=90, code_review_score=80, user_acceptance=85,
|
||||
budget_usd=10, actual_cost_usd=5, model_tokens=50000,
|
||||
target_time_s=600, actual_time_s=500, recovered_failures=1, total_failures=1,
|
||||
)
|
||||
sub = BenchmarkRunRecord(
|
||||
system="sub", scenario="coding", task_set_id="coding-set-1", n_agent=3,
|
||||
completed_tasks=9, total_tasks=10,
|
||||
test_pass_rate=70, code_review_score=60, user_acceptance=65,
|
||||
budget_usd=10, actual_cost_usd=6, model_tokens=42000,
|
||||
target_time_s=600, actual_time_s=720, recovered_failures=1, total_failures=2,
|
||||
)
|
||||
|
||||
# Quality: 0.4*90+0.3*80+0.3*85 = 85.5 ; 0.4*70+0.3*60+0.3*65 = 65.5
|
||||
check("Q_swarm = 85.5", quality(swarm) == 85.5)
|
||||
check("Q_sub = 65.5", quality(sub) == 65.5)
|
||||
|
||||
# DEFAULT is now symmetric C_base (Owner-ratified): C_base = n_base*(CostEff_base/100+0.5) = 3*(166.667/100+0.5) = 6.5
|
||||
r = compare(swarm, sub)
|
||||
check("default mode is symmetric", r["cost_mode"] == "symmetric" and r["c_base"] == 6.5)
|
||||
check("G_E = 20.0", r["g_e"] == 20.0)
|
||||
check("C_swarm = 12.5 (5*(200/100+0.5))", r["c_swarm"] == 12.5)
|
||||
check("G_E,c symmetric = (85.5/12.5)-(65.5/6.5)", round(r["g_e_cost"], 4) == round((85.5 / 12.5) - (65.5 / 6.5), 4))
|
||||
check("base_coefficient(sub) = 0.90", r["base_coefficient"] == 0.90)
|
||||
check("raw gain positive", r["raw_gain_positive"] is True)
|
||||
check("cost-normalized negative (5 agents not worth it here)", r["cost_normalized_positive"] is False)
|
||||
|
||||
ev = evaluate(swarm, [sub])
|
||||
check("swarm_valid False (fails cost-normalized)", ev["swarm_valid"] is False)
|
||||
|
||||
# Literal v2.0 (reference only): C_base = 1.0
|
||||
rl = compare(swarm, sub, symmetric_cost=False)
|
||||
check("literal mode label + C_base=1.0", rl["cost_mode"] == "literal_v2.0" and rl["c_base"] == 1.0)
|
||||
check("literal G_E,c = (85.5/12.5)-65.5", round(rl["g_e_cost"], 4) == round((85.5 / 12.5) - 65.5, 4))
|
||||
|
||||
um = unified_metrics(sub)
|
||||
check("unified_metrics keys", set(um.keys()) == {"completion", "quality", "cost_efficiency", "speed", "robustness"})
|
||||
check("unified completion 90", um["completion"] == 90.0)
|
||||
|
||||
# from_dict validation
|
||||
ok = BenchmarkRunRecord.from_dict({f: getattr(sub, f) for f in BenchmarkRunRecord.__dataclass_fields__})
|
||||
check("from_dict round-trips", ok == sub)
|
||||
try:
|
||||
BenchmarkRunRecord.from_dict({"system": "sub"})
|
||||
check("from_dict rejects missing fields", False)
|
||||
except ValueError:
|
||||
check("from_dict rejects missing fields", True)
|
||||
try:
|
||||
compare(sub, swarm) # first arg must be system='swarm'
|
||||
check("compare requires swarm record first", False)
|
||||
except ValueError:
|
||||
check("compare requires swarm record first", True)
|
||||
try:
|
||||
bad = BenchmarkRunRecord.from_dict({f: getattr(sub, f) for f in BenchmarkRunRecord.__dataclass_fields__} | {"task_set_id": "other"})
|
||||
compare(swarm, bad)
|
||||
check("compare requires same task_set_id", False)
|
||||
except ValueError:
|
||||
check("compare requires same task_set_id", True)
|
||||
|
||||
print()
|
||||
if failures:
|
||||
print(f"{len(failures)} comparison check(s) FAILED: {failures}")
|
||||
sys.exit(1)
|
||||
print("all baseline comparison checks passed (v2.0)")
|
||||
@@ -0,0 +1,95 @@
|
||||
"""Test SwarmRunMetricsCollector against a synthetic run with known states.
|
||||
|
||||
Hermetic (REDIS_FAKE). Run from agent_swarm_v6: python scripts/test-benchmark-collector.py
|
||||
"""
|
||||
import asyncio
|
||||
import json
|
||||
import math
|
||||
import os
|
||||
import sys
|
||||
from pathlib import Path
|
||||
|
||||
os.environ["REDIS_FAKE"] = "1"
|
||||
sys.path.insert(0, str(Path(__file__).resolve().parents[1]))
|
||||
|
||||
from orchestrator.redis_client import redis_client
|
||||
from orchestrator import swarm_runtime as sr_mod
|
||||
from orchestrator.swarm_runtime import swarm_runtime
|
||||
from orchestrator.task_queue import task_queue, TaskStatus
|
||||
from benchmark.collectors.run_collector import SwarmRunMetricsCollector
|
||||
|
||||
failures = []
|
||||
|
||||
|
||||
def check(name, cond):
|
||||
print(("PASS" if cond else "FAIL"), "-", name)
|
||||
if not cond:
|
||||
failures.append(name)
|
||||
|
||||
|
||||
async def _noop(self, *a, **k):
|
||||
return None
|
||||
|
||||
|
||||
async def add_task(run, task_id, *, status, agent, cost=0.0, retry=0, depends_on=None):
|
||||
t = await task_queue.create_task(task_id=task_id, description=task_id, agent_role=task_id.split("-")[-1],
|
||||
depends_on=depends_on or [], enqueue=False)
|
||||
t.status = status
|
||||
t.assigned_agent_id = agent
|
||||
t.retry_count = retry
|
||||
t.result = json.dumps({"usage": {"model_cost_usd": cost}})
|
||||
await task_queue._save_task(t)
|
||||
await swarm_runtime.attach_task(run, t.task_id)
|
||||
return t
|
||||
|
||||
|
||||
async def main():
|
||||
await redis_client.connect()
|
||||
sr_mod.SwarmRuntime._post_callback = _noop # no real HTTP
|
||||
|
||||
body = {
|
||||
"mode": "swarm",
|
||||
"requirement": {"objective": "collector test"},
|
||||
"orchestration_plan": {"budget": {"max_cost_usd": 10}},
|
||||
"callback": {"url": "http://127.0.0.1:9/cb", "subscribed_events": []},
|
||||
"metadata": {"manager_deployment_id": "m-col"},
|
||||
}
|
||||
run, _ = await swarm_runtime.get_or_create_run(body=body, idempotency_key=None, correlation_id="c")
|
||||
|
||||
await add_task(run, "t-implementation", status=TaskStatus.COMPLETED, agent="A", cost=2.0)
|
||||
await add_task(run, "t-testing", status=TaskStatus.COMPLETED, agent="B", cost=3.0, retry=1,
|
||||
depends_on=["t-implementation"])
|
||||
await add_task(run, "t-documentation", status=TaskStatus.FAILED, agent="A", cost=0.0, retry=3)
|
||||
|
||||
await swarm_runtime.emit_event(run, "handoff.requested", payload={"task_id": "t1", "from_role": "implementation", "to_role": "testing"})
|
||||
await swarm_runtime.emit_event(run, "handoff.completed", payload={"task_id": "t1", "from_role": "implementation", "to_role": "testing"})
|
||||
|
||||
collector = SwarmRunMetricsCollector(run.swarm_id)
|
||||
metrics = await collector.collect()
|
||||
cov = collector.coverage
|
||||
|
||||
# s_completion = 2/3*100
|
||||
check("s_completion = 66.67", round(metrics.s_completion, 2) == 66.67 and cov["s_completion"])
|
||||
# s_collaboration = 0.5*100(handoff) + 0.3*100(dep resolved) + 0.2*50(workload A:2,B:1) = 90
|
||||
check("s_collaboration = 90.0", round(metrics.s_collaboration, 1) == 90.0 and cov["s_collaboration"])
|
||||
# s_robustness: failures={t-testing(retry),t-documentation(failed)}=2, recovered={t-testing completed}=1 => 50
|
||||
check("s_robustness = 50.0", round(metrics.s_robustness, 1) == 50.0 and cov["s_robustness"])
|
||||
# s_cost = 100*budget(10)/actual(5) = 200
|
||||
check("s_cost = 200.0", round(metrics.s_cost, 1) == 200.0 and cov["s_cost"])
|
||||
# governance: no approvals -> NaN, coverage False
|
||||
check("s_governance NaN + coverage False", math.isnan(metrics.s_governance) and cov["s_governance"] is False)
|
||||
# not-yet-collectable -> NaN + coverage False
|
||||
check("uncollectable metrics NaN + coverage False",
|
||||
all(math.isnan(getattr(metrics, k)) and cov[k] is False
|
||||
for k in ("tau", "eta", "p_decision", "reward", "s_gain", "s_communication",
|
||||
"s_swarm", "g_e", "g_e_cost", "benchmark")))
|
||||
|
||||
print()
|
||||
if failures:
|
||||
print(f"{len(failures)} collector check(s) FAILED: {failures}")
|
||||
sys.exit(1)
|
||||
print("all benchmark collector checks passed")
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
asyncio.run(main())
|
||||
@@ -0,0 +1,72 @@
|
||||
"""Unit tests for benchmark metric formulas — Agent 蜂群指标量化与标准 v2.0.
|
||||
|
||||
Run from agent_swarm_v6: python scripts/test-benchmark-metrics.py
|
||||
"""
|
||||
import sys
|
||||
from pathlib import Path
|
||||
|
||||
sys.path.insert(0, str(Path(__file__).resolve().parents[1]))
|
||||
|
||||
import benchmark.metrics as m
|
||||
|
||||
failures = []
|
||||
|
||||
|
||||
def check(name, cond):
|
||||
print(("PASS" if cond else "FAIL"), "-", name)
|
||||
if not cond:
|
||||
failures.append(name)
|
||||
|
||||
|
||||
# --- weight tables sum/shape (v2.0) ---
|
||||
check("Σλ = 1.0", abs(sum(m.LAMBDA_WEIGHTS.values()) - 1.0) < 1e-9)
|
||||
check("τ weights sum = 1.0", abs(sum(m.TAU_WEIGHTS.values()) - 1.0) < 1e-9)
|
||||
check("η weights sum = 1.0", abs(sum(m.ETA_WEIGHTS.values()) - 1.0) < 1e-9)
|
||||
check("reward weights sum = 1.0", abs(sum(m.REWARD_WEIGHTS.values()) - 1.0) < 1e-9)
|
||||
check("S_swarm weights sum = 1.0", abs(sum(m.SWARM_WEIGHTS.values()) - 1.0) < 1e-9)
|
||||
check("η includes Confidence (v2.0)", "confidence" in m.ETA_WEIGHTS)
|
||||
|
||||
# --- decision layer ---
|
||||
# pheromone with all=1 except penalties=0 -> sum of positive weights 0.25+0.20+0.20 = 0.65
|
||||
check("pheromone positive-only", round(m.pheromone(success=1, quality=1, acceptance=1, cost=0, time=0, risk=0, rollback=0), 4) == 0.65)
|
||||
check("heuristic positive-only", round(m.heuristic(match=1, urgency=1, dependency=1, resource=1, confidence=1, risk=0, budget_pressure=0), 4) == 0.80)
|
||||
# p_decision = τ^α·η^β·100 ; τ=0.5, η=0.5, α=1, β=2 -> 0.5*0.25*100 = 12.5
|
||||
check("p_decision = τ^α·η^β·100", round(m.p_decision(0.5, 0.5, 1.0, 2.0), 4) == 12.5)
|
||||
# action_probability normalizes
|
||||
ap = m.action_probability(2.0, 2.0, 1.0, 1.0, [(2.0, 2.0), (1.0, 1.0)])
|
||||
check("action_probability normalized", round(ap, 4) == round(4.0 / 5.0, 4))
|
||||
|
||||
# --- execution layer ---
|
||||
# reward with all components=100, penalties=0 -> sum of positive weights *100 = (0.20+0.20+0.12+0.13+0.13+0.10)*100 = 88
|
||||
check("reward positive-only = 88.0",
|
||||
round(m.reward(s_task=100, q_quality=100, v_speed=100, e_cost=100, r_robust=100, g_gov=100, p_risk=0, p_rework=0), 4) == 88.0)
|
||||
check("quality_score", m.quality_score(100, 100, 100) == 100.0)
|
||||
|
||||
# --- emergence + cost-normalized (v2.0 CHANGED to difference + dynamic C_swarm) ---
|
||||
check("emergence_gain", m.emergence_gain(90, 80) == 10)
|
||||
# C_swarm = N_agent*(CostEfficiency/100 + 0.5); N=4, CE=50 -> 4*(0.5+0.5)=4.0
|
||||
check("swarm_cost dynamic", m.swarm_cost(4, 50) == 4.0)
|
||||
# G_E,c = (Q_swarm/C_swarm) - (Q_base/C_base); (80/4)-(80/1)... use (80/4)-(20/1)=20-20=0
|
||||
check("cost_normalized_gain is a difference", m.cost_normalized_gain(80, 4.0, 20, 1.0) == 0.0)
|
||||
|
||||
# --- benchmark_agent: Σλ enforcement ---
|
||||
check("benchmark_agent uses default Σλ=1 weights",
|
||||
round(m.benchmark_agent(s_swarm=10, g_e=0, reward=0, observability=0, governance=0), 4) == round(0.30 * 10, 4))
|
||||
try:
|
||||
m.benchmark_agent(s_swarm=1, g_e=1, reward=1, observability=1, governance=1,
|
||||
weights={"lambda1": 0.5, "lambda2": 0.5, "lambda3": 0.5, "lambda4": 0.5, "lambda5": 0.5})
|
||||
check("benchmark_agent rejects Σλ != 1.0", False)
|
||||
except ValueError:
|
||||
check("benchmark_agent rejects Σλ != 1.0", True)
|
||||
|
||||
# --- dataclass shape (v2.0) ---
|
||||
expected_fields = {"tau", "eta", "p_decision", "reward", "s_completion", "s_gain", "s_collaboration",
|
||||
"s_communication", "s_cost", "s_robustness", "s_governance", "s_swarm",
|
||||
"g_e", "g_e_cost", "benchmark"}
|
||||
check("SwarmMetrics has v2.0 fields", set(m.SwarmMetrics.__dataclass_fields__.keys()) == expected_fields)
|
||||
|
||||
print()
|
||||
if failures:
|
||||
print(f"{len(failures)} formula check(s) FAILED: {failures}")
|
||||
sys.exit(1)
|
||||
print("all benchmark metric formula checks passed (v2.0)")
|
||||
@@ -0,0 +1,137 @@
|
||||
"""Manager event-contract test.
|
||||
|
||||
Validates that Swarm's emitted callback events conform to the Heicode Manager handler
|
||||
(`heicode/controller/agent_callback.go`):
|
||||
- each event_type carries HM's required payload fields,
|
||||
- the HMAC signing canonical string / headers match HM's verification.
|
||||
|
||||
Hermetic: REDIS_FAKE in-memory store; callbacks are captured in-process (not sent).
|
||||
Run from agent_swarm_v6: python scripts/test-contract-events.py
|
||||
"""
|
||||
import asyncio
|
||||
import hashlib
|
||||
import hmac
|
||||
import json
|
||||
import os
|
||||
import sys
|
||||
from pathlib import Path
|
||||
|
||||
os.environ["REDIS_FAKE"] = "1"
|
||||
os.environ["AGENT_CALLBACK_SIGNING_SECRET"] = "test-signing-secret"
|
||||
sys.path.insert(0, str(Path(__file__).resolve().parents[1]))
|
||||
|
||||
from orchestrator.redis_client import redis_client
|
||||
from orchestrator import swarm_runtime as sr_mod
|
||||
from orchestrator.swarm_runtime import swarm_runtime
|
||||
|
||||
# HM required payload fields per event_type (agent_callback.go registry).
|
||||
REQUIRED = {
|
||||
"deployment.status_changed": ["status"],
|
||||
"task.created": ["task_id", "title"],
|
||||
"task.claimed": ["task_id", "agent_role"],
|
||||
"task.running": ["task_id", "agent_role"],
|
||||
"task.heartbeat": ["task_id", "agent_role"],
|
||||
"task.blocked": ["task_id", "reason"],
|
||||
"task.retried": ["task_id", "attempt"],
|
||||
"task.released": ["task_id", "agent_role"],
|
||||
"task.failed": ["task_id", "reason"],
|
||||
"task.completed": ["task_id"],
|
||||
"handoff.requested": ["task_id", "from_role", "to_role"],
|
||||
"handoff.completed": ["task_id", "from_role", "to_role"],
|
||||
"approval.requested": ["approval_id", "operation", "risk_level"],
|
||||
"artifact.created": ["artifact_id"],
|
||||
"timeline.updated": ["title"],
|
||||
"budget.alert": ["threshold_pct"],
|
||||
}
|
||||
|
||||
SECRET = os.environ["AGENT_CALLBACK_SIGNING_SECRET"]
|
||||
captured = [] # (raw_body, headers)
|
||||
failures = []
|
||||
|
||||
|
||||
def check(name, cond):
|
||||
print(("PASS" if cond else "FAIL"), "-", name)
|
||||
if not cond:
|
||||
failures.append(name)
|
||||
|
||||
|
||||
def validate_envelope(raw_body: str, headers: dict) -> None:
|
||||
env = json.loads(raw_body)
|
||||
et = env["event_type"]
|
||||
# required fields
|
||||
req = REQUIRED.get(et, [])
|
||||
payload = env.get("payload") or {}
|
||||
missing = [f for f in req if f not in payload or payload.get(f) in (None, "")]
|
||||
check(f"{et}: required fields present {req}", not missing)
|
||||
# HMAC canonical string == ts + "." + event_id + "." + raw_body
|
||||
ts = headers.get("X-Agent-Timestamp")
|
||||
sig = headers.get("X-Agent-Signature", "")
|
||||
eid = headers.get("X-Agent-Event-Id")
|
||||
expect = "sha256=" + hmac.new(SECRET.encode(), f"{ts}.{eid}.{raw_body}".encode(), hashlib.sha256).hexdigest()
|
||||
check(f"{et}: HMAC signature matches HM canonical string", sig == expect)
|
||||
check(f"{et}: event_id header == body event_id", eid == env.get("event_id"))
|
||||
|
||||
|
||||
async def _capture(self, swarm_id, url, raw_body, headers, event_type, event_id):
|
||||
captured.append((raw_body, headers))
|
||||
|
||||
|
||||
async def emit(run, event_type, **payload):
|
||||
captured.clear()
|
||||
await swarm_runtime.emit_event(run, event_type, payload=payload)
|
||||
await asyncio.sleep(0) # let the create_task'd callback run
|
||||
assert captured, f"no callback captured for {event_type}"
|
||||
return captured[-1]
|
||||
|
||||
|
||||
async def main():
|
||||
await redis_client.connect()
|
||||
sr_mod.SwarmRuntime._post_callback = _capture # capture instead of HTTP POST
|
||||
|
||||
body = {
|
||||
"mode": "swarm",
|
||||
"requirement": {"objective": "contract test"},
|
||||
"callback": {"url": "http://127.0.0.1:9/cb", "subscribed_events": []},
|
||||
"metadata": {"manager_deployment_id": "m-contract"},
|
||||
}
|
||||
run, _ = await swarm_runtime.get_or_create_run(body=body, idempotency_key=None, correlation_id="corr-contract")
|
||||
|
||||
# 1) centralized normalization (the 2 emit_event fixes)
|
||||
raw, hdr = await emit(run, "timeline.updated", summary="did the thing")
|
||||
check("timeline.updated normalized to include title", json.loads(raw)["payload"].get("title") == "did the thing")
|
||||
validate_envelope(raw, hdr)
|
||||
|
||||
raw, hdr = await emit(run, "budget.alert", threshold=0.8)
|
||||
check("budget.alert normalized to threshold_pct=80", json.loads(raw)["payload"].get("threshold_pct") == 80.0)
|
||||
validate_envelope(raw, hdr)
|
||||
|
||||
# 2) the handler-built payloads for the other two fixed types
|
||||
raw, hdr = await emit(run, "handoff.completed", task_id="t1", from_role="implementation", to_role="testing")
|
||||
validate_envelope(raw, hdr)
|
||||
raw, hdr = await emit(run, "task.released", task_id="t1", agent_role="testing", reason="at_capacity")
|
||||
validate_envelope(raw, hdr)
|
||||
|
||||
# 3) representative coverage of the common event types
|
||||
samples = {
|
||||
"deployment.status_changed": {"status": "running"},
|
||||
"task.created": {"task_id": "t1", "title": "impl"},
|
||||
"task.claimed": {"task_id": "t1", "agent_role": "implementation"},
|
||||
"task.completed": {"task_id": "t1"},
|
||||
"task.failed": {"task_id": "t1", "reason": "boom"},
|
||||
"task.retried": {"task_id": "t1", "attempt": 1},
|
||||
"approval.requested": {"approval_id": "ap1", "operation": "git.write", "risk_level": "high"},
|
||||
"artifact.created": {"artifact_id": "art1"},
|
||||
}
|
||||
for et, pl in samples.items():
|
||||
raw, hdr = await emit(run, et, **pl)
|
||||
validate_envelope(raw, hdr)
|
||||
|
||||
print()
|
||||
if failures:
|
||||
print(f"{len(failures)} contract check(s) FAILED: {failures}")
|
||||
sys.exit(1)
|
||||
print("all event-contract checks passed")
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
asyncio.run(main())
|
||||
@@ -113,10 +113,11 @@ async def test_agent_peer_routing():
|
||||
"type": "peer_message",
|
||||
"from_agent_id": "agent-impl",
|
||||
"task_id": "t1",
|
||||
"content": "please advise",
|
||||
"content": "", # empty -> deterministic fallback reply (no model needed)
|
||||
"correlation_id": "corr-1",
|
||||
"is_reply": False,
|
||||
})
|
||||
await asyncio.sleep(0.05) # answer_peer_query is scheduled as a background task
|
||||
check("peer query produces a reply", len(sent) == 1 and sent[0]["is_reply"] is True)
|
||||
check("peer reply targets requester", sent[0]["target_agent_id"] == "agent-impl")
|
||||
|
||||
@@ -239,8 +240,26 @@ async def test_agent_peer_shares_summary():
|
||||
sent.append(payload)
|
||||
|
||||
a.safe_send = fake_send
|
||||
|
||||
# Fallback path: no query content -> cached summary (no model needed).
|
||||
await a.answer_peer_query({"from_agent_id": "agent-test", "correlation_id": "c", "is_reply": False})
|
||||
check("peer reply shares last summary", "implemented add()" in sent[0]["content"])
|
||||
check("peer reply falls back to last summary", "implemented add()" in sent[0]["content"])
|
||||
|
||||
# Substantive path: a stubbed executor returns a grounded, structured reply.
|
||||
class _FakeExec:
|
||||
async def peer_reply(self, *, query, capabilities, last_summary, max_tokens=None):
|
||||
return {"content": f"grounded answer to: {query}", "stance": "agree",
|
||||
"evidence": "src", "refs": ["a.py"]}
|
||||
|
||||
a._peer_executor = _FakeExec()
|
||||
sent.clear()
|
||||
await a.answer_peer_query({
|
||||
"from_agent_id": "agent-test", "correlation_id": "c2", "task_id": "t",
|
||||
"content": "does add() raise ValueError?", "is_reply": False,
|
||||
})
|
||||
check("substantive peer reply uses executor content",
|
||||
sent[0]["content"] == "grounded answer to: does add() raise ValueError?"
|
||||
and sent[0]["stance"] == "agree" and sent[0]["refs"] == ["a.py"])
|
||||
|
||||
|
||||
async def main():
|
||||
|
||||
Reference in New Issue
Block a user