Songhaoz666andClaude Opus 4.8
d487923646
四块互相交织的 benchmark 覆盖增量，统一提交： 1) 通信遥测（#23）：orchestrator 路由 peer 消息时按 correlation_id 计请求/应答到 SwarmRun.collaboration（内部状态，不进 Manager 事件流）；collector 算 s_communication。 治理计数由 run.approvals 派生（合规/总数）→ s_governance。 2) Q_quality 掩码归一（v2.1 裁定）：metrics.quality_score 改为对 present 输入加权归一， 非编码任务自动忽略 TestPassRate，全缺 → NaN（不伪造）。 3) 质量插桩 / Group B：新增 Pod 内代码测试沙箱（orchestrator/sandbox.py，环境清洗 + 超时强杀 + 资源限额 + 路径越界校验，门控 ENABLE_QUALITY_EVAL）与 held-out fixture （benchmark/fixtures/）；run 完成时用留出测试评分得 TestPassRate → Q_quality → collector 合成 reward。安全边界见 docs/integration/security-boundary.md §8.1。 4) 决策引擎 / Group A（#10，Option A score-at-pull）：新增 orchestrator/decision_engine.py —— 信息素 τ（Redis 持久、(role,agent) 键控、冷启动 0.5、ρ 蒸发、夹紧、学习常开）+ η 启发式评分 + ε-greedy 概率采样；每次 dispatch 产一条 DecisionTrace → SwarmRun.decisions；collector 算 tau/eta/p_decision。概率选择门控 ENABLE_ACO_DISPATCH （默认关，CI 用 ACO_SEED 固定）。 覆盖：单次 run 真实可算字段由 4 提升至最多 10/15（新增 communication/reward/tau/eta/ p_decision，外加 governance 有条件）。 测试：新增 test-sandbox / test-quality / test-decision-engine；扩充 collector/metrics 用例； CI 纳入全部 benchmark 套件 + flag-on 的 ACO e2e。本地 11 项 gate 全绿。 诚实边界（未越界声称）： - Group A 为单边匹配（Option B 待 Group C）；概率派发优于贪心未证；默认关闭。 - reward 的 CodeReview/UserAcceptance 未采集（掩码忽略）；P_risk 为审批派生低估。 - s_gain/s_swarm/g_e/g_e_cost/benchmark 仍 NaN —— 需基线（#21/#13），本 PR 不动验收。 影响范围：Swarm（orchestrator + benchmark + docs + CI）。不改 Manager↔Swarm 事件契约 （遥测均为运行时内部状态）；不影响 Client/计费/密钥/发布链路。新增 ENABLE_QUALITY_EVAL / ENABLE_ACO_DISPATCH 两个开关，默认关闭。 Closes #10 Closes #23 Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
84 lines
4.3 KiB
Python
84 lines
4.3 KiB
Python
"""Unit tests for benchmark metric formulas — Agent 蜂群指标量化与标准 v2.0.
|
||
|
||
Run from agent_swarm_v6: python scripts/test-benchmark-metrics.py
|
||
"""
|
||
import math
|
||
import sys
|
||
from pathlib import Path
|
||
|
||
sys.path.insert(0, str(Path(__file__).resolve().parents[1]))
|
||
|
||
import benchmark.metrics as m
|
||
|
||
failures = []
|
||
|
||
|
||
def check(name, cond):
|
||
print(("PASS" if cond else "FAIL"), "-", name)
|
||
if not cond:
|
||
failures.append(name)
|
||
|
||
|
||
# --- weight tables sum/shape (v2.0) ---
|
||
check("Σλ = 1.0", abs(sum(m.LAMBDA_WEIGHTS.values()) - 1.0) < 1e-9)
|
||
check("τ weights sum = 1.0", abs(sum(m.TAU_WEIGHTS.values()) - 1.0) < 1e-9)
|
||
check("η weights sum = 1.0", abs(sum(m.ETA_WEIGHTS.values()) - 1.0) < 1e-9)
|
||
check("reward weights sum = 1.0", abs(sum(m.REWARD_WEIGHTS.values()) - 1.0) < 1e-9)
|
||
check("S_swarm weights sum = 1.0", abs(sum(m.SWARM_WEIGHTS.values()) - 1.0) < 1e-9)
|
||
check("η includes Confidence (v2.0)", "confidence" in m.ETA_WEIGHTS)
|
||
|
||
# --- decision layer ---
|
||
# pheromone with all=1 except penalties=0 -> sum of positive weights 0.25+0.20+0.20 = 0.65
|
||
check("pheromone positive-only", round(m.pheromone(success=1, quality=1, acceptance=1, cost=0, time=0, risk=0, rollback=0), 4) == 0.65)
|
||
check("heuristic positive-only", round(m.heuristic(match=1, urgency=1, dependency=1, resource=1, confidence=1, risk=0, budget_pressure=0), 4) == 0.80)
|
||
# p_decision = τ^α·η^β·100 ; τ=0.5, η=0.5, α=1, β=2 -> 0.5*0.25*100 = 12.5
|
||
check("p_decision = τ^α·η^β·100", round(m.p_decision(0.5, 0.5, 1.0, 2.0), 4) == 12.5)
|
||
# action_probability normalizes
|
||
ap = m.action_probability(2.0, 2.0, 1.0, 1.0, [(2.0, 2.0), (1.0, 1.0)])
|
||
check("action_probability normalized", round(ap, 4) == round(4.0 / 5.0, 4))
|
||
|
||
# --- execution layer ---
|
||
# reward with all components=100, penalties=0 -> sum of positive weights *100 = (0.20+0.20+0.12+0.13+0.13+0.10)*100 = 88
|
||
check("reward positive-only = 88.0",
|
||
round(m.reward(s_task=100, q_quality=100, v_speed=100, e_cost=100, r_robust=100, g_gov=100, p_risk=0, p_rework=0), 4) == 88.0)
|
||
check("quality_score", m.quality_score(100, 100, 100) == 100.0)
|
||
# masked & renormalized (v2.1): all present reduces to the 0.4/0.3/0.3 blend
|
||
check("quality_score full blend = 0.4/0.3/0.3", m.quality_score(50, 100, 100) == 80.0)
|
||
# non-code task: test_pass_rate=None drops out; renorm over {review 0.3, acceptance 0.3} -> 0.5/0.5
|
||
check("quality_score non-code renormalized", m.quality_score(None, 80, 60) == 70.0)
|
||
# renormalization preserves scale: missing input never deflates a perfect score
|
||
check("quality_score stays on [0,100] when masked", m.quality_score(None, 100, 100) == 100.0)
|
||
# a single present input renormalizes to weight 1.0
|
||
check("quality_score single input", m.quality_score(None, 90, None) == 90.0)
|
||
# no applicable inputs -> NaN (rule #9: no fabricated 0)
|
||
check("quality_score all-absent = NaN", math.isnan(m.quality_score(None, None, None)))
|
||
|
||
# --- emergence + cost-normalized (v2.0 CHANGED to difference + dynamic C_swarm) ---
|
||
check("emergence_gain", m.emergence_gain(90, 80) == 10)
|
||
# C_swarm = N_agent*(CostEfficiency/100 + 0.5); N=4, CE=50 -> 4*(0.5+0.5)=4.0
|
||
check("swarm_cost dynamic", m.swarm_cost(4, 50) == 4.0)
|
||
# G_E,c = (Q_swarm/C_swarm) - (Q_base/C_base); (80/4)-(80/1)... use (80/4)-(20/1)=20-20=0
|
||
check("cost_normalized_gain is a difference", m.cost_normalized_gain(80, 4.0, 20, 1.0) == 0.0)
|
||
|
||
# --- benchmark_agent: Σλ enforcement ---
|
||
check("benchmark_agent uses default Σλ=1 weights",
|
||
round(m.benchmark_agent(s_swarm=10, g_e=0, reward=0, observability=0, governance=0), 4) == round(0.30 * 10, 4))
|
||
try:
|
||
m.benchmark_agent(s_swarm=1, g_e=1, reward=1, observability=1, governance=1,
|
||
weights={"lambda1": 0.5, "lambda2": 0.5, "lambda3": 0.5, "lambda4": 0.5, "lambda5": 0.5})
|
||
check("benchmark_agent rejects Σλ != 1.0", False)
|
||
except ValueError:
|
||
check("benchmark_agent rejects Σλ != 1.0", True)
|
||
|
||
# --- dataclass shape (v2.0) ---
|
||
expected_fields = {"tau", "eta", "p_decision", "reward", "s_completion", "s_gain", "s_collaboration",
|
||
"s_communication", "s_cost", "s_robustness", "s_governance", "s_swarm",
|
||
"g_e", "g_e_cost", "benchmark"}
|
||
check("SwarmMetrics has v2.0 fields", set(m.SwarmMetrics.__dataclass_fields__.keys()) == expected_fields)
|
||
|
||
print()
|
||
if failures:
|
||
print(f"{len(failures)} formula check(s) FAILED: {failures}")
|
||
sys.exit(1)
|
||
print("all benchmark metric formula checks passed (v2.0)")
|