Songhaoz666andClaude Opus 4.8
d487923646
四块互相交织的 benchmark 覆盖增量，统一提交： 1) 通信遥测（#23）：orchestrator 路由 peer 消息时按 correlation_id 计请求/应答到 SwarmRun.collaboration（内部状态，不进 Manager 事件流）；collector 算 s_communication。 治理计数由 run.approvals 派生（合规/总数）→ s_governance。 2) Q_quality 掩码归一（v2.1 裁定）：metrics.quality_score 改为对 present 输入加权归一， 非编码任务自动忽略 TestPassRate，全缺 → NaN（不伪造）。 3) 质量插桩 / Group B：新增 Pod 内代码测试沙箱（orchestrator/sandbox.py，环境清洗 + 超时强杀 + 资源限额 + 路径越界校验，门控 ENABLE_QUALITY_EVAL）与 held-out fixture （benchmark/fixtures/）；run 完成时用留出测试评分得 TestPassRate → Q_quality → collector 合成 reward。安全边界见 docs/integration/security-boundary.md §8.1。 4) 决策引擎 / Group A（#10，Option A score-at-pull）：新增 orchestrator/decision_engine.py —— 信息素 τ（Redis 持久、(role,agent) 键控、冷启动 0.5、ρ 蒸发、夹紧、学习常开）+ η 启发式评分 + ε-greedy 概率采样；每次 dispatch 产一条 DecisionTrace → SwarmRun.decisions；collector 算 tau/eta/p_decision。概率选择门控 ENABLE_ACO_DISPATCH （默认关，CI 用 ACO_SEED 固定）。 覆盖：单次 run 真实可算字段由 4 提升至最多 10/15（新增 communication/reward/tau/eta/ p_decision，外加 governance 有条件）。 测试：新增 test-sandbox / test-quality / test-decision-engine；扩充 collector/metrics 用例； CI 纳入全部 benchmark 套件 + flag-on 的 ACO e2e。本地 11 项 gate 全绿。 诚实边界（未越界声称）： - Group A 为单边匹配（Option B 待 Group C）；概率派发优于贪心未证；默认关闭。 - reward 的 CodeReview/UserAcceptance 未采集（掩码忽略）；P_risk 为审批派生低估。 - s_gain/s_swarm/g_e/g_e_cost/benchmark 仍 NaN —— 需基线（#21/#13），本 PR 不动验收。 影响范围：Swarm（orchestrator + benchmark + docs + CI）。不改 Manager↔Swarm 事件契约 （遥测均为运行时内部状态）；不影响 Client/计费/密钥/发布链路。新增 ENABLE_QUALITY_EVAL / ENABLE_ACO_DISPATCH 两个开关，默认关闭。 Closes #10 Closes #23 Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
83 lines
2.9 KiB
Python
83 lines
2.9 KiB
Python
"""Test the in-pod code sandbox: real subprocess execution, counting, timeout, isolation.
|
|
|
|
Hermetic, no model key. Run from agent_swarm_v6: python scripts/test-sandbox.py
|
|
"""
|
|
import sys
|
|
from pathlib import Path
|
|
|
|
sys.path.insert(0, str(Path(__file__).resolve().parents[1]))
|
|
|
|
from orchestrator.sandbox import SandboxFile, run_tests
|
|
|
|
failures = []
|
|
|
|
|
|
def check(name, cond):
|
|
print(("PASS" if cond else "FAIL"), "-", name)
|
|
if not cond:
|
|
failures.append(name)
|
|
|
|
|
|
# --- 1. bare pytest-style tests against a generated module: 2/3 pass ---
|
|
impl = [SandboxFile("calc.py", "def add(a, b):\n return a + b\n")]
|
|
tests = [SandboxFile(
|
|
"test_calc.py",
|
|
"from calc import add\n"
|
|
"def test_add_pos():\n assert add(1, 2) == 3\n"
|
|
"def test_add_zero():\n assert add(0, 0) == 0\n"
|
|
"def test_wrong():\n assert add(1, 1) == 3\n",
|
|
)]
|
|
r = run_tests(impl, tests)
|
|
check("runs bare functions: total=3", r.total == 3)
|
|
check("counts passed=2", r.passed == 2)
|
|
check("counts failed=1", r.failed == 1)
|
|
check("pass_rate ~= 66.67", r.pass_rate is not None and round(r.pass_rate, 2) == 66.67)
|
|
|
|
# --- 2. unittest.TestCase style is also collected ---
|
|
ut = [SandboxFile(
|
|
"test_ut.py",
|
|
"import unittest\n"
|
|
"class T(unittest.TestCase):\n"
|
|
" def test_ok(self):\n self.assertEqual(2 + 2, 4)\n"
|
|
" def test_bad(self):\n self.assertEqual(2 + 2, 5)\n",
|
|
)]
|
|
r2 = run_tests([], ut)
|
|
check("unittest TestCase collected: total=2", r2.total == 2)
|
|
check("unittest passed=1 failed=1", r2.passed == 1 and r2.failed == 1)
|
|
|
|
# --- 3. import error in test counts as errored, not a crash ---
|
|
bad = [SandboxFile("test_imp.py", "import this_module_does_not_exist_xyz\n")]
|
|
r3 = run_tests([], bad)
|
|
check("import error -> errored>=1, no crash", r3.errored >= 1 and r3.error is None)
|
|
|
|
# --- 4. timeout is enforced and reported (no hang) ---
|
|
loop = [SandboxFile("test_loop.py", "def test_spin():\n while True:\n pass\n")]
|
|
r4 = run_tests([], loop, timeout_seconds=3)
|
|
check("infinite loop times out", r4.timed_out is True)
|
|
|
|
# --- 5. secrets are NOT visible to sandboxed code (env scrub) ---
|
|
import os as _os
|
|
_os.environ["OPENAI_API_KEY"] = "sk-should-not-leak"
|
|
_os.environ["AWS_SECRET_ACCESS_KEY"] = "should-not-leak"
|
|
leak = [SandboxFile(
|
|
"test_leak.py",
|
|
"import os\n"
|
|
"def test_no_openai():\n assert os.environ.get('OPENAI_API_KEY') is None\n"
|
|
"def test_no_aws():\n assert os.environ.get('AWS_SECRET_ACCESS_KEY') is None\n",
|
|
)]
|
|
r5 = run_tests([], leak)
|
|
check("env scrub: no secrets leak into sandbox", r5.total == 2 and r5.passed == 2)
|
|
|
|
# --- 6. path traversal is rejected ---
|
|
try:
|
|
run_tests([SandboxFile("../escape.py", "x=1")], [])
|
|
check("path traversal rejected", False)
|
|
except ValueError:
|
|
check("path traversal rejected", True)
|
|
|
|
print()
|
|
if failures:
|
|
print(f"{len(failures)} sandbox check(s) FAILED: {failures}")
|
|
sys.exit(1)
|
|
print("all sandbox checks passed")
|