Songhaoz666andClaude Opus 4.8
d487923646
四块互相交织的 benchmark 覆盖增量，统一提交： 1) 通信遥测（#23）：orchestrator 路由 peer 消息时按 correlation_id 计请求/应答到 SwarmRun.collaboration（内部状态，不进 Manager 事件流）；collector 算 s_communication。 治理计数由 run.approvals 派生（合规/总数）→ s_governance。 2) Q_quality 掩码归一（v2.1 裁定）：metrics.quality_score 改为对 present 输入加权归一， 非编码任务自动忽略 TestPassRate，全缺 → NaN（不伪造）。 3) 质量插桩 / Group B：新增 Pod 内代码测试沙箱（orchestrator/sandbox.py，环境清洗 + 超时强杀 + 资源限额 + 路径越界校验，门控 ENABLE_QUALITY_EVAL）与 held-out fixture （benchmark/fixtures/）；run 完成时用留出测试评分得 TestPassRate → Q_quality → collector 合成 reward。安全边界见 docs/integration/security-boundary.md §8.1。 4) 决策引擎 / Group A（#10，Option A score-at-pull）：新增 orchestrator/decision_engine.py —— 信息素 τ（Redis 持久、(role,agent) 键控、冷启动 0.5、ρ 蒸发、夹紧、学习常开）+ η 启发式评分 + ε-greedy 概率采样；每次 dispatch 产一条 DecisionTrace → SwarmRun.decisions；collector 算 tau/eta/p_decision。概率选择门控 ENABLE_ACO_DISPATCH （默认关，CI 用 ACO_SEED 固定）。 覆盖：单次 run 真实可算字段由 4 提升至最多 10/15（新增 communication/reward/tau/eta/ p_decision，外加 governance 有条件）。 测试：新增 test-sandbox / test-quality / test-decision-engine；扩充 collector/metrics 用例； CI 纳入全部 benchmark 套件 + flag-on 的 ACO e2e。本地 11 项 gate 全绿。 诚实边界（未越界声称）： - Group A 为单边匹配（Option B 待 Group C）；概率派发优于贪心未证；默认关闭。 - reward 的 CodeReview/UserAcceptance 未采集（掩码忽略）；P_risk 为审批派生低估。 - s_gain/s_swarm/g_e/g_e_cost/benchmark 仍 NaN —— 需基线（#21/#13），本 PR 不动验收。 影响范围：Swarm（orchestrator + benchmark + docs + CI）。不改 Manager↔Swarm 事件契约 （遥测均为运行时内部状态）；不影响 Client/计费/密钥/发布链路。新增 ENABLE_QUALITY_EVAL / ENABLE_ACO_DISPATCH 两个开关，默认关闭。 Closes #10 Closes #23 Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
89 lines
3.1 KiB
Python
89 lines
3.1 KiB
Python
"""Benchmark task fixtures with HELD-OUT acceptance tests.
|
|
|
|
A fixture pairs a task objective with authoritative tests that the swarm never sees. These tests
|
|
— not the swarm's own testing-agent output — are what produce TestPassRate (Owner ruling: avoid
|
|
self-grading; the swarm's own tests are kept as a separate signal in quality.py).
|
|
|
|
Layout per fixture: benchmark/fixtures/<id>/fixture.json + <id>/tests/test_*.py
|
|
|
|
`fixture.json` schema:
|
|
{
|
|
"id": "add_function",
|
|
"objective": "...", # what the swarm is asked to build
|
|
"required_capabilities": [...], # used to decide expects_code()
|
|
"entrypoint": "calc.py", # the module the tests import (informational)
|
|
"target_time_seconds": 60, # V_speed target for reward()
|
|
"test_files": ["tests/test_add.py"] # held-out acceptance tests (relative to <id>/)
|
|
}
|
|
"""
|
|
from __future__ import annotations
|
|
|
|
import json
|
|
from dataclasses import dataclass, field
|
|
from pathlib import Path
|
|
from typing import List, Optional
|
|
|
|
_FIXTURE_ROOT = Path(__file__).resolve().parent
|
|
|
|
# Capabilities that imply the deliverable is runnable/testable code → TestPassRate applies.
|
|
CODE_CAPABILITIES = {"python", "code_generation", "coding", "testing", "pytest", "implementation"}
|
|
|
|
|
|
@dataclass
|
|
class FixtureTest:
|
|
path: str
|
|
content: str
|
|
|
|
|
|
@dataclass
|
|
class Fixture:
|
|
id: str
|
|
objective: str
|
|
required_capabilities: List[str] = field(default_factory=list)
|
|
entrypoint: Optional[str] = None
|
|
target_time_seconds: Optional[float] = None
|
|
test_files: List[FixtureTest] = field(default_factory=list)
|
|
|
|
@property
|
|
def expects_code(self) -> bool:
|
|
return expects_code(self.required_capabilities)
|
|
|
|
|
|
def expects_code(required_capabilities) -> bool:
|
|
"""A task is code-shaped (TestPassRate applies) if it requires any coding capability."""
|
|
return bool(CODE_CAPABILITIES & {str(c).lower() for c in (required_capabilities or [])})
|
|
|
|
|
|
def fixture_dir(fixture_id: str) -> Path:
|
|
return _FIXTURE_ROOT / fixture_id
|
|
|
|
|
|
def available_fixtures() -> List[str]:
|
|
return sorted(
|
|
p.name for p in _FIXTURE_ROOT.iterdir()
|
|
if p.is_dir() and (p / "fixture.json").exists()
|
|
)
|
|
|
|
|
|
def load_fixture(fixture_id: str) -> Fixture:
|
|
base = fixture_dir(fixture_id)
|
|
meta_path = base / "fixture.json"
|
|
if not meta_path.exists():
|
|
raise FileNotFoundError(f"unknown fixture: {fixture_id}")
|
|
meta = json.loads(meta_path.read_text(encoding="utf-8"))
|
|
tests = []
|
|
for rel in meta.get("test_files", []):
|
|
tpath = base / rel
|
|
if not tpath.exists():
|
|
raise FileNotFoundError(f"fixture {fixture_id} missing test file: {rel}")
|
|
# Flatten to a basename so the sandbox runner (which scans CWD for test_*.py) finds it.
|
|
tests.append(FixtureTest(path=Path(rel).name, content=tpath.read_text(encoding="utf-8")))
|
|
return Fixture(
|
|
id=meta.get("id", fixture_id),
|
|
objective=meta.get("objective", ""),
|
|
required_capabilities=meta.get("required_capabilities", []),
|
|
entrypoint=meta.get("entrypoint"),
|
|
target_time_seconds=meta.get("target_time_seconds"),
|
|
test_files=tests,
|
|
)
|