Songhaoz666andClaude Opus 4.8
baa67350e6
在去中心化重构之上落地 benchmark 对比管线：5 个系统（single/strong/chain/sub_agent/swarm） 跑同一任务集、同一执行后端，产出统一 BenchmarkRunRecord → 评估器算 G_E/G_E,c → 报告 + 回放。 - benchmark/runners/：backend（Offline 确定性 / OpenAI 真实）+ base + 5 个 runner。各 runner 用 held-out fixture 测试在 Group B 沙箱里评分得 TestPassRate（权威，非自评）。 - benchmark/tasksets/：统一任务集 + 加载器（coding-set-1，1 个 fixture）。 - benchmark/reports/、benchmark/replay/：G_E/G_E,c/coverage/confidence + 归档。 - benchmark/baselines/comparison.py：BenchmarkRunRecord 的 CodeReview/UserAcceptance 改为 Optional（掩码归一，未采集即 None，规则 #9）。 - scripts/run-benchmark-suite.py harness + scripts/test-benchmark-runners.py。 与去中心化重构对齐：swarm runner 拓扑已**重指向去中心化流程**（种子→自选→自主分解→竞争→ 同伴交叉评审→收敛，calls=6/review=1），非旧 Master「分解→派发→单评审」。仍用同一离线后端 建模以保证公平对比（驱动活体编排器会换后端→记录不可比；活体全流程由 test-workflow-e2e 验证）。 沙箱适配：runner 评分走 fail-closed 沙箱（#24），故 test + CI 步骤设 HEICODE_SANDBOX_ISOLATED=1 （仅 CI/隔离 Pod）。 影响范围：agent_swarm（benchmark 层 + 测试 + docs + CI）。不碰 orchestrator 编排逻辑、 不改 Manager↔Swarm 契约、不影响 Client/计费/密钥/审计/发布链路。 诚实边界： - **离线后端只验证管线**：所有系统拿同一参考解 → quality 相同 → G_E=0、swarm_valid=False， 刻意不显示蜂群优势（反造假）。真实 G_E>0 需 --backend openai + 足量冻结任务集 + 多次运行。 - 故 Closes #21（运行器 + 统一记录已落地并产出合规非 NaN 记录）；Refs #20（仅 1/5 场景）、 Refs #22（评估器/报告/回放已建，但 Quality 仅 TestPassRate，CodeReview/UserAcceptance 缺）、 Refs #13（验收 EPIC，需真实 run 证明 Swarm>baselines，未满足）。 Closes #21 Refs #20 Refs #22 Refs #13 Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
69 lines
2.4 KiB
Python
69 lines
2.4 KiB
Python
"""Unified benchmark task sets (#20) — the shared, frozen tasks every system runs.
|
|
|
|
Fairness rule (standard §9.1 / baseline-comparison §3): swarm and every baseline run the SAME
|
|
task_set_id under the SAME model gateway. A task set groups tasks of one scenario; each task
|
|
references a fixture (objective + HELD-OUT acceptance tests + offline reference solution).
|
|
|
|
Layout: benchmark/tasksets/<task_set_id>/taskset.json
|
|
|
|
`taskset.json`:
|
|
{
|
|
"task_set_id": "coding-set-1",
|
|
"scenario": "coding", # coding | refactoring | architecture | devops | bugfix
|
|
"tasks": [ { "id": "add_function", "fixture": "add_function" } ]
|
|
}
|
|
|
|
Coverage is honest: only scenarios with real fixtures are loadable. The other four scenarios
|
|
(refactoring/architecture/devops/bugfix) are declared TODO in this package's README until their
|
|
fixtures + held-out tests exist — we do not ship empty task sets that would silently pass.
|
|
"""
|
|
from __future__ import annotations
|
|
|
|
import json
|
|
from dataclasses import dataclass, field
|
|
from pathlib import Path
|
|
from typing import List
|
|
|
|
from ..fixtures import Fixture, load_fixture
|
|
|
|
_TASKSET_ROOT = Path(__file__).resolve().parent
|
|
SCENARIOS = {"coding", "refactoring", "architecture", "devops", "bugfix"}
|
|
|
|
|
|
@dataclass
|
|
class TaskSetItem:
|
|
id: str
|
|
fixture_id: str
|
|
|
|
@property
|
|
def fixture(self) -> Fixture:
|
|
return load_fixture(self.fixture_id)
|
|
|
|
|
|
@dataclass
|
|
class TaskSet:
|
|
task_set_id: str
|
|
scenario: str
|
|
tasks: List[TaskSetItem] = field(default_factory=list)
|
|
|
|
|
|
def available_tasksets() -> List[str]:
|
|
return sorted(
|
|
p.name for p in _TASKSET_ROOT.iterdir()
|
|
if p.is_dir() and (p / "taskset.json").exists()
|
|
)
|
|
|
|
|
|
def load_taskset(task_set_id: str) -> TaskSet:
|
|
meta_path = _TASKSET_ROOT / task_set_id / "taskset.json"
|
|
if not meta_path.exists():
|
|
raise FileNotFoundError(f"unknown task set: {task_set_id}")
|
|
meta = json.loads(meta_path.read_text(encoding="utf-8"))
|
|
scenario = meta.get("scenario", "")
|
|
if scenario not in SCENARIOS:
|
|
raise ValueError(f"task set {task_set_id} has unknown scenario '{scenario}'")
|
|
tasks = [TaskSetItem(id=t["id"], fixture_id=t["fixture"]) for t in meta.get("tasks", [])]
|
|
if not tasks:
|
|
raise ValueError(f"task set {task_set_id} has no tasks (refusing to ship an empty set)")
|
|
return TaskSet(task_set_id=meta["task_set_id"], scenario=scenario, tasks=tasks)
|