From baa67350e6f117e2825c0dfd12a21961c99dc0b4 Mon Sep 17 00:00:00 2001 From: Songhaoz666 Date: Wed, 10 Jun 2026 16:28:33 +0800 Subject: [PATCH] =?UTF-8?q?benchmark=20Group=20C=EF=BC=9A=E5=9F=BA?= =?UTF-8?q?=E7=BA=BF=E8=BF=90=E8=A1=8C=E5=99=A8=20+=20=E7=BB=9F=E4=B8=80?= =?UTF-8?q?=20BenchmarkRunRecord=20+=20=E6=8A=A5=E5=91=8A/=E5=9B=9E?= =?UTF-8?q?=E6=94=BE=EF=BC=88Closes=20#21=EF=BC=89?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit 在去中心化重构之上落地 benchmark 对比管线：5 个系统（single/strong/chain/sub_agent/swarm） 跑同一任务集、同一执行后端，产出统一 BenchmarkRunRecord → 评估器算 G_E/G_E,c → 报告 + 回放。 - benchmark/runners/：backend（Offline 确定性 / OpenAI 真实）+ base + 5 个 runner。各 runner 用 held-out fixture 测试在 Group B 沙箱里评分得 TestPassRate（权威，非自评）。 - benchmark/tasksets/：统一任务集 + 加载器（coding-set-1，1 个 fixture）。 - benchmark/reports/、benchmark/replay/：G_E/G_E,c/coverage/confidence + 归档。 - benchmark/baselines/comparison.py：BenchmarkRunRecord 的 CodeReview/UserAcceptance 改为 Optional（掩码归一，未采集即 None，规则 #9）。 - scripts/run-benchmark-suite.py harness + scripts/test-benchmark-runners.py。 与去中心化重构对齐：swarm runner 拓扑已**重指向去中心化流程**（种子→自选→自主分解→竞争→ 同伴交叉评审→收敛，calls=6/review=1），非旧 Master「分解→派发→单评审」。仍用同一离线后端 建模以保证公平对比（驱动活体编排器会换后端→记录不可比；活体全流程由 test-workflow-e2e 验证）。 沙箱适配：runner 评分走 fail-closed 沙箱（#24），故 test + CI 步骤设 HEICODE_SANDBOX_ISOLATED=1 （仅 CI/隔离 Pod）。 影响范围：agent_swarm（benchmark 层 + 测试 + docs + CI）。不碰 orchestrator 编排逻辑、 不改 Manager↔Swarm 契约、不影响 Client/计费/密钥/审计/发布链路。 诚实边界： - **离线后端只验证管线**：所有系统拿同一参考解 → quality 相同 → G_E=0、swarm_valid=False， 刻意不显示蜂群优势（反造假）。真实 G_E>0 需 --backend openai + 足量冻结任务集 + 多次运行。 - 故 Closes #21（运行器 + 统一记录已落地并产出合规非 NaN 记录）；Refs #20（仅 1/5 场景）、 Refs #22（评估器/报告/回放已建，但 Quality 仅 TestPassRate，CodeReview/UserAcceptance 缺）、 Refs #13（验收 EPIC，需真实 run 证明 Swarm>baselines，未满足）。 Closes #21 Refs #20 Refs #22 Refs #13 Co-Authored-By: Claude Opus 4.8 (1M context) --- .github/workflows/ci.yml | 12 ++ .gitignore | 4 + benchmark/README.md | 13 +- benchmark/baselines/comparison.py | 22 ++- benchmark/fixtures/__init__.py | 15 +++ .../add_function/reference_solution/calc.py | 11 ++ benchmark/replay/__init__.py | 41 +++++- benchmark/reports/__init__.py | 126 ++++++++++++++++++ benchmark/runners/__init__.py | 44 ++++++ benchmark/runners/backend.py | 115 ++++++++++++++++ benchmark/runners/base.py | 125 +++++++++++++++++ benchmark/runners/chain.py | 9 ++ benchmark/runners/single.py | 8 ++ benchmark/runners/strong.py | 14 ++ benchmark/runners/sub_agent.py | 11 ++ benchmark/runners/swarm.py | 30 +++++ benchmark/tasksets/README.md | 24 ++++ benchmark/tasksets/__init__.py | 68 ++++++++++ benchmark/tasksets/coding-set-1/taskset.json | 7 + .../IMPORTANT-metric-coverage-gaps.md | 2 +- docs/benchmark/baseline-comparison.md | 14 +- docs/benchmark/baseline-record-schema.md | 6 +- docs/benchmark/baseline-runners.md | 59 ++++++++ scripts/run-benchmark-suite.py | 65 +++++++++ scripts/test-benchmark-runners.py | 91 +++++++++++++ 25 files changed, 916 insertions(+), 20 deletions(-) create mode 100644 benchmark/fixtures/add_function/reference_solution/calc.py create mode 100644 benchmark/reports/__init__.py create mode 100644 benchmark/runners/__init__.py create mode 100644 benchmark/runners/backend.py create mode 100644 benchmark/runners/base.py create mode 100644 benchmark/runners/chain.py create mode 100644 benchmark/runners/single.py create mode 100644 benchmark/runners/strong.py create mode 100644 benchmark/runners/sub_agent.py create mode 100644 benchmark/runners/swarm.py create mode 100644 benchmark/tasksets/README.md create mode 100644 benchmark/tasksets/__init__.py create mode 100644 benchmark/tasksets/coding-set-1/taskset.json create mode 100644 docs/benchmark/baseline-runners.md create mode 100644 scripts/run-benchmark-suite.py create mode 100644 scripts/test-benchmark-runners.py diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml index c2b547c..a464938 100644 --- a/.github/workflows/ci.yml +++ b/.github/workflows/ci.yml @@ -122,3 +122,15 @@ jobs: python scripts/test-task-competition.py python scripts/test-cross-review.py python scripts/test-convergence.py + + # --- benchmark baseline runners + suite smoke (Group C, #21) --- + # HEICODE_SANDBOX_ISOLATED: the runners grade generated code in the fail-closed sandbox; the + # CI runner is ephemeral/isolated, so confirm isolation here (see security-boundary §8.1). + - name: Benchmark runners (Group C) + env: { HEICODE_SANDBOX_ISOLATED: "1" } + run: python scripts/test-benchmark-runners.py + + # Offline = pipeline validation only (deterministic, no real G_E). Guards against regressions. + - name: Benchmark suite smoke (offline) + env: { HEICODE_SANDBOX_ISOLATED: "1" } + run: python scripts/run-benchmark-suite.py --taskset coding-set-1 diff --git a/.gitignore b/.gitignore index 6ba2abc..8b8f90a 100644 --- a/.gitignore +++ b/.gitignore @@ -46,3 +46,7 @@ tmp-workspace/ Thumbs.db .idea/ .vscode/ + +# Benchmark replay archives (generated by run-benchmark-suite.py --archive) +benchmark/runs/ +runs/ diff --git a/benchmark/README.md b/benchmark/README.md index ffe6dc0..e311e67 100644 --- a/benchmark/README.md +++ b/benchmark/README.md @@ -10,12 +10,17 @@ | `collectors/` | base `SwarmMetricsCollector` + `SwarmRunMetricsCollector`（无条件计算 `s_completion`/`s_collaboration`/`s_robustness`，有条件计算 `s_cost`/`s_governance`/`s_communication`/`reward`/`tau`/`eta`/`p_decision`；其余标记 NaN + `coverage=False`） | 🟡 部分落地（单次 run 最多 10/15 字段真实可算） | | `fixtures/` | 留出（held-out）任务 fixture + 加载器；`reward` 的 `Q_quality` 评分源 | 🟡 1 个示例 fixture（`add_function`），统一任务集待扩充 | | 决策引擎 | `orchestrator/decision_engine.py`（τ trail + η 评分 + ε-greedy 采样，`ENABLE_ACO_DISPATCH` 门控；见 docs/benchmark/decision-engine.md） | 🟡 Option A 单边已落地；Option B 与决策质量验证待 Group C | -| `baselines/` | Single / Chain / Sub-Agent / Strong 基线运行器 | 🔴 未落地 | -| `replay/` | 执行回放 | 🔴 未落地 | +| `baselines/` | `BenchmarkRunRecord` + `compare`/`evaluate`（G_E/G_E,c） | ✅ 已实现 | +| `tasksets/` | 统一基准任务集（`coding-set-1`） | 🟡 1 个示例任务集（#20） | +| `runners/` | single/strong/chain/sub_agent/swarm + 共享 backend（Offline/OpenAI） | 🟡 管线已落地；真实数值需 key（#21） | +| `reports/` | G_E/G_E,c/coverage/confidence + Markdown | ✅ 已实现（#22 部分） | +| `replay/` | 记录/报告归档（records/report/md/meta） | 🟡 最小回放（#22 部分） | | `leaderboard/` | 排行榜聚合 | 🔴 未落地 | -> 公式可计算 ≠ 能自证。**数据采集（collectors）与基线（baselines）未落地前，不得宣称已具备 -> Agent Swarm 量化自证能力**（不能证明 `Swarm > Single/Chain/Sub-Agent/Strong`，也无法输出完整 `Benchmark_Agent`）。 +> 公式可计算 ≠ 能自证。基线运行管线（runners/tasksets/reports/replay）已落地，但 **Offline 仅验证管线 +> （无偏、G_E=0），真实 `G_E>0` 需 `--backend openai` + 冻结任务集 + 多次运行**。在拿到真实结果前， +> **不得宣称已具备 Agent Swarm 量化自证能力**（不能证明 `Swarm > Single/Chain/Sub/Strong`，也无法输出完整 +> `Benchmark_Agent`）。入口 `scripts/run-benchmark-suite.py`，说明见 `docs/benchmark/baseline-runners.md`。 ## 待落地（按依赖顺序） 1. `collectors/`：接入事件流 / Prometheus / OTel / 任务与用量（telemetry-architecture）。 diff --git a/benchmark/baselines/comparison.py b/benchmark/baselines/comparison.py index 3203625..12151aa 100644 --- a/benchmark/baselines/comparison.py +++ b/benchmark/baselines/comparison.py @@ -9,6 +9,7 @@ docs/benchmark/baseline-record-schema.md). No record → no comparison (we never from __future__ import annotations from dataclasses import dataclass +from typing import Optional from ..metrics import ( quality_score, completion_score, speed_score, cost_score, robustness_score, @@ -17,19 +18,25 @@ from ..metrics import ( SYSTEMS = {"swarm", "single", "chain", "sub", "strong"} +# Quality sub-inputs that may legitimately be absent (masked-renormalized in quality_score, rule #9). +_OPTIONAL_FIELDS = {"code_review_score", "user_acceptance"} + @dataclass class BenchmarkRunRecord: - """Unified per-run record (v2.0 §9 collection). Both swarm and baselines emit this.""" + """Unified per-run record (v2.0 §9 collection). Both swarm and baselines emit this. + + CodeReview/UserAcceptance are Optional: a runner that can only measure TestPassRate (the + common case today — no reviewer/acceptance signal wired) leaves them None, and quality() + renormalizes over the present inputs. None ≠ 0 (never fabricate a score, rule #9). + """ system: str # one of SYSTEMS scenario: str # coding | refactoring | architecture | devops | bugfix task_set_id: str # identifies the shared task set (fairness: same id across systems) n_agent: int # agent count used by this system completed_tasks: int total_tasks: int - test_pass_rate: float # 0..100 (Quality input) - code_review_score: float # 0..100 (Quality input) - user_acceptance: float # 0..100 (Quality input) + test_pass_rate: float # 0..100 (Quality input — held-out fixture tests) budget_usd: float # planned cost actual_cost_usd: float # measured model cost model_tokens: int @@ -37,15 +44,18 @@ class BenchmarkRunRecord: actual_time_s: float # measured time recovered_failures: int total_failures: int + code_review_score: Optional[float] = None # 0..100, None when not collected + user_acceptance: Optional[float] = None # 0..100, None when not collected @classmethod def from_dict(cls, data: dict) -> "BenchmarkRunRecord": - missing = [f for f in cls.__dataclass_fields__ if f not in data] + required = [f for f in cls.__dataclass_fields__ if f not in _OPTIONAL_FIELDS] + missing = [f for f in required if f not in data] if missing: raise ValueError(f"benchmark record missing required fields: {missing}") if data["system"] not in SYSTEMS: raise ValueError(f"unknown system '{data['system']}' (expected one of {sorted(SYSTEMS)})") - return cls(**{f: data[f] for f in cls.__dataclass_fields__}) + return cls(**{f: data[f] for f in cls.__dataclass_fields__ if f in data}) # --- derived quantities (from a single record) --- diff --git a/benchmark/fixtures/__init__.py b/benchmark/fixtures/__init__.py index 8ec78fa..5291e08 100644 --- a/benchmark/fixtures/__init__.py +++ b/benchmark/fixtures/__init__.py @@ -86,3 +86,18 @@ def load_fixture(fixture_id: str) -> Fixture: target_time_seconds=meta.get("target_time_seconds"), test_files=tests, ) + + +def load_reference_solution(fixture_id: str) -> List[FixtureTest]: + """Offline reference solution files (path/content), used ONLY by the offline backend. + + Returns [] if the fixture has no reference_solution/ dir. These files are NOT the swarm's + work — they exist so hermetic, key-free runs can validate the benchmark pipeline. + """ + ref_dir = fixture_dir(fixture_id) / "reference_solution" + if not ref_dir.is_dir(): + return [] + return [ + FixtureTest(path=p.name, content=p.read_text(encoding="utf-8")) + for p in sorted(ref_dir.glob("*.py")) + ] diff --git a/benchmark/fixtures/add_function/reference_solution/calc.py b/benchmark/fixtures/add_function/reference_solution/calc.py new file mode 100644 index 0000000..59bd942 --- /dev/null +++ b/benchmark/fixtures/add_function/reference_solution/calc.py @@ -0,0 +1,11 @@ +"""Offline reference solution for the add_function fixture. + +Used ONLY by the offline execution backend (benchmark/runners/backend.py) for hermetic, +key-free pipeline validation. It is NOT the swarm's work and is NOT used when a real model +backend runs. Every system gets the same reference output offline, so offline runs prove the +benchmark pipeline (records → G_E → report, no NaN) WITHOUT fabricating any swarm advantage. +""" + + +def add(a, b): + return a + b diff --git a/benchmark/replay/__init__.py b/benchmark/replay/__init__.py index 712c8b2..c3ec315 100644 --- a/benchmark/replay/__init__.py +++ b/benchmark/replay/__init__.py @@ -1,5 +1,40 @@ -"""Benchmark replay — 回放一次 swarm/benchmark 执行。 +"""Benchmark replay archive (#22, minimal): persist a run's records + report for later inspection. -状态：**未落地（骨架）**。事件流已按 swarm_id 顺序持久化，可作为回放数据源； -需定义快照格式与回放接口。见 docs/integration/audit-trace-schema.md §4。 +Writes, under //: + - records.json : every system's BenchmarkRunRecord (the raw, comparable inputs) + - report.json : the computed report (G_E / G_E,c / coverage / confidence) + - report.md : human-readable rendering + - meta.json : run metadata (backend, task_set_id, run_label, + any extra passed in) + +Honest scope: this archives the BENCHMARK layer (records/report/metadata). A full audit replay +(prompts, model versions, per-step traces) belongs to docs/integration/audit-trace-schema.md and +is NOT claimed here. No timestamps are generated internally — the caller passes run_label so the +archive is deterministic and testable. """ +from __future__ import annotations + +import json +from dataclasses import asdict +from pathlib import Path +from typing import Dict, Optional + +from ..baselines import BenchmarkRunRecord +from ..reports import render_markdown + + +def save_archive(out_dir: str, run_label: str, *, records: Dict[str, BenchmarkRunRecord], + report: dict, meta: Optional[dict] = None) -> str: + base = Path(out_dir) / run_label + base.mkdir(parents=True, exist_ok=True) + (base / "records.json").write_text( + json.dumps({s: asdict(r) for s, r in records.items()}, indent=2), encoding="utf-8") + (base / "report.json").write_text(json.dumps(report, indent=2), encoding="utf-8") + (base / "report.md").write_text(render_markdown(report), encoding="utf-8") + (base / "meta.json").write_text(json.dumps({ + "run_label": run_label, + "task_set_id": report.get("task_set_id"), + "backend": report.get("backend"), + "n_runs": report.get("n_runs"), + **(meta or {}), + }, indent=2), encoding="utf-8") + return str(base) diff --git a/benchmark/reports/__init__.py b/benchmark/reports/__init__.py new file mode 100644 index 0000000..8381312 --- /dev/null +++ b/benchmark/reports/__init__.py @@ -0,0 +1,126 @@ +"""Benchmark report (#22): turn per-system BenchmarkRunRecords into G_E / G_E,c / coverage / +confidence, plus a human-readable rendering. + +What the report HONESTLY claims: + - G_E, G_E,c per baseline + swarm_valid (from benchmark.baselines.evaluate). + - per-system unified metrics (completion / quality / cost-efficiency / speed / robustness). + - coverage: which Quality inputs were real (TestPassRate) vs masked-absent (CodeReview/UserAccept). + - confidence: LOW unless many runs on a real backend (offline/1-run = pipeline validation only). + +What it does NOT fabricate: + - The full `Benchmark_Agent` aggregate needs S_swarm (with communication/governance), Reward, + Observability — these come from a LIVE swarm run's SwarmMetrics, not from these topology + records. So `benchmark_agent` is reported as NOT-AVAILABLE here, with the reason, rather than + a guessed number. Wiring the live collector + multi-run + a real backend is the remaining step. +""" +from __future__ import annotations + +import math +from typing import Dict, List + +from ..baselines import BenchmarkRunRecord, evaluate, quality, unified_metrics + + +def _coverage(swarm: BenchmarkRunRecord) -> Dict[str, bool]: + return { + "completion": True, + "quality.test_pass_rate": True, + "quality.code_review_score": swarm.code_review_score is not None, + "quality.user_acceptance": swarm.user_acceptance is not None, + "cost_efficiency": swarm.actual_cost_usd > 0, + "robustness": True, + "g_e": True, # computed vs baselines below + "g_e_cost": True, + # Requires a LIVE swarm run's SwarmMetrics (S_swarm components, Reward, Observability) — + # not derivable from topology records alone. + "benchmark_agent": False, + } + + +def _confidence(backend_name: str, n_runs: int, swarm: BenchmarkRunRecord) -> Dict[str, object]: + reasons = [] + level = "high" + if backend_name != "openai": + level = "none" + reasons.append("offline backend: reference solution echoed, no real model differentiation") + if n_runs < 3: + level = "low" if level != "none" else level + reasons.append(f"only {n_runs} run(s); no variance/significance (need ≥3)") + if swarm.code_review_score is None or swarm.user_acceptance is None: + reasons.append("Quality partial: CodeReview/UserAcceptance not collected (masked)") + return {"level": level, "reasons": reasons} + + +def build_report(records: Dict[str, BenchmarkRunRecord], *, symmetric_cost: bool = True, + backend_name: str = "offline", n_runs: int = 1) -> dict: + if "swarm" not in records: + raise ValueError("records must include a 'swarm' record") + swarm = records["swarm"] + baselines = [r for s, r in records.items() if s != "swarm"] + ev = evaluate(swarm, baselines, symmetric_cost=symmetric_cost) + + per_system = {} + for system, rec in records.items(): + m = unified_metrics(rec) + per_system[system] = { + "n_agent": rec.n_agent, + "completed": f"{rec.completed_tasks}/{rec.total_tasks}", + "quality": round(quality(rec), 4), + "actual_cost_usd": rec.actual_cost_usd, + "cost_efficiency": round(m["cost_efficiency"], 4), + "speed": round(m["speed"], 4), + "robustness": round(m["robustness"], 4), + } + + return { + "task_set_id": swarm.task_set_id, + "scenario": swarm.scenario, + "backend": backend_name, + "n_runs": n_runs, + "cost_mode": ev["cost_mode"], + "per_system": per_system, + "comparisons": ev["comparisons"], + "swarm_valid": ev["swarm_valid"], + "coverage": _coverage(swarm), + "confidence": _confidence(backend_name, n_runs, swarm), + "benchmark_agent": { + "available": False, + "reason": "needs live swarm SwarmMetrics (S_swarm/Reward/Observability) + multi-run on a real backend", + }, + } + + +def render_markdown(report: dict) -> str: + lines: List[str] = [] + a = lines.append + a(f"# Benchmark report — {report['scenario']} / `{report['task_set_id']}`") + a("") + a(f"- backend: **{report['backend']}** · runs: **{report['n_runs']}** · cost mode: {report['cost_mode']}") + a(f"- **swarm_valid: {report['swarm_valid']}** · confidence: **{report['confidence']['level']}**") + if report["confidence"]["reasons"]: + for r in report["confidence"]["reasons"]: + a(f" - ⚠️ {r}") + a("") + a("## Per-system") + a("| system | n_agent | completed | quality | cost($) | cost_eff | speed | robustness |") + a("|---|---|---|---|---|---|---|---|") + for s, v in report["per_system"].items(): + a(f"| {s} | {v['n_agent']} | {v['completed']} | {v['quality']} | {v['actual_cost_usd']} " + f"| {v['cost_efficiency']} | {v['speed']} | {v['robustness']} |") + a("") + a("## Swarm vs baselines") + a("| baseline | G_E | G_E,c | G_E>0 | G_E,c>0 |") + a("|---|---|---|---|---|") + for c in report["comparisons"]: + a(f"| {c['base_system']} | {round(c['g_e'], 4)} | {round(c['g_e_cost'], 4)} " + f"| {c['raw_gain_positive']} | {c['cost_normalized_positive']} |") + a("") + ba = report["benchmark_agent"] + a(f"## Benchmark_Agent: {'available' if ba['available'] else 'NOT AVAILABLE'}") + if not ba["available"]: + a(f"> {ba['reason']}") + a("") + a("## Coverage") + for k, v in report["coverage"].items(): + a(f"- {'✅' if v else '🔴'} {k}") + return "\n".join(lines) diff --git a/benchmark/runners/__init__.py b/benchmark/runners/__init__.py new file mode 100644 index 0000000..93830fd --- /dev/null +++ b/benchmark/runners/__init__.py @@ -0,0 +1,44 @@ +"""Benchmark runners (#21): run a shared task set through each system → BenchmarkRunRecord. + +Same backend + same task set for all systems (fairness); only topology differs. `run_all` is the +convenience entry the harness uses; `strong_backend` lets the Strong baseline use a costlier/larger +model than the others (its definitional difference). +""" +from __future__ import annotations + +from typing import Dict, List, Optional + +from ..baselines import BenchmarkRunRecord +from ..tasksets import TaskSet +from .backend import ExecutionBackend, GenerationResult, OfflineBackend, OpenAIBackend +from .base import BaseRunner, Topology +from .single import SingleRunner +from .strong import StrongRunner +from .chain import ChainRunner +from .sub_agent import SubAgentRunner +from .swarm import SwarmRunner + +RUNNERS = { + "single": SingleRunner, + "strong": StrongRunner, + "chain": ChainRunner, + "sub": SubAgentRunner, + "swarm": SwarmRunner, +} + + +def run_all(taskset: TaskSet, backend: ExecutionBackend, *, + strong_backend: Optional[ExecutionBackend] = None) -> Dict[str, BenchmarkRunRecord]: + """Run every system on the task set; return {system: record}. swarm + 4 baselines.""" + out: Dict[str, BenchmarkRunRecord] = {} + for system, runner_cls in RUNNERS.items(): + be = strong_backend if (system == "strong" and strong_backend is not None) else backend + out[system] = runner_cls().run(taskset, be) + return out + + +__all__ = [ + "BenchmarkRunRecord", "ExecutionBackend", "GenerationResult", "OfflineBackend", "OpenAIBackend", + "BaseRunner", "Topology", "SingleRunner", "StrongRunner", "ChainRunner", "SubAgentRunner", + "SwarmRunner", "RUNNERS", "run_all", +] diff --git a/benchmark/runners/backend.py b/benchmark/runners/backend.py new file mode 100644 index 0000000..c741e41 --- /dev/null +++ b/benchmark/runners/backend.py @@ -0,0 +1,115 @@ +"""Execution backends for benchmark runners — the SHARED model gateway all systems run through. + +Fairness (standard §9.1): swarm and every baseline must use the same backend on the same task +set; only the topology (call pattern) differs. Two backends: + + - OfflineBackend: deterministic, no network, no key. Echoes the fixture's offline reference + solution and charges a notional per-call cost. Used for hermetic CI + pipeline validation. + It is IDENTICAL across systems, so offline runs can prove the pipeline (records → G_E → + report, no NaN) but CANNOT show a swarm quality advantage — by design (anti-fabrication). + + - OpenAIBackend: real OpenAI-compatible generation (needs OPENAI_API_KEY / OPENAI_BASE_URL). + Produces real files + real token/cost. This is what yields REAL G_E numbers. + +`generate()` returns the produced implementation files plus measured token/cost/time. +""" +from __future__ import annotations + +import json +import os +import re +from abc import ABC, abstractmethod +from dataclasses import dataclass, field +from typing import List, Optional, Tuple + +# (path, content) pairs. +Files = List[Tuple[str, str]] + + +@dataclass +class GenerationResult: + files: Files = field(default_factory=list) + tokens: int = 0 + cost_usd: float = 0.0 + elapsed_s: float = 0.0 + + +class ExecutionBackend(ABC): + name: str = "abstract" + + @abstractmethod + def generate(self, *, objective: str, reference_files: Files, max_tokens: int) -> GenerationResult: + """Produce implementation files for one objective.""" + + +class OfflineBackend(ExecutionBackend): + """Deterministic, key-free. Echoes the fixture reference solution; charges a notional cost. + + cost_per_call / tokens_per_call / time_per_call let the harness model a 'stronger' (costlier) + configuration without a real model — see runners/strong.py. + """ + name = "offline" + + def __init__(self, *, cost_per_call: float = 0.002, tokens_per_call: int = 800, + time_per_call: float = 0.5): + self.cost_per_call = cost_per_call + self.tokens_per_call = tokens_per_call + self.time_per_call = time_per_call + + def generate(self, *, objective: str, reference_files: Files, max_tokens: int) -> GenerationResult: + # Deterministic: same reference output regardless of system/topology (no swarm bias). + return GenerationResult( + files=list(reference_files), + tokens=self.tokens_per_call, + cost_usd=self.cost_per_call, + elapsed_s=self.time_per_call, + ) + + +class OpenAIBackend(ExecutionBackend): + """Real OpenAI-compatible generation. Lazy client; raises clearly if no key is configured.""" + name = "openai" + + def __init__(self, *, model: Optional[str] = None, price_per_1k_tokens: float = 0.0): + self.model = model or os.getenv("OPENAI_MODEL", "gpt-4o-mini") + self.price_per_1k_tokens = price_per_1k_tokens + self._client = None + + def _get_client(self): + if self._client is None: + from openai import OpenAI # lazy: keep offline/CI import-light + if not os.getenv("OPENAI_API_KEY"): + raise RuntimeError("OpenAIBackend requires OPENAI_API_KEY (use OfflineBackend for hermetic runs)") + self._client = OpenAI(base_url=os.getenv("OPENAI_BASE_URL") or None) + return self._client + + @staticmethod + def _parse_files(content: str) -> Files: + # Accept a JSON {"files":[{"path","content"}]} (same shape the agent uses), else empty. + try: + text = re.sub(r"^```(json)?|```$", "", content.strip(), flags=re.MULTILINE).strip() + data = json.loads(text) + return [(f["path"], f["content"]) for f in data.get("files", []) if f.get("path")] + except Exception: + return [] + + def generate(self, *, objective: str, reference_files: Files, max_tokens: int) -> GenerationResult: + import time as _time + client = self._get_client() + prompt = ( + f"{objective}\n\nReturn ONLY JSON: " + '{"files":[{"path":"relative/path.py","content":"complete file content"}]}' + ) + start = _time.time() + resp = client.chat.completions.create( + model=self.model, + messages=[{"role": "user", "content": prompt}], + max_tokens=max_tokens, + ) + elapsed = _time.time() - start + content = resp.choices[0].message.content or "" + usage = getattr(resp, "usage", None) + tokens = int(getattr(usage, "total_tokens", 0) or 0) + cost = tokens / 1000.0 * self.price_per_1k_tokens + return GenerationResult(files=self._parse_files(content), tokens=tokens, + cost_usd=cost, elapsed_s=elapsed) diff --git a/benchmark/runners/base.py b/benchmark/runners/base.py new file mode 100644 index 0000000..0e261b6 --- /dev/null +++ b/benchmark/runners/base.py @@ -0,0 +1,125 @@ +"""Shared runner: run a task set through a backend with a given topology → BenchmarkRunRecord. + +Each system (single/strong/chain/sub_agent/swarm) is the same pipeline with a different topology +(how many model calls, how many agents, whether a review round happens). All systems grade the +SAME generated deliverable against the fixture's HELD-OUT tests in the Group B sandbox, so quality +is measured identically and fairly. + +The deliverable graded is the LAST generation's implementation files (test files the system may +emit are excluded — held-out fixture tests are the grader). completed = produced gradeable impl +files; test_pass_rate = mean over tasks; CodeReview/UserAcceptance are NOT collected (None → +masked in quality()). +""" +from __future__ import annotations + +import os +from dataclasses import dataclass + +from orchestrator.sandbox import SandboxFile, run_tests + +from ..baselines import BenchmarkRunRecord +from ..fixtures import load_reference_solution +from ..tasksets import TaskSet +from .backend import ExecutionBackend, Files + + +def _is_test_file(path: str) -> bool: + base = os.path.basename(path or "") + return base.startswith("test_") and base.endswith(".py") + + +@dataclass +class Topology: + """How a system uses the backend (the only thing that differs between systems).""" + system: str # swarm | single | chain | sub | strong + n_agent: int # agents this system uses (for cost-normalized gain) + calls_per_task: int # backend generations per task (drives cost/time/tokens) + review_rounds: int = 0 # extra grade+regenerate rounds (swarm/strong-ish); 0 = none + + +class BaseRunner: + """Run a task set under one topology and emit a unified BenchmarkRunRecord.""" + + topology: Topology + + def __init__(self, max_tokens: int = 1500): + self.max_tokens = max_tokens + + def run(self, taskset: TaskSet, backend: ExecutionBackend) -> BenchmarkRunRecord: + topo = self.topology + total = len(taskset.tasks) + completed = 0 + total_tokens = 0 + total_cost = 0.0 + total_time = 0.0 + target_time = 0.0 + pass_rates: list[float] = [] + recovered = 0 + failures = 0 + + for item in taskset.tasks: + fixture = item.fixture + target_time += float(fixture.target_time_seconds or 0.0) + reference: Files = [(f.path, f.content) for f in load_reference_solution(fixture.id)] + test_files = [SandboxFile(t.path, t.content) for t in fixture.test_files] + + # Topology: N generation calls (sequential stages / sub-agents / specialists). The last + # call's files are the deliverable; earlier calls still cost tokens/time/$. + gen = None + for _ in range(max(1, topo.calls_per_task)): + gen = backend.generate(objective=fixture.objective, reference_files=reference, + max_tokens=self.max_tokens) + total_tokens += gen.tokens + total_cost += gen.cost_usd + total_time += gen.elapsed_s + + impl = [SandboxFile(p, c) for p, c in (gen.files if gen else []) if not _is_test_file(p)] + pass_rate = 0.0 + if impl and test_files: + result = run_tests(impl, test_files) + pass_rate = result.pass_rate if result.pass_rate is not None else 0.0 + if gen and gen.files: + completed += 1 + pass_rates.append(pass_rate) + + # Review rounds (swarm): if the deliverable failed, grade-and-regenerate up to N times. + # Offline the reference always passes, so this is exercised mainly under a real backend; + # each round is counted as a recovered failure when it lifts the pass rate. + rounds = 0 + while rounds < topo.review_rounds and pass_rate < 100.0: + failures += 1 + gen = backend.generate(objective=fixture.objective, reference_files=reference, + max_tokens=self.max_tokens) + total_tokens += gen.tokens + total_cost += gen.cost_usd + total_time += gen.elapsed_s + impl = [SandboxFile(p, c) for p, c in gen.files if not _is_test_file(p)] + if impl and test_files: + new_rate = run_tests(impl, test_files).pass_rate or 0.0 + if new_rate > pass_rate: + recovered += 1 + pass_rates[-1] = new_rate + pass_rate = new_rate + rounds += 1 + + test_pass_rate = sum(pass_rates) / len(pass_rates) if pass_rates else 0.0 + # Budget口径: a flat planned budget per call across topology (fair: same per-call budget). + budget = max(total_cost, topo.calls_per_task * total * 0.0) or 1.0 + return BenchmarkRunRecord( + system=topo.system, + scenario=taskset.scenario, + task_set_id=taskset.task_set_id, + n_agent=topo.n_agent, + completed_tasks=completed, + total_tasks=total, + test_pass_rate=test_pass_rate, + code_review_score=None, # not collected (no reviewer wired) + user_acceptance=None, # not collected (external signal) + budget_usd=round(budget * 1.5, 6), # planned = 1.5× measured (headroom), same rule for all + actual_cost_usd=round(total_cost, 6), + model_tokens=total_tokens, + target_time_s=target_time or 1.0, + actual_time_s=round(total_time, 6) or 1.0, + recovered_failures=recovered, + total_failures=failures, + ) diff --git a/benchmark/runners/chain.py b/benchmark/runners/chain.py new file mode 100644 index 0000000..d5e94f0 --- /dev/null +++ b/benchmark/runners/chain.py @@ -0,0 +1,9 @@ +"""Baseline B — Chain Agent: sequential pipeline (implement → test → document), NO swarm +collaboration and NO review loop (standard §9.1). 3 sequential stages = 3 agents, 3 calls.""" +from __future__ import annotations + +from .base import BaseRunner, Topology + + +class ChainRunner(BaseRunner): + topology = Topology(system="chain", n_agent=3, calls_per_task=3, review_rounds=0) diff --git a/benchmark/runners/single.py b/benchmark/runners/single.py new file mode 100644 index 0000000..0c55eae --- /dev/null +++ b/benchmark/runners/single.py @@ -0,0 +1,8 @@ +"""Baseline A — Single Agent: one agent, one generation per task, no review (standard §9.1).""" +from __future__ import annotations + +from .base import BaseRunner, Topology + + +class SingleRunner(BaseRunner): + topology = Topology(system="single", n_agent=1, calls_per_task=1, review_rounds=0) diff --git a/benchmark/runners/strong.py b/benchmark/runners/strong.py new file mode 100644 index 0000000..90b7689 --- /dev/null +++ b/benchmark/runners/strong.py @@ -0,0 +1,14 @@ +"""Baseline D — Strong Agent: a single high-capability agent (bigger model / longer context). + +Same topology as Single (1 agent, 1 call, no review); the difference is the BACKEND — the harness +gives StrongRunner a stronger, costlier backend (real: a larger model; offline: higher per-call +cost). So it should reach similar/better quality at higher cost than Single — the hardest baseline +to beat (reference coefficient ×0.95). +""" +from __future__ import annotations + +from .base import BaseRunner, Topology + + +class StrongRunner(BaseRunner): + topology = Topology(system="strong", n_agent=1, calls_per_task=1, review_rounds=0) diff --git a/benchmark/runners/sub_agent.py b/benchmark/runners/sub_agent.py new file mode 100644 index 0000000..2551a31 --- /dev/null +++ b/benchmark/runners/sub_agent.py @@ -0,0 +1,11 @@ +"""Baseline C — Sub-Agent: supervisor delegates to sub-agents (hierarchical), NO peer +collaboration and NO review/redo loop (standard §9.1). 1 supervisor (decompose) + 3 sub-agents += 4 agents, 4 calls. Reference coefficient ×0.90 — the closest baseline to the swarm topology, so +the most important to out-perform on G_E.""" +from __future__ import annotations + +from .base import BaseRunner, Topology + + +class SubAgentRunner(BaseRunner): + topology = Topology(system="sub", n_agent=4, calls_per_task=4, review_rounds=0) diff --git a/benchmark/runners/swarm.py b/benchmark/runners/swarm.py new file mode 100644 index 0000000..4360407 --- /dev/null +++ b/benchmark/runners/swarm.py @@ -0,0 +1,30 @@ +"""Experimental — Swarm: the DECENTRALIZED self-organizing flow (post-rework). + +Models the live swarm's call/cost structure (see docs/swarm/decentralized-rework-plan.md): + seed (1) → agents self-select + bottom-up DECOMPOSE into subtasks (proposals) → specialists + EXECUTE → peer CROSS-REVIEW (>=2 reviewers) → CONVERGE → synthesize. +Per task-set item: 1 seed-perceive + ~3 proposed specialist subtask generations + a synthesis, +plus 1 cross-review pass (>=2 independent reviewers). That is materially MORE model work than a +single agent — the swarm trades cost for collaboration/validation, which is exactly what G_E,c is +meant to weigh. + +`calls_per_task=6` (seed + ~3 proposed specialist subtasks + synthesis + a cross-review pass) and +`review_rounds=1` (the cross-review redo cycle); `n_agent=3` specialists. These reflect the +DECENTRALIZED topology, NOT the deleted Master decompose→dispatch→single-critic flow. + +WHY a topology model and not the live orchestrator: a FAIR G_E requires every system (this swarm + +all baselines) to run the SAME task set through the SAME execution backend; only the topology may +differ. Driving the live orchestrator would execute via the WS agents (a different backend), making +its record incomparable to the offline baselines. A live-orchestrator + SwarmRunMetricsCollector +record is faithful but only comparable against OTHER live runs — tracked as a follow-on. The +end-to-end decentralized flow itself is exercised by scripts/test-workflow-e2e.py. +""" +from __future__ import annotations + +from .base import BaseRunner, Topology + + +class SwarmRunner(BaseRunner): + # Decentralized flow: seed + bottom-up decomposition + specialist execution + peer cross-review + # + synthesis — more calls than any baseline (the cost of collaboration/validation). + topology = Topology(system="swarm", n_agent=3, calls_per_task=6, review_rounds=1) diff --git a/benchmark/tasksets/README.md b/benchmark/tasksets/README.md new file mode 100644 index 0000000..dbf6750 --- /dev/null +++ b/benchmark/tasksets/README.md @@ -0,0 +1,24 @@ +# benchmark/tasksets/ —— 统一基准任务集（#20） + +蜂群与全部基线在**同一 `task_set_id`、同一模型网关**下运行同一组任务，才可比（标准 §9.1）。 +每个任务引用一个 fixture（目标 + 留出验收测试 + 离线参考解）。 + +## 实现状态（诚实覆盖） + +| 场景 scenario | task set | 状态 | +|---|---|---| +| coding | `coding-set-1`（`add_function`） | 🟡 1 个真实任务（示例规模） | +| refactoring | — | 🔴 未建（缺 fixture + 留出测试） | +| architecture | — | 🔴 未建 | +| devops | — | 🔴 未建 | +| bugfix | — | 🔴 未建 | + +> 只有带真实 fixture 的场景才可加载；**不发布空任务集**（空集会「静默通过」，违反规则 #9）。 +> 扩充 = 新增 `benchmark/fixtures//`（含 `tests/test_*.py` 留出测试 + 可选 `reference_solution/`） +> 后，在对应 `benchmark/tasksets//taskset.json` 引用。 + +## 与验收的关系 + +`G_E>0` / `Benchmark_Agent` 验收要求在**冻结的、足够规模的**任务集上、用**真实模型**多次运行 +（见 `docs/benchmark/baseline-comparison.md`）。当前任务集为**示例规模**，仅用于打通管线， +**不构成验收**。 diff --git a/benchmark/tasksets/__init__.py b/benchmark/tasksets/__init__.py new file mode 100644 index 0000000..09eb494 --- /dev/null +++ b/benchmark/tasksets/__init__.py @@ -0,0 +1,68 @@ +"""Unified benchmark task sets (#20) — the shared, frozen tasks every system runs. + +Fairness rule (standard §9.1 / baseline-comparison §3): swarm and every baseline run the SAME +task_set_id under the SAME model gateway. A task set groups tasks of one scenario; each task +references a fixture (objective + HELD-OUT acceptance tests + offline reference solution). + +Layout: benchmark/tasksets//taskset.json + +`taskset.json`: + { + "task_set_id": "coding-set-1", + "scenario": "coding", # coding | refactoring | architecture | devops | bugfix + "tasks": [ { "id": "add_function", "fixture": "add_function" } ] + } + +Coverage is honest: only scenarios with real fixtures are loadable. The other four scenarios +(refactoring/architecture/devops/bugfix) are declared TODO in this package's README until their +fixtures + held-out tests exist — we do not ship empty task sets that would silently pass. +""" +from __future__ import annotations + +import json +from dataclasses import dataclass, field +from pathlib import Path +from typing import List + +from ..fixtures import Fixture, load_fixture + +_TASKSET_ROOT = Path(__file__).resolve().parent +SCENARIOS = {"coding", "refactoring", "architecture", "devops", "bugfix"} + + +@dataclass +class TaskSetItem: + id: str + fixture_id: str + + @property + def fixture(self) -> Fixture: + return load_fixture(self.fixture_id) + + +@dataclass +class TaskSet: + task_set_id: str + scenario: str + tasks: List[TaskSetItem] = field(default_factory=list) + + +def available_tasksets() -> List[str]: + return sorted( + p.name for p in _TASKSET_ROOT.iterdir() + if p.is_dir() and (p / "taskset.json").exists() + ) + + +def load_taskset(task_set_id: str) -> TaskSet: + meta_path = _TASKSET_ROOT / task_set_id / "taskset.json" + if not meta_path.exists(): + raise FileNotFoundError(f"unknown task set: {task_set_id}") + meta = json.loads(meta_path.read_text(encoding="utf-8")) + scenario = meta.get("scenario", "") + if scenario not in SCENARIOS: + raise ValueError(f"task set {task_set_id} has unknown scenario '{scenario}'") + tasks = [TaskSetItem(id=t["id"], fixture_id=t["fixture"]) for t in meta.get("tasks", [])] + if not tasks: + raise ValueError(f"task set {task_set_id} has no tasks (refusing to ship an empty set)") + return TaskSet(task_set_id=meta["task_set_id"], scenario=scenario, tasks=tasks) diff --git a/benchmark/tasksets/coding-set-1/taskset.json b/benchmark/tasksets/coding-set-1/taskset.json new file mode 100644 index 0000000..a2bf7a6 --- /dev/null +++ b/benchmark/tasksets/coding-set-1/taskset.json @@ -0,0 +1,7 @@ +{ + "task_set_id": "coding-set-1", + "scenario": "coding", + "tasks": [ + { "id": "add_function", "fixture": "add_function" } + ] +} diff --git a/docs/benchmark/IMPORTANT-metric-coverage-gaps.md b/docs/benchmark/IMPORTANT-metric-coverage-gaps.md index 12ca440..b82d652 100644 --- a/docs/benchmark/IMPORTANT-metric-coverage-gaps.md +++ b/docs/benchmark/IMPORTANT-metric-coverage-gaps.md @@ -59,7 +59,7 @@ 2. **中成本**：~~`reward`~~（✅ 已关闭，本仓）— `Q_quality` 主输入已由 fixture 沙箱采集；剩余为**扩充统一任务集** + 接入 `CodeReview`/`UserAcceptance` 评审/验收信号（部分需 Product/Manager）。 3. **高成本**： - ~~`p_decision`~~（✅ 已关闭，本仓，Option A）— `τ/η/P` 决策引擎与信息素历史库已落地（`ENABLE_ACO_DISPATCH` 门控）；剩余为 Option B 双边匹配与决策质量验证（依赖 Group C）。 - - `gain`（及由其驱动的 `Benchmark_Agent`、`G_E,c`、「Swarm > baselines」验收）— 实现 4 类基线运行器 + 统一数据集 + 对比/显著性，属跨团队与基础设施工作（见 baseline-comparison）。**这是最后一块，也是唯一动验收的一块。** + - `gain`（及由其驱动的 `Benchmark_Agent`、`G_E,c`、「Swarm > baselines」验收）— 🟡 **基线对比管线已落地**（`benchmark/runners/` + `tasksets/` + `reports/` + `replay/`，#21/#22），CI offline smoke 跑通；但 **Offline 仅验证管线（G_E=0、无偏）**，真实 `G_E>0` 仍需 `--backend openai` + 足量冻结任务集 + 多次运行（方差/显著性）+ Quality 评审/验收输入。**这是最后一块，也是唯一动验收的一块**（见 baseline-runners.md）。 ## 6. 影响 diff --git a/docs/benchmark/baseline-comparison.md b/docs/benchmark/baseline-comparison.md index b755403..78634eb 100644 --- a/docs/benchmark/baseline-comparison.md +++ b/docs/benchmark/baseline-comparison.md @@ -55,10 +55,16 @@ | 组件 | 位置 | 状态 | |---|---|---| -| 基线运行器 A–D | `benchmark/baselines/` | 🔴 未实现 | -| 统一任务集 / 数据集 | `test-data/` + `benchmark/baselines/` | 🔴 未实现 | -| 指标采集 | `benchmark/collectors/` | 🔴 未实现 | -| 对比与显著性 | `benchmark/` | 🔴 未实现 | +| 基线运行器 A–D + Swarm | `benchmark/runners/`（single/strong/chain/sub_agent/swarm + base/backend） | 🟡 已实现（管线）；真实数值需模型 key | +| 共享执行后端（公平网关） | `benchmark/runners/backend.py`（Offline + OpenAI） | 🟡 Offline 已实现（确定性、无偏）；OpenAI 需 `OPENAI_API_KEY` | +| 统一任务集 / 数据集 | `benchmark/tasksets/`（`coding-set-1`） | 🟡 1 个示例任务集；其余场景待建 | +| 指标采集（评分） | held-out fixture + 沙箱（Group B）→ runner | 🟡 TestPassRate 真实；CodeReview/UserAcceptance 掩码缺 | +| 对比 + 报告 + 回放 | `benchmark/reports/`、`benchmark/replay/`、`baselines/comparison.py` | ✅ 已实现 | +| 显著性（多 run 方差） | — | 🔴 未实现（confidence 现按 run 数标注） | + +> ⚠️ **Offline 后端为管线验证**：所有系统拿到同一参考解 → quality 相同 → **G_E=0、swarm_valid=False**， +> **刻意不显示蜂群优势**（防造假）。真实 `G_E>0` 需 `--backend openai` + 足量冻结任务集 + 多次运行。 +> 入口：`scripts/run-benchmark-suite.py`；说明见 [`baseline-runners.md`](./baseline-runners.md)。 ## 6. 待对齐 diff --git a/docs/benchmark/baseline-record-schema.md b/docs/benchmark/baseline-record-schema.md index 8fc31e8..383fbba 100644 --- a/docs/benchmark/baseline-record-schema.md +++ b/docs/benchmark/baseline-record-schema.md @@ -1,8 +1,10 @@ # Baseline Run Record Schema（基线运行记录 · 共享契约） -> 状态：**Schema + 比较评估器已落地；记录生产（尤其 Quality）未落地**。 +> 状态：**Schema + 评估器 + 记录生产（runner）均已落地；Quality 仅 TestPassRate 真实，CodeReview/UserAcceptance 仍缺；真实数值需模型 key**。 > -> 依据：**Agent 蜂群指标量化与标准 v2.0 §6, §9**。实现：`benchmark/baselines/comparison.py`（`BenchmarkRunRecord` / `compare` / `evaluate`）。配套：[`baseline-comparison.md`](./baseline-comparison.md)、[`emergence-evaluation.md`](./emergence-evaluation.md)、[`cost-normalized-gain.md`](./cost-normalized-gain.md)、[`IMPORTANT-metric-coverage-gaps.md`](./IMPORTANT-metric-coverage-gaps.md)。 +> 依据：**Agent 蜂群指标量化与标准 v2.0 §6, §9**。实现：`benchmark/baselines/comparison.py`（`BenchmarkRunRecord` / `compare` / `evaluate`）+ `benchmark/runners/`（5 系统产出记录，见 [`baseline-runners.md`](./baseline-runners.md)）。配套：[`baseline-comparison.md`](./baseline-comparison.md)、[`emergence-evaluation.md`](./emergence-evaluation.md)、[`cost-normalized-gain.md`](./cost-normalized-gain.md)、[`IMPORTANT-metric-coverage-gaps.md`](./IMPORTANT-metric-coverage-gaps.md)。 +> +> 更新：`code_review_score`/`user_acceptance` 现为 **Optional**（runner 仅采 TestPassRate 时传 None，`quality()` 掩码归一；None ≠ 0）。 ## 1. 目的 diff --git a/docs/benchmark/baseline-runners.md b/docs/benchmark/baseline-runners.md new file mode 100644 index 0000000..333cb74 --- /dev/null +++ b/docs/benchmark/baseline-runners.md @@ -0,0 +1,59 @@ +# 基线运行器与基准套件（Benchmark Group C · #21/#22） + +把「swarm vs 基线」从纸面公式变成**可运行的管线**：统一任务集 → 5 个系统各跑一遍 → 统一 +`BenchmarkRunRecord` → 评估器算 `G_E`/`G_E,c` → 报告 + 回放归档。本文是 Group C 的唯一入口。 + +## 1. 管线 + +``` +benchmark/tasksets/ ──┐ + ├─ runners.run_all(taskset, backend) ──► {system: BenchmarkRunRecord} +共享 ExecutionBackend ──────┘ │（5 系统同后端同任务集，仅拓扑不同） + ▼ + reports.build_report ──► G_E / G_E,c / coverage / confidence / swarm_valid + ▼ + replay.save_archive ──► records.json / report.json / report.md / meta.json +``` + +入口：`scripts/run-benchmark-suite.py --taskset coding-set-1 [--backend offline|openai]`。 + +## 2. 五个系统（标准 §9.1，仅拓扑不同） + +| system | 文件 | 拓扑（公平：同后端同任务集） | n_agent | +|---|---|---|---| +| single | `runners/single.py` | 1 次生成，无评审 | 1 | +| strong | `runners/strong.py` | 1 次生成，**更强/更贵后端**（×0.95 最难超） | 1 | +| chain | `runners/chain.py` | 串行 impl→test→doc，无协作/评审 | 3 | +| sub | `runners/sub_agent.py` | 主管分解 + 3 子агент，无对等/评审（×0.90 最近邻） | 4 | +| swarm | `runners/swarm.py` | **去中心化**：种子 → 自选 + 自主分解 → 专家执行 → **同伴交叉评审** → 收敛 → 综合（calls=6, review=1） | 3 | + +> swarm 拓扑已对齐**去中心化重构**（播种/自选/自主分解/竞争/交叉评审/收敛，见 `docs/swarm/decentralized-rework-plan.md`），非旧 Master「分解→派发→单评审」。仍用**同一离线后端**建模以保证公平对比；驱动活体编排器会换后端→记录不可比，故另计（见 `runners/swarm.py` 说明），活体全流程由 `scripts/test-workflow-e2e.py` 验证。 + +每个系统的产出物用 **fixture 留出测试**在 Group B 沙箱里评分得 `TestPassRate`（权威，非自评）。 + +## 3. 执行后端（公平网关，`runners/backend.py`） + +- **OfflineBackend**（默认）：确定性、无 key、无网络。回显 fixture 的 `reference_solution/`，按调用数计名义成本。**对所有系统完全相同** → 管线可验证，但**不可能也不应显示蜂群质量优势**（防造假）。 +- **OpenAIBackend**：真实 OpenAI 兼容生成（需 `OPENAI_API_KEY`），产出真实文件 + token/成本。**真实 `G_E` 由它产生。** + +## 4. 当前能得到什么 / 不能得到什么 + +**能**（Offline，CI 每次跑）：完整管线、5 份非 NaN 记录、`G_E`/`G_E,c`/coverage/confidence、回放归档、Markdown 报告。 + +**不能**（诚实）： +- Offline `G_E=0`、`swarm_valid=False`——同一参考解，无差异。**这是正确的反造假结果，不是 bug。** +- 真实 `G_E>0` 需 `--backend openai` + **足量冻结任务集**（当前仅 1 个示例任务）+ **多次运行求方差**（当前 confidence=none/low）。 +- `CodeReview`/`UserAcceptance` 未采集（掩码忽略）。 +- 完整 `Benchmark_Agent` 需**活体 swarm run 的 SwarmMetrics**（S_swarm/Reward/Observability，来自 Group A/B 的 collector）与本套件 `G_E` 的合成——报告里标为 NOT AVAILABLE，**不猜数**。 + +## 5. 与验收（#13/Issue #2）的关系 + +本套件是 #13 验收的**必要管线**，但**本身不构成验收**。验收要求：真实后端、冻结任务集、多次运行、 +对**全部** A–D 满足 `G_E>0` 且 `G_E,c>0`、并能排除「靠堆 Agent/Token 变强」。在拿到这些真实结果前， +**不得关闭 Issue #2/#13，不得宣称 Agent Swarm 正式验收通过。** + +## 6. 测试 + +- `scripts/test-benchmark-runners.py`：5 runner 产出有效记录、offline 质量一致（G_E=0 反造假）、 + 报告/coverage/confidence、回放归档（hermetic）。 +- `scripts/run-benchmark-suite.py`：CI offline smoke gate（防回归）。 diff --git a/scripts/run-benchmark-suite.py b/scripts/run-benchmark-suite.py new file mode 100644 index 0000000..1ae4a4b --- /dev/null +++ b/scripts/run-benchmark-suite.py @@ -0,0 +1,65 @@ +"""Benchmark suite harness (#13/#21/#22): run a task set through all 5 systems → report. + +Default backend is OFFLINE (hermetic, key-free) — it validates the whole pipeline but, by design, +shows NO swarm quality advantage (every system gets the same reference solution). REAL G_E numbers +require `--backend openai` with OPENAI_API_KEY set and a sufficiently large, frozen task set. + +Grading executes generated code in the fail-closed sandbox, so HEICODE_SANDBOX_ISOLATED must be +set (only inside an isolated pod / ephemeral CI runner — see security-boundary §8.1). + +Usage: + HEICODE_SANDBOX_ISOLATED=1 python scripts/run-benchmark-suite.py --taskset coding-set-1 \ + [--backend offline|openai] [--archive runs/