benchmark Group C：基线运行器 + 统一 BenchmarkRunRecord + 报告/回放（Closes #21）
在去中心化重构之上落地 benchmark 对比管线：5 个系统（single/strong/chain/sub_agent/swarm） 跑同一任务集、同一执行后端，产出统一 BenchmarkRunRecord → 评估器算 G_E/G_E,c → 报告 + 回放。 - benchmark/runners/：backend（Offline 确定性 / OpenAI 真实）+ base + 5 个 runner。各 runner 用 held-out fixture 测试在 Group B 沙箱里评分得 TestPassRate（权威，非自评）。 - benchmark/tasksets/：统一任务集 + 加载器（coding-set-1，1 个 fixture）。 - benchmark/reports/、benchmark/replay/：G_E/G_E,c/coverage/confidence + 归档。 - benchmark/baselines/comparison.py：BenchmarkRunRecord 的 CodeReview/UserAcceptance 改为 Optional（掩码归一，未采集即 None，规则 #9）。 - scripts/run-benchmark-suite.py harness + scripts/test-benchmark-runners.py。 与去中心化重构对齐：swarm runner 拓扑已**重指向去中心化流程**（种子→自选→自主分解→竞争→ 同伴交叉评审→收敛，calls=6/review=1），非旧 Master「分解→派发→单评审」。仍用同一离线后端 建模以保证公平对比（驱动活体编排器会换后端→记录不可比；活体全流程由 test-workflow-e2e 验证）。 沙箱适配：runner 评分走 fail-closed 沙箱（#24），故 test + CI 步骤设 HEICODE_SANDBOX_ISOLATED=1 （仅 CI/隔离 Pod）。 影响范围：agent_swarm（benchmark 层 + 测试 + docs + CI）。不碰 orchestrator 编排逻辑、 不改 Manager↔Swarm 契约、不影响 Client/计费/密钥/审计/发布链路。 诚实边界： - **离线后端只验证管线**：所有系统拿同一参考解 → quality 相同 → G_E=0、swarm_valid=False， 刻意不显示蜂群优势（反造假）。真实 G_E>0 需 --backend openai + 足量冻结任务集 + 多次运行。 - 故 Closes #21（运行器 + 统一记录已落地并产出合规非 NaN 记录）；Refs #20（仅 1/5 场景）、 Refs #22（评估器/报告/回放已建，但 Quality 仅 TestPassRate，CodeReview/UserAcceptance 缺）、 Refs #13（验收 EPIC，需真实 run 证明 Swarm>baselines，未满足）。 Closes #21 Refs #20 Refs #22 Refs #13 Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This commit is contained in:
Songhaoz666
co-authored by
Claude Opus 4.8
Claude Opus 4.8
parent
8b5eea296a
commit
baa67350e6
@@ -122,3 +122,15 @@ jobs:
|
||||
python scripts/test-task-competition.py
|
||||
python scripts/test-cross-review.py
|
||||
python scripts/test-convergence.py
|
||||
|
||||
# --- benchmark baseline runners + suite smoke (Group C, #21) ---
|
||||
# HEICODE_SANDBOX_ISOLATED: the runners grade generated code in the fail-closed sandbox; the
|
||||
# CI runner is ephemeral/isolated, so confirm isolation here (see security-boundary §8.1).
|
||||
- name: Benchmark runners (Group C)
|
||||
env: { HEICODE_SANDBOX_ISOLATED: "1" }
|
||||
run: python scripts/test-benchmark-runners.py
|
||||
|
||||
# Offline = pipeline validation only (deterministic, no real G_E). Guards against regressions.
|
||||
- name: Benchmark suite smoke (offline)
|
||||
env: { HEICODE_SANDBOX_ISOLATED: "1" }
|
||||
run: python scripts/run-benchmark-suite.py --taskset coding-set-1
|
||||
|
||||
@@ -46,3 +46,7 @@ tmp-workspace/
|
||||
Thumbs.db
|
||||
.idea/
|
||||
.vscode/
|
||||
|
||||
# Benchmark replay archives (generated by run-benchmark-suite.py --archive)
|
||||
benchmark/runs/
|
||||
runs/
|
||||
|
||||
+9
-4
@@ -10,12 +10,17 @@
|
||||
| `collectors/` | base `SwarmMetricsCollector` + `SwarmRunMetricsCollector`（无条件计算 `s_completion`/`s_collaboration`/`s_robustness`，有条件计算 `s_cost`/`s_governance`/`s_communication`/`reward`/`tau`/`eta`/`p_decision`；其余标记 NaN + `coverage=False`） | 🟡 部分落地（单次 run 最多 10/15 字段真实可算） |
|
||||
| `fixtures/` | 留出（held-out）任务 fixture + 加载器；`reward` 的 `Q_quality` 评分源 | 🟡 1 个示例 fixture（`add_function`），统一任务集待扩充 |
|
||||
| 决策引擎 | `orchestrator/decision_engine.py`（τ trail + η 评分 + ε-greedy 采样，`ENABLE_ACO_DISPATCH` 门控；见 docs/benchmark/decision-engine.md） | 🟡 Option A 单边已落地；Option B 与决策质量验证待 Group C |
|
||||
| `baselines/` | Single / Chain / Sub-Agent / Strong 基线运行器 | 🔴 未落地 |
|
||||
| `replay/` | 执行回放 | 🔴 未落地 |
|
||||
| `baselines/` | `BenchmarkRunRecord` + `compare`/`evaluate`（G_E/G_E,c） | ✅ 已实现 |
|
||||
| `tasksets/` | 统一基准任务集（`coding-set-1`） | 🟡 1 个示例任务集（#20） |
|
||||
| `runners/` | single/strong/chain/sub_agent/swarm + 共享 backend（Offline/OpenAI） | 🟡 管线已落地；真实数值需 key（#21） |
|
||||
| `reports/` | G_E/G_E,c/coverage/confidence + Markdown | ✅ 已实现（#22 部分） |
|
||||
| `replay/` | 记录/报告归档（records/report/md/meta） | 🟡 最小回放（#22 部分） |
|
||||
| `leaderboard/` | 排行榜聚合 | 🔴 未落地 |
|
||||
|
||||
> 公式可计算 ≠ 能自证。**数据采集（collectors）与基线（baselines）未落地前，不得宣称已具备
|
||||
> Agent Swarm 量化自证能力**（不能证明 `Swarm > Single/Chain/Sub-Agent/Strong`，也无法输出完整 `Benchmark_Agent`）。
|
||||
> 公式可计算 ≠ 能自证。基线运行管线（runners/tasksets/reports/replay）已落地，但 **Offline 仅验证管线
|
||||
> （无偏、G_E=0），真实 `G_E>0` 需 `--backend openai` + 冻结任务集 + 多次运行**。在拿到真实结果前，
|
||||
> **不得宣称已具备 Agent Swarm 量化自证能力**（不能证明 `Swarm > Single/Chain/Sub/Strong`，也无法输出完整
|
||||
> `Benchmark_Agent`）。入口 `scripts/run-benchmark-suite.py`，说明见 `docs/benchmark/baseline-runners.md`。
|
||||
|
||||
## 待落地（按依赖顺序）
|
||||
1. `collectors/`：接入事件流 / Prometheus / OTel / 任务与用量（telemetry-architecture）。
|
||||
|
||||
@@ -9,6 +9,7 @@ docs/benchmark/baseline-record-schema.md). No record → no comparison (we never
|
||||
from __future__ import annotations
|
||||
|
||||
from dataclasses import dataclass
|
||||
from typing import Optional
|
||||
|
||||
from ..metrics import (
|
||||
quality_score, completion_score, speed_score, cost_score, robustness_score,
|
||||
@@ -17,19 +18,25 @@ from ..metrics import (
|
||||
|
||||
SYSTEMS = {"swarm", "single", "chain", "sub", "strong"}
|
||||
|
||||
# Quality sub-inputs that may legitimately be absent (masked-renormalized in quality_score, rule #9).
|
||||
_OPTIONAL_FIELDS = {"code_review_score", "user_acceptance"}
|
||||
|
||||
|
||||
@dataclass
|
||||
class BenchmarkRunRecord:
|
||||
"""Unified per-run record (v2.0 §9 collection). Both swarm and baselines emit this."""
|
||||
"""Unified per-run record (v2.0 §9 collection). Both swarm and baselines emit this.
|
||||
|
||||
CodeReview/UserAcceptance are Optional: a runner that can only measure TestPassRate (the
|
||||
common case today — no reviewer/acceptance signal wired) leaves them None, and quality()
|
||||
renormalizes over the present inputs. None ≠ 0 (never fabricate a score, rule #9).
|
||||
"""
|
||||
system: str # one of SYSTEMS
|
||||
scenario: str # coding | refactoring | architecture | devops | bugfix
|
||||
task_set_id: str # identifies the shared task set (fairness: same id across systems)
|
||||
n_agent: int # agent count used by this system
|
||||
completed_tasks: int
|
||||
total_tasks: int
|
||||
test_pass_rate: float # 0..100 (Quality input)
|
||||
code_review_score: float # 0..100 (Quality input)
|
||||
user_acceptance: float # 0..100 (Quality input)
|
||||
test_pass_rate: float # 0..100 (Quality input — held-out fixture tests)
|
||||
budget_usd: float # planned cost
|
||||
actual_cost_usd: float # measured model cost
|
||||
model_tokens: int
|
||||
@@ -37,15 +44,18 @@ class BenchmarkRunRecord:
|
||||
actual_time_s: float # measured time
|
||||
recovered_failures: int
|
||||
total_failures: int
|
||||
code_review_score: Optional[float] = None # 0..100, None when not collected
|
||||
user_acceptance: Optional[float] = None # 0..100, None when not collected
|
||||
|
||||
@classmethod
|
||||
def from_dict(cls, data: dict) -> "BenchmarkRunRecord":
|
||||
missing = [f for f in cls.__dataclass_fields__ if f not in data]
|
||||
required = [f for f in cls.__dataclass_fields__ if f not in _OPTIONAL_FIELDS]
|
||||
missing = [f for f in required if f not in data]
|
||||
if missing:
|
||||
raise ValueError(f"benchmark record missing required fields: {missing}")
|
||||
if data["system"] not in SYSTEMS:
|
||||
raise ValueError(f"unknown system '{data['system']}' (expected one of {sorted(SYSTEMS)})")
|
||||
return cls(**{f: data[f] for f in cls.__dataclass_fields__})
|
||||
return cls(**{f: data[f] for f in cls.__dataclass_fields__ if f in data})
|
||||
|
||||
|
||||
# --- derived quantities (from a single record) ---
|
||||
|
||||
@@ -86,3 +86,18 @@ def load_fixture(fixture_id: str) -> Fixture:
|
||||
target_time_seconds=meta.get("target_time_seconds"),
|
||||
test_files=tests,
|
||||
)
|
||||
|
||||
|
||||
def load_reference_solution(fixture_id: str) -> List[FixtureTest]:
|
||||
"""Offline reference solution files (path/content), used ONLY by the offline backend.
|
||||
|
||||
Returns [] if the fixture has no reference_solution/ dir. These files are NOT the swarm's
|
||||
work — they exist so hermetic, key-free runs can validate the benchmark pipeline.
|
||||
"""
|
||||
ref_dir = fixture_dir(fixture_id) / "reference_solution"
|
||||
if not ref_dir.is_dir():
|
||||
return []
|
||||
return [
|
||||
FixtureTest(path=p.name, content=p.read_text(encoding="utf-8"))
|
||||
for p in sorted(ref_dir.glob("*.py"))
|
||||
]
|
||||
|
||||
@@ -0,0 +1,11 @@
|
||||
"""Offline reference solution for the add_function fixture.
|
||||
|
||||
Used ONLY by the offline execution backend (benchmark/runners/backend.py) for hermetic,
|
||||
key-free pipeline validation. It is NOT the swarm's work and is NOT used when a real model
|
||||
backend runs. Every system gets the same reference output offline, so offline runs prove the
|
||||
benchmark pipeline (records → G_E → report, no NaN) WITHOUT fabricating any swarm advantage.
|
||||
"""
|
||||
|
||||
|
||||
def add(a, b):
|
||||
return a + b
|
||||
@@ -1,5 +1,40 @@
|
||||
"""Benchmark replay — 回放一次 swarm/benchmark 执行。
|
||||
"""Benchmark replay archive (#22, minimal): persist a run's records + report for later inspection.
|
||||
|
||||
状态：**未落地（骨架）**。事件流已按 swarm_id 顺序持久化，可作为回放数据源；
|
||||
需定义快照格式与回放接口。见 docs/integration/audit-trace-schema.md §4。
|
||||
Writes, under <out_dir>/<run_label>/:
|
||||
- records.json : every system's BenchmarkRunRecord (the raw, comparable inputs)
|
||||
- report.json : the computed report (G_E / G_E,c / coverage / confidence)
|
||||
- report.md : human-readable rendering
|
||||
- meta.json : run metadata (backend, task_set_id, run_label, + any extra passed in)
|
||||
|
||||
Honest scope: this archives the BENCHMARK layer (records/report/metadata). A full audit replay
|
||||
(prompts, model versions, per-step traces) belongs to docs/integration/audit-trace-schema.md and
|
||||
is NOT claimed here. No timestamps are generated internally — the caller passes run_label so the
|
||||
archive is deterministic and testable.
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
import json
|
||||
from dataclasses import asdict
|
||||
from pathlib import Path
|
||||
from typing import Dict, Optional
|
||||
|
||||
from ..baselines import BenchmarkRunRecord
|
||||
from ..reports import render_markdown
|
||||
|
||||
|
||||
def save_archive(out_dir: str, run_label: str, *, records: Dict[str, BenchmarkRunRecord],
|
||||
report: dict, meta: Optional[dict] = None) -> str:
|
||||
base = Path(out_dir) / run_label
|
||||
base.mkdir(parents=True, exist_ok=True)
|
||||
(base / "records.json").write_text(
|
||||
json.dumps({s: asdict(r) for s, r in records.items()}, indent=2), encoding="utf-8")
|
||||
(base / "report.json").write_text(json.dumps(report, indent=2), encoding="utf-8")
|
||||
(base / "report.md").write_text(render_markdown(report), encoding="utf-8")
|
||||
(base / "meta.json").write_text(json.dumps({
|
||||
"run_label": run_label,
|
||||
"task_set_id": report.get("task_set_id"),
|
||||
"backend": report.get("backend"),
|
||||
"n_runs": report.get("n_runs"),
|
||||
**(meta or {}),
|
||||
}, indent=2), encoding="utf-8")
|
||||
return str(base)
|
||||
|
||||
@@ -0,0 +1,126 @@
|
||||
"""Benchmark report (#22): turn per-system BenchmarkRunRecords into G_E / G_E,c / coverage /
|
||||
confidence, plus a human-readable rendering.
|
||||
|
||||
What the report HONESTLY claims:
|
||||
- G_E, G_E,c per baseline + swarm_valid (from benchmark.baselines.evaluate).
|
||||
- per-system unified metrics (completion / quality / cost-efficiency / speed / robustness).
|
||||
- coverage: which Quality inputs were real (TestPassRate) vs masked-absent (CodeReview/UserAccept).
|
||||
- confidence: LOW unless many runs on a real backend (offline/1-run = pipeline validation only).
|
||||
|
||||
What it does NOT fabricate:
|
||||
- The full `Benchmark_Agent` aggregate needs S_swarm (with communication/governance), Reward,
|
||||
Observability — these come from a LIVE swarm run's SwarmMetrics, not from these topology
|
||||
records. So `benchmark_agent` is reported as NOT-AVAILABLE here, with the reason, rather than
|
||||
a guessed number. Wiring the live collector + multi-run + a real backend is the remaining step.
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
import math
|
||||
from typing import Dict, List
|
||||
|
||||
from ..baselines import BenchmarkRunRecord, evaluate, quality, unified_metrics
|
||||
|
||||
|
||||
def _coverage(swarm: BenchmarkRunRecord) -> Dict[str, bool]:
|
||||
return {
|
||||
"completion": True,
|
||||
"quality.test_pass_rate": True,
|
||||
"quality.code_review_score": swarm.code_review_score is not None,
|
||||
"quality.user_acceptance": swarm.user_acceptance is not None,
|
||||
"cost_efficiency": swarm.actual_cost_usd > 0,
|
||||
"robustness": True,
|
||||
"g_e": True, # computed vs baselines below
|
||||
"g_e_cost": True,
|
||||
# Requires a LIVE swarm run's SwarmMetrics (S_swarm components, Reward, Observability) —
|
||||
# not derivable from topology records alone.
|
||||
"benchmark_agent": False,
|
||||
}
|
||||
|
||||
|
||||
def _confidence(backend_name: str, n_runs: int, swarm: BenchmarkRunRecord) -> Dict[str, object]:
|
||||
reasons = []
|
||||
level = "high"
|
||||
if backend_name != "openai":
|
||||
level = "none"
|
||||
reasons.append("offline backend: reference solution echoed, no real model differentiation")
|
||||
if n_runs < 3:
|
||||
level = "low" if level != "none" else level
|
||||
reasons.append(f"only {n_runs} run(s); no variance/significance (need ≥3)")
|
||||
if swarm.code_review_score is None or swarm.user_acceptance is None:
|
||||
reasons.append("Quality partial: CodeReview/UserAcceptance not collected (masked)")
|
||||
return {"level": level, "reasons": reasons}
|
||||
|
||||
|
||||
def build_report(records: Dict[str, BenchmarkRunRecord], *, symmetric_cost: bool = True,
|
||||
backend_name: str = "offline", n_runs: int = 1) -> dict:
|
||||
if "swarm" not in records:
|
||||
raise ValueError("records must include a 'swarm' record")
|
||||
swarm = records["swarm"]
|
||||
baselines = [r for s, r in records.items() if s != "swarm"]
|
||||
ev = evaluate(swarm, baselines, symmetric_cost=symmetric_cost)
|
||||
|
||||
per_system = {}
|
||||
for system, rec in records.items():
|
||||
m = unified_metrics(rec)
|
||||
per_system[system] = {
|
||||
"n_agent": rec.n_agent,
|
||||
"completed": f"{rec.completed_tasks}/{rec.total_tasks}",
|
||||
"quality": round(quality(rec), 4),
|
||||
"actual_cost_usd": rec.actual_cost_usd,
|
||||
"cost_efficiency": round(m["cost_efficiency"], 4),
|
||||
"speed": round(m["speed"], 4),
|
||||
"robustness": round(m["robustness"], 4),
|
||||
}
|
||||
|
||||
return {
|
||||
"task_set_id": swarm.task_set_id,
|
||||
"scenario": swarm.scenario,
|
||||
"backend": backend_name,
|
||||
"n_runs": n_runs,
|
||||
"cost_mode": ev["cost_mode"],
|
||||
"per_system": per_system,
|
||||
"comparisons": ev["comparisons"],
|
||||
"swarm_valid": ev["swarm_valid"],
|
||||
"coverage": _coverage(swarm),
|
||||
"confidence": _confidence(backend_name, n_runs, swarm),
|
||||
"benchmark_agent": {
|
||||
"available": False,
|
||||
"reason": "needs live swarm SwarmMetrics (S_swarm/Reward/Observability) + multi-run on a real backend",
|
||||
},
|
||||
}
|
||||
|
||||
|
||||
def render_markdown(report: dict) -> str:
|
||||
lines: List[str] = []
|
||||
a = lines.append
|
||||
a(f"# Benchmark report — {report['scenario']} / `{report['task_set_id']}`")
|
||||
a("")
|
||||
a(f"- backend: **{report['backend']}** · runs: **{report['n_runs']}** · cost mode: {report['cost_mode']}")
|
||||
a(f"- **swarm_valid: {report['swarm_valid']}** · confidence: **{report['confidence']['level']}**")
|
||||
if report["confidence"]["reasons"]:
|
||||
for r in report["confidence"]["reasons"]:
|
||||
a(f" - ⚠️ {r}")
|
||||
a("")
|
||||
a("## Per-system")
|
||||
a("| system | n_agent | completed | quality | cost($) | cost_eff | speed | robustness |")
|
||||
a("|---|---|---|---|---|---|---|---|")
|
||||
for s, v in report["per_system"].items():
|
||||
a(f"| {s} | {v['n_agent']} | {v['completed']} | {v['quality']} | {v['actual_cost_usd']} "
|
||||
f"| {v['cost_efficiency']} | {v['speed']} | {v['robustness']} |")
|
||||
a("")
|
||||
a("## Swarm vs baselines")
|
||||
a("| baseline | G_E | G_E,c | G_E>0 | G_E,c>0 |")
|
||||
a("|---|---|---|---|---|")
|
||||
for c in report["comparisons"]:
|
||||
a(f"| {c['base_system']} | {round(c['g_e'], 4)} | {round(c['g_e_cost'], 4)} "
|
||||
f"| {c['raw_gain_positive']} | {c['cost_normalized_positive']} |")
|
||||
a("")
|
||||
ba = report["benchmark_agent"]
|
||||
a(f"## Benchmark_Agent: {'available' if ba['available'] else 'NOT AVAILABLE'}")
|
||||
if not ba["available"]:
|
||||
a(f"> {ba['reason']}")
|
||||
a("")
|
||||
a("## Coverage")
|
||||
for k, v in report["coverage"].items():
|
||||
a(f"- {'✅' if v else '🔴'} {k}")
|
||||
return "\n".join(lines)
|
||||
@@ -0,0 +1,44 @@
|
||||
"""Benchmark runners (#21): run a shared task set through each system → BenchmarkRunRecord.
|
||||
|
||||
Same backend + same task set for all systems (fairness); only topology differs. `run_all` is the
|
||||
convenience entry the harness uses; `strong_backend` lets the Strong baseline use a costlier/larger
|
||||
model than the others (its definitional difference).
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
from typing import Dict, List, Optional
|
||||
|
||||
from ..baselines import BenchmarkRunRecord
|
||||
from ..tasksets import TaskSet
|
||||
from .backend import ExecutionBackend, GenerationResult, OfflineBackend, OpenAIBackend
|
||||
from .base import BaseRunner, Topology
|
||||
from .single import SingleRunner
|
||||
from .strong import StrongRunner
|
||||
from .chain import ChainRunner
|
||||
from .sub_agent import SubAgentRunner
|
||||
from .swarm import SwarmRunner
|
||||
|
||||
RUNNERS = {
|
||||
"single": SingleRunner,
|
||||
"strong": StrongRunner,
|
||||
"chain": ChainRunner,
|
||||
"sub": SubAgentRunner,
|
||||
"swarm": SwarmRunner,
|
||||
}
|
||||
|
||||
|
||||
def run_all(taskset: TaskSet, backend: ExecutionBackend, *,
|
||||
strong_backend: Optional[ExecutionBackend] = None) -> Dict[str, BenchmarkRunRecord]:
|
||||
"""Run every system on the task set; return {system: record}. swarm + 4 baselines."""
|
||||
out: Dict[str, BenchmarkRunRecord] = {}
|
||||
for system, runner_cls in RUNNERS.items():
|
||||
be = strong_backend if (system == "strong" and strong_backend is not None) else backend
|
||||
out[system] = runner_cls().run(taskset, be)
|
||||
return out
|
||||
|
||||
|
||||
__all__ = [
|
||||
"BenchmarkRunRecord", "ExecutionBackend", "GenerationResult", "OfflineBackend", "OpenAIBackend",
|
||||
"BaseRunner", "Topology", "SingleRunner", "StrongRunner", "ChainRunner", "SubAgentRunner",
|
||||
"SwarmRunner", "RUNNERS", "run_all",
|
||||
]
|
||||
@@ -0,0 +1,115 @@
|
||||
"""Execution backends for benchmark runners — the SHARED model gateway all systems run through.
|
||||
|
||||
Fairness (standard §9.1): swarm and every baseline must use the same backend on the same task
|
||||
set; only the topology (call pattern) differs. Two backends:
|
||||
|
||||
- OfflineBackend: deterministic, no network, no key. Echoes the fixture's offline reference
|
||||
solution and charges a notional per-call cost. Used for hermetic CI + pipeline validation.
|
||||
It is IDENTICAL across systems, so offline runs can prove the pipeline (records → G_E →
|
||||
report, no NaN) but CANNOT show a swarm quality advantage — by design (anti-fabrication).
|
||||
|
||||
- OpenAIBackend: real OpenAI-compatible generation (needs OPENAI_API_KEY / OPENAI_BASE_URL).
|
||||
Produces real files + real token/cost. This is what yields REAL G_E numbers.
|
||||
|
||||
`generate()` returns the produced implementation files plus measured token/cost/time.
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
import json
|
||||
import os
|
||||
import re
|
||||
from abc import ABC, abstractmethod
|
||||
from dataclasses import dataclass, field
|
||||
from typing import List, Optional, Tuple
|
||||
|
||||
# (path, content) pairs.
|
||||
Files = List[Tuple[str, str]]
|
||||
|
||||
|
||||
@dataclass
|
||||
class GenerationResult:
|
||||
files: Files = field(default_factory=list)
|
||||
tokens: int = 0
|
||||
cost_usd: float = 0.0
|
||||
elapsed_s: float = 0.0
|
||||
|
||||
|
||||
class ExecutionBackend(ABC):
|
||||
name: str = "abstract"
|
||||
|
||||
@abstractmethod
|
||||
def generate(self, *, objective: str, reference_files: Files, max_tokens: int) -> GenerationResult:
|
||||
"""Produce implementation files for one objective."""
|
||||
|
||||
|
||||
class OfflineBackend(ExecutionBackend):
|
||||
"""Deterministic, key-free. Echoes the fixture reference solution; charges a notional cost.
|
||||
|
||||
cost_per_call / tokens_per_call / time_per_call let the harness model a 'stronger' (costlier)
|
||||
configuration without a real model — see runners/strong.py.
|
||||
"""
|
||||
name = "offline"
|
||||
|
||||
def __init__(self, *, cost_per_call: float = 0.002, tokens_per_call: int = 800,
|
||||
time_per_call: float = 0.5):
|
||||
self.cost_per_call = cost_per_call
|
||||
self.tokens_per_call = tokens_per_call
|
||||
self.time_per_call = time_per_call
|
||||
|
||||
def generate(self, *, objective: str, reference_files: Files, max_tokens: int) -> GenerationResult:
|
||||
# Deterministic: same reference output regardless of system/topology (no swarm bias).
|
||||
return GenerationResult(
|
||||
files=list(reference_files),
|
||||
tokens=self.tokens_per_call,
|
||||
cost_usd=self.cost_per_call,
|
||||
elapsed_s=self.time_per_call,
|
||||
)
|
||||
|
||||
|
||||
class OpenAIBackend(ExecutionBackend):
|
||||
"""Real OpenAI-compatible generation. Lazy client; raises clearly if no key is configured."""
|
||||
name = "openai"
|
||||
|
||||
def __init__(self, *, model: Optional[str] = None, price_per_1k_tokens: float = 0.0):
|
||||
self.model = model or os.getenv("OPENAI_MODEL", "gpt-4o-mini")
|
||||
self.price_per_1k_tokens = price_per_1k_tokens
|
||||
self._client = None
|
||||
|
||||
def _get_client(self):
|
||||
if self._client is None:
|
||||
from openai import OpenAI # lazy: keep offline/CI import-light
|
||||
if not os.getenv("OPENAI_API_KEY"):
|
||||
raise RuntimeError("OpenAIBackend requires OPENAI_API_KEY (use OfflineBackend for hermetic runs)")
|
||||
self._client = OpenAI(base_url=os.getenv("OPENAI_BASE_URL") or None)
|
||||
return self._client
|
||||
|
||||
@staticmethod
|
||||
def _parse_files(content: str) -> Files:
|
||||
# Accept a JSON {"files":[{"path","content"}]} (same shape the agent uses), else empty.
|
||||
try:
|
||||
text = re.sub(r"^```(json)?|```$", "", content.strip(), flags=re.MULTILINE).strip()
|
||||
data = json.loads(text)
|
||||
return [(f["path"], f["content"]) for f in data.get("files", []) if f.get("path")]
|
||||
except Exception:
|
||||
return []
|
||||
|
||||
def generate(self, *, objective: str, reference_files: Files, max_tokens: int) -> GenerationResult:
|
||||
import time as _time
|
||||
client = self._get_client()
|
||||
prompt = (
|
||||
f"{objective}\n\nReturn ONLY JSON: "
|
||||
'{"files":[{"path":"relative/path.py","content":"complete file content"}]}'
|
||||
)
|
||||
start = _time.time()
|
||||
resp = client.chat.completions.create(
|
||||
model=self.model,
|
||||
messages=[{"role": "user", "content": prompt}],
|
||||
max_tokens=max_tokens,
|
||||
)
|
||||
elapsed = _time.time() - start
|
||||
content = resp.choices[0].message.content or ""
|
||||
usage = getattr(resp, "usage", None)
|
||||
tokens = int(getattr(usage, "total_tokens", 0) or 0)
|
||||
cost = tokens / 1000.0 * self.price_per_1k_tokens
|
||||
return GenerationResult(files=self._parse_files(content), tokens=tokens,
|
||||
cost_usd=cost, elapsed_s=elapsed)
|
||||
@@ -0,0 +1,125 @@
|
||||
"""Shared runner: run a task set through a backend with a given topology → BenchmarkRunRecord.
|
||||
|
||||
Each system (single/strong/chain/sub_agent/swarm) is the same pipeline with a different topology
|
||||
(how many model calls, how many agents, whether a review round happens). All systems grade the
|
||||
SAME generated deliverable against the fixture's HELD-OUT tests in the Group B sandbox, so quality
|
||||
is measured identically and fairly.
|
||||
|
||||
The deliverable graded is the LAST generation's implementation files (test files the system may
|
||||
emit are excluded — held-out fixture tests are the grader). completed = produced gradeable impl
|
||||
files; test_pass_rate = mean over tasks; CodeReview/UserAcceptance are NOT collected (None →
|
||||
masked in quality()).
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
import os
|
||||
from dataclasses import dataclass
|
||||
|
||||
from orchestrator.sandbox import SandboxFile, run_tests
|
||||
|
||||
from ..baselines import BenchmarkRunRecord
|
||||
from ..fixtures import load_reference_solution
|
||||
from ..tasksets import TaskSet
|
||||
from .backend import ExecutionBackend, Files
|
||||
|
||||
|
||||
def _is_test_file(path: str) -> bool:
|
||||
base = os.path.basename(path or "")
|
||||
return base.startswith("test_") and base.endswith(".py")
|
||||
|
||||
|
||||
@dataclass
|
||||
class Topology:
|
||||
"""How a system uses the backend (the only thing that differs between systems)."""
|
||||
system: str # swarm | single | chain | sub | strong
|
||||
n_agent: int # agents this system uses (for cost-normalized gain)
|
||||
calls_per_task: int # backend generations per task (drives cost/time/tokens)
|
||||
review_rounds: int = 0 # extra grade+regenerate rounds (swarm/strong-ish); 0 = none
|
||||
|
||||
|
||||
class BaseRunner:
|
||||
"""Run a task set under one topology and emit a unified BenchmarkRunRecord."""
|
||||
|
||||
topology: Topology
|
||||
|
||||
def __init__(self, max_tokens: int = 1500):
|
||||
self.max_tokens = max_tokens
|
||||
|
||||
def run(self, taskset: TaskSet, backend: ExecutionBackend) -> BenchmarkRunRecord:
|
||||
topo = self.topology
|
||||
total = len(taskset.tasks)
|
||||
completed = 0
|
||||
total_tokens = 0
|
||||
total_cost = 0.0
|
||||
total_time = 0.0
|
||||
target_time = 0.0
|
||||
pass_rates: list[float] = []
|
||||
recovered = 0
|
||||
failures = 0
|
||||
|
||||
for item in taskset.tasks:
|
||||
fixture = item.fixture
|
||||
target_time += float(fixture.target_time_seconds or 0.0)
|
||||
reference: Files = [(f.path, f.content) for f in load_reference_solution(fixture.id)]
|
||||
test_files = [SandboxFile(t.path, t.content) for t in fixture.test_files]
|
||||
|
||||
# Topology: N generation calls (sequential stages / sub-agents / specialists). The last
|
||||
# call's files are the deliverable; earlier calls still cost tokens/time/$.
|
||||
gen = None
|
||||
for _ in range(max(1, topo.calls_per_task)):
|
||||
gen = backend.generate(objective=fixture.objective, reference_files=reference,
|
||||
max_tokens=self.max_tokens)
|
||||
total_tokens += gen.tokens
|
||||
total_cost += gen.cost_usd
|
||||
total_time += gen.elapsed_s
|
||||
|
||||
impl = [SandboxFile(p, c) for p, c in (gen.files if gen else []) if not _is_test_file(p)]
|
||||
pass_rate = 0.0
|
||||
if impl and test_files:
|
||||
result = run_tests(impl, test_files)
|
||||
pass_rate = result.pass_rate if result.pass_rate is not None else 0.0
|
||||
if gen and gen.files:
|
||||
completed += 1
|
||||
pass_rates.append(pass_rate)
|
||||
|
||||
# Review rounds (swarm): if the deliverable failed, grade-and-regenerate up to N times.
|
||||
# Offline the reference always passes, so this is exercised mainly under a real backend;
|
||||
# each round is counted as a recovered failure when it lifts the pass rate.
|
||||
rounds = 0
|
||||
while rounds < topo.review_rounds and pass_rate < 100.0:
|
||||
failures += 1
|
||||
gen = backend.generate(objective=fixture.objective, reference_files=reference,
|
||||
max_tokens=self.max_tokens)
|
||||
total_tokens += gen.tokens
|
||||
total_cost += gen.cost_usd
|
||||
total_time += gen.elapsed_s
|
||||
impl = [SandboxFile(p, c) for p, c in gen.files if not _is_test_file(p)]
|
||||
if impl and test_files:
|
||||
new_rate = run_tests(impl, test_files).pass_rate or 0.0
|
||||
if new_rate > pass_rate:
|
||||
recovered += 1
|
||||
pass_rates[-1] = new_rate
|
||||
pass_rate = new_rate
|
||||
rounds += 1
|
||||
|
||||
test_pass_rate = sum(pass_rates) / len(pass_rates) if pass_rates else 0.0
|
||||
# Budget口径: a flat planned budget per call across topology (fair: same per-call budget).
|
||||
budget = max(total_cost, topo.calls_per_task * total * 0.0) or 1.0
|
||||
return BenchmarkRunRecord(
|
||||
system=topo.system,
|
||||
scenario=taskset.scenario,
|
||||
task_set_id=taskset.task_set_id,
|
||||
n_agent=topo.n_agent,
|
||||
completed_tasks=completed,
|
||||
total_tasks=total,
|
||||
test_pass_rate=test_pass_rate,
|
||||
code_review_score=None, # not collected (no reviewer wired)
|
||||
user_acceptance=None, # not collected (external signal)
|
||||
budget_usd=round(budget * 1.5, 6), # planned = 1.5× measured (headroom), same rule for all
|
||||
actual_cost_usd=round(total_cost, 6),
|
||||
model_tokens=total_tokens,
|
||||
target_time_s=target_time or 1.0,
|
||||
actual_time_s=round(total_time, 6) or 1.0,
|
||||
recovered_failures=recovered,
|
||||
total_failures=failures,
|
||||
)
|
||||
@@ -0,0 +1,9 @@
|
||||
"""Baseline B — Chain Agent: sequential pipeline (implement → test → document), NO swarm
|
||||
collaboration and NO review loop (standard §9.1). 3 sequential stages = 3 agents, 3 calls."""
|
||||
from __future__ import annotations
|
||||
|
||||
from .base import BaseRunner, Topology
|
||||
|
||||
|
||||
class ChainRunner(BaseRunner):
|
||||
topology = Topology(system="chain", n_agent=3, calls_per_task=3, review_rounds=0)
|
||||
@@ -0,0 +1,8 @@
|
||||
"""Baseline A — Single Agent: one agent, one generation per task, no review (standard §9.1)."""
|
||||
from __future__ import annotations
|
||||
|
||||
from .base import BaseRunner, Topology
|
||||
|
||||
|
||||
class SingleRunner(BaseRunner):
|
||||
topology = Topology(system="single", n_agent=1, calls_per_task=1, review_rounds=0)
|
||||
@@ -0,0 +1,14 @@
|
||||
"""Baseline D — Strong Agent: a single high-capability agent (bigger model / longer context).
|
||||
|
||||
Same topology as Single (1 agent, 1 call, no review); the difference is the BACKEND — the harness
|
||||
gives StrongRunner a stronger, costlier backend (real: a larger model; offline: higher per-call
|
||||
cost). So it should reach similar/better quality at higher cost than Single — the hardest baseline
|
||||
to beat (reference coefficient ×0.95).
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
from .base import BaseRunner, Topology
|
||||
|
||||
|
||||
class StrongRunner(BaseRunner):
|
||||
topology = Topology(system="strong", n_agent=1, calls_per_task=1, review_rounds=0)
|
||||
@@ -0,0 +1,11 @@
|
||||
"""Baseline C — Sub-Agent: supervisor delegates to sub-agents (hierarchical), NO peer
|
||||
collaboration and NO review/redo loop (standard §9.1). 1 supervisor (decompose) + 3 sub-agents
|
||||
= 4 agents, 4 calls. Reference coefficient ×0.90 — the closest baseline to the swarm topology, so
|
||||
the most important to out-perform on G_E."""
|
||||
from __future__ import annotations
|
||||
|
||||
from .base import BaseRunner, Topology
|
||||
|
||||
|
||||
class SubAgentRunner(BaseRunner):
|
||||
topology = Topology(system="sub", n_agent=4, calls_per_task=4, review_rounds=0)
|
||||
@@ -0,0 +1,30 @@
|
||||
"""Experimental — Swarm: the DECENTRALIZED self-organizing flow (post-rework).
|
||||
|
||||
Models the live swarm's call/cost structure (see docs/swarm/decentralized-rework-plan.md):
|
||||
seed (1) → agents self-select + bottom-up DECOMPOSE into subtasks (proposals) → specialists
|
||||
EXECUTE → peer CROSS-REVIEW (>=2 reviewers) → CONVERGE → synthesize.
|
||||
Per task-set item: 1 seed-perceive + ~3 proposed specialist subtask generations + a synthesis,
|
||||
plus 1 cross-review pass (>=2 independent reviewers). That is materially MORE model work than a
|
||||
single agent — the swarm trades cost for collaboration/validation, which is exactly what G_E,c is
|
||||
meant to weigh.
|
||||
|
||||
`calls_per_task=6` (seed + ~3 proposed specialist subtasks + synthesis + a cross-review pass) and
|
||||
`review_rounds=1` (the cross-review redo cycle); `n_agent=3` specialists. These reflect the
|
||||
DECENTRALIZED topology, NOT the deleted Master decompose→dispatch→single-critic flow.
|
||||
|
||||
WHY a topology model and not the live orchestrator: a FAIR G_E requires every system (this swarm +
|
||||
all baselines) to run the SAME task set through the SAME execution backend; only the topology may
|
||||
differ. Driving the live orchestrator would execute via the WS agents (a different backend), making
|
||||
its record incomparable to the offline baselines. A live-orchestrator + SwarmRunMetricsCollector
|
||||
record is faithful but only comparable against OTHER live runs — tracked as a follow-on. The
|
||||
end-to-end decentralized flow itself is exercised by scripts/test-workflow-e2e.py.
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
from .base import BaseRunner, Topology
|
||||
|
||||
|
||||
class SwarmRunner(BaseRunner):
|
||||
# Decentralized flow: seed + bottom-up decomposition + specialist execution + peer cross-review
|
||||
# + synthesis — more calls than any baseline (the cost of collaboration/validation).
|
||||
topology = Topology(system="swarm", n_agent=3, calls_per_task=6, review_rounds=1)
|
||||
@@ -0,0 +1,24 @@
|
||||
# benchmark/tasksets/ —— 统一基准任务集（#20）
|
||||
|
||||
蜂群与全部基线在**同一 `task_set_id`、同一模型网关**下运行同一组任务，才可比（标准 §9.1）。
|
||||
每个任务引用一个 fixture（目标 + 留出验收测试 + 离线参考解）。
|
||||
|
||||
## 实现状态（诚实覆盖）
|
||||
|
||||
| 场景 scenario | task set | 状态 |
|
||||
|---|---|---|
|
||||
| coding | `coding-set-1`（`add_function`） | 🟡 1 个真实任务（示例规模） |
|
||||
| refactoring | — | 🔴 未建（缺 fixture + 留出测试） |
|
||||
| architecture | — | 🔴 未建 |
|
||||
| devops | — | 🔴 未建 |
|
||||
| bugfix | — | 🔴 未建 |
|
||||
|
||||
> 只有带真实 fixture 的场景才可加载；**不发布空任务集**（空集会「静默通过」，违反规则 #9）。
|
||||
> 扩充 = 新增 `benchmark/fixtures/<id>/`（含 `tests/test_*.py` 留出测试 + 可选 `reference_solution/`）
|
||||
> 后，在对应 `benchmark/tasksets/<set>/taskset.json` 引用。
|
||||
|
||||
## 与验收的关系
|
||||
|
||||
`G_E>0` / `Benchmark_Agent` 验收要求在**冻结的、足够规模的**任务集上、用**真实模型**多次运行
|
||||
（见 `docs/benchmark/baseline-comparison.md`）。当前任务集为**示例规模**，仅用于打通管线，
|
||||
**不构成验收**。
|
||||
@@ -0,0 +1,68 @@
|
||||
"""Unified benchmark task sets (#20) — the shared, frozen tasks every system runs.
|
||||
|
||||
Fairness rule (standard §9.1 / baseline-comparison §3): swarm and every baseline run the SAME
|
||||
task_set_id under the SAME model gateway. A task set groups tasks of one scenario; each task
|
||||
references a fixture (objective + HELD-OUT acceptance tests + offline reference solution).
|
||||
|
||||
Layout: benchmark/tasksets/<task_set_id>/taskset.json
|
||||
|
||||
`taskset.json`:
|
||||
{
|
||||
"task_set_id": "coding-set-1",
|
||||
"scenario": "coding", # coding | refactoring | architecture | devops | bugfix
|
||||
"tasks": [ { "id": "add_function", "fixture": "add_function" } ]
|
||||
}
|
||||
|
||||
Coverage is honest: only scenarios with real fixtures are loadable. The other four scenarios
|
||||
(refactoring/architecture/devops/bugfix) are declared TODO in this package's README until their
|
||||
fixtures + held-out tests exist — we do not ship empty task sets that would silently pass.
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
import json
|
||||
from dataclasses import dataclass, field
|
||||
from pathlib import Path
|
||||
from typing import List
|
||||
|
||||
from ..fixtures import Fixture, load_fixture
|
||||
|
||||
_TASKSET_ROOT = Path(__file__).resolve().parent
|
||||
SCENARIOS = {"coding", "refactoring", "architecture", "devops", "bugfix"}
|
||||
|
||||
|
||||
@dataclass
|
||||
class TaskSetItem:
|
||||
id: str
|
||||
fixture_id: str
|
||||
|
||||
@property
|
||||
def fixture(self) -> Fixture:
|
||||
return load_fixture(self.fixture_id)
|
||||
|
||||
|
||||
@dataclass
|
||||
class TaskSet:
|
||||
task_set_id: str
|
||||
scenario: str
|
||||
tasks: List[TaskSetItem] = field(default_factory=list)
|
||||
|
||||
|
||||
def available_tasksets() -> List[str]:
|
||||
return sorted(
|
||||
p.name for p in _TASKSET_ROOT.iterdir()
|
||||
if p.is_dir() and (p / "taskset.json").exists()
|
||||
)
|
||||
|
||||
|
||||
def load_taskset(task_set_id: str) -> TaskSet:
|
||||
meta_path = _TASKSET_ROOT / task_set_id / "taskset.json"
|
||||
if not meta_path.exists():
|
||||
raise FileNotFoundError(f"unknown task set: {task_set_id}")
|
||||
meta = json.loads(meta_path.read_text(encoding="utf-8"))
|
||||
scenario = meta.get("scenario", "")
|
||||
if scenario not in SCENARIOS:
|
||||
raise ValueError(f"task set {task_set_id} has unknown scenario '{scenario}'")
|
||||
tasks = [TaskSetItem(id=t["id"], fixture_id=t["fixture"]) for t in meta.get("tasks", [])]
|
||||
if not tasks:
|
||||
raise ValueError(f"task set {task_set_id} has no tasks (refusing to ship an empty set)")
|
||||
return TaskSet(task_set_id=meta["task_set_id"], scenario=scenario, tasks=tasks)
|
||||
@@ -0,0 +1,7 @@
|
||||
{
|
||||
"task_set_id": "coding-set-1",
|
||||
"scenario": "coding",
|
||||
"tasks": [
|
||||
{ "id": "add_function", "fixture": "add_function" }
|
||||
]
|
||||
}
|
||||
@@ -59,7 +59,7 @@
|
||||
2. **中成本**：~~`reward`~~（✅ 已关闭，本仓）— `Q_quality` 主输入已由 fixture 沙箱采集；剩余为**扩充统一任务集** + 接入 `CodeReview`/`UserAcceptance` 评审/验收信号（部分需 Product/Manager）。
|
||||
3. **高成本**：
|
||||
- ~~`p_decision`~~（✅ 已关闭，本仓，Option A）— `τ/η/P` 决策引擎与信息素历史库已落地（`ENABLE_ACO_DISPATCH` 门控）；剩余为 Option B 双边匹配与决策质量验证（依赖 Group C）。
|
||||
- `gain`（及由其驱动的 `Benchmark_Agent`、`G_E,c`、「Swarm > baselines」验收）— 实现 4 类基线运行器 + 统一数据集 + 对比/显著性，属跨团队与基础设施工作（见 baseline-comparison）。**这是最后一块，也是唯一动验收的一块。**
|
||||
- `gain`（及由其驱动的 `Benchmark_Agent`、`G_E,c`、「Swarm > baselines」验收）— 🟡 **基线对比管线已落地**（`benchmark/runners/` + `tasksets/` + `reports/` + `replay/`，#21/#22），CI offline smoke 跑通；但 **Offline 仅验证管线（G_E=0、无偏）**，真实 `G_E>0` 仍需 `--backend openai` + 足量冻结任务集 + 多次运行（方差/显著性）+ Quality 评审/验收输入。**这是最后一块，也是唯一动验收的一块**（见 baseline-runners.md）。
|
||||
|
||||
## 6. 影响
|
||||
|
||||
|
||||
@@ -55,10 +55,16 @@
|
||||
|
||||
| 组件 | 位置 | 状态 |
|
||||
|---|---|---|
|
||||
| 基线运行器 A–D | `benchmark/baselines/` | 🔴 未实现 |
|
||||
| 统一任务集 / 数据集 | `test-data/` + `benchmark/baselines/` | 🔴 未实现 |
|
||||
| 指标采集 | `benchmark/collectors/` | 🔴 未实现 |
|
||||
| 对比与显著性 | `benchmark/` | 🔴 未实现 |
|
||||
| 基线运行器 A–D + Swarm | `benchmark/runners/`（single/strong/chain/sub_agent/swarm + base/backend） | 🟡 已实现（管线）；真实数值需模型 key |
|
||||
| 共享执行后端（公平网关） | `benchmark/runners/backend.py`（Offline + OpenAI） | 🟡 Offline 已实现（确定性、无偏）；OpenAI 需 `OPENAI_API_KEY` |
|
||||
| 统一任务集 / 数据集 | `benchmark/tasksets/`（`coding-set-1`） | 🟡 1 个示例任务集；其余场景待建 |
|
||||
| 指标采集（评分） | held-out fixture + 沙箱（Group B）→ runner | 🟡 TestPassRate 真实；CodeReview/UserAcceptance 掩码缺 |
|
||||
| 对比 + 报告 + 回放 | `benchmark/reports/`、`benchmark/replay/`、`baselines/comparison.py` | ✅ 已实现 |
|
||||
| 显著性（多 run 方差） | — | 🔴 未实现（confidence 现按 run 数标注） |
|
||||
|
||||
> ⚠️ **Offline 后端为管线验证**：所有系统拿到同一参考解 → quality 相同 → **G_E=0、swarm_valid=False**，
|
||||
> **刻意不显示蜂群优势**（防造假）。真实 `G_E>0` 需 `--backend openai` + 足量冻结任务集 + 多次运行。
|
||||
> 入口：`scripts/run-benchmark-suite.py`；说明见 [`baseline-runners.md`](./baseline-runners.md)。
|
||||
|
||||
## 6. 待对齐
|
||||
|
||||
|
||||
@@ -1,8 +1,10 @@
|
||||
# Baseline Run Record Schema（基线运行记录 · 共享契约）
|
||||
|
||||
> 状态：**Schema + 比较评估器已落地；记录生产（尤其 Quality）未落地**。
|
||||
> 状态：**Schema + 评估器 + 记录生产（runner）均已落地；Quality 仅 TestPassRate 真实，CodeReview/UserAcceptance 仍缺；真实数值需模型 key**。
|
||||
>
|
||||
> 依据：**Agent 蜂群指标量化与标准 v2.0 §6, §9**。实现：`benchmark/baselines/comparison.py`（`BenchmarkRunRecord` / `compare` / `evaluate`）。配套：[`baseline-comparison.md`](./baseline-comparison.md)、[`emergence-evaluation.md`](./emergence-evaluation.md)、[`cost-normalized-gain.md`](./cost-normalized-gain.md)、[`IMPORTANT-metric-coverage-gaps.md`](./IMPORTANT-metric-coverage-gaps.md)。
|
||||
> 依据：**Agent 蜂群指标量化与标准 v2.0 §6, §9**。实现：`benchmark/baselines/comparison.py`（`BenchmarkRunRecord` / `compare` / `evaluate`）+ `benchmark/runners/`（5 系统产出记录，见 [`baseline-runners.md`](./baseline-runners.md)）。配套：[`baseline-comparison.md`](./baseline-comparison.md)、[`emergence-evaluation.md`](./emergence-evaluation.md)、[`cost-normalized-gain.md`](./cost-normalized-gain.md)、[`IMPORTANT-metric-coverage-gaps.md`](./IMPORTANT-metric-coverage-gaps.md)。
|
||||
>
|
||||
> 更新：`code_review_score`/`user_acceptance` 现为 **Optional**（runner 仅采 TestPassRate 时传 None，`quality()` 掩码归一；None ≠ 0）。
|
||||
|
||||
## 1. 目的
|
||||
|
||||
|
||||
@@ -0,0 +1,59 @@
|
||||
# 基线运行器与基准套件（Benchmark Group C · #21/#22）
|
||||
|
||||
把「swarm vs 基线」从纸面公式变成**可运行的管线**：统一任务集 → 5 个系统各跑一遍 → 统一
|
||||
`BenchmarkRunRecord` → 评估器算 `G_E`/`G_E,c` → 报告 + 回放归档。本文是 Group C 的唯一入口。
|
||||
|
||||
## 1. 管线
|
||||
|
||||
```
|
||||
benchmark/tasksets/<id> ──┐
|
||||
├─ runners.run_all(taskset, backend) ──► {system: BenchmarkRunRecord}
|
||||
共享 ExecutionBackend ──────┘ │（5 系统同后端同任务集，仅拓扑不同）
|
||||
▼
|
||||
reports.build_report ──► G_E / G_E,c / coverage / confidence / swarm_valid
|
||||
▼
|
||||
replay.save_archive ──► records.json / report.json / report.md / meta.json
|
||||
```
|
||||
|
||||
入口：`scripts/run-benchmark-suite.py --taskset coding-set-1 [--backend offline|openai]`。
|
||||
|
||||
## 2. 五个系统（标准 §9.1，仅拓扑不同）
|
||||
|
||||
| system | 文件 | 拓扑（公平：同后端同任务集） | n_agent |
|
||||
|---|---|---|---|
|
||||
| single | `runners/single.py` | 1 次生成，无评审 | 1 |
|
||||
| strong | `runners/strong.py` | 1 次生成，**更强/更贵后端**（×0.95 最难超） | 1 |
|
||||
| chain | `runners/chain.py` | 串行 impl→test→doc，无协作/评审 | 3 |
|
||||
| sub | `runners/sub_agent.py` | 主管分解 + 3 子агент，无对等/评审（×0.90 最近邻） | 4 |
|
||||
| swarm | `runners/swarm.py` | **去中心化**：种子 → 自选 + 自主分解 → 专家执行 → **同伴交叉评审** → 收敛 → 综合（calls=6, review=1） | 3 |
|
||||
|
||||
> swarm 拓扑已对齐**去中心化重构**（播种/自选/自主分解/竞争/交叉评审/收敛，见 `docs/swarm/decentralized-rework-plan.md`），非旧 Master「分解→派发→单评审」。仍用**同一离线后端**建模以保证公平对比；驱动活体编排器会换后端→记录不可比，故另计（见 `runners/swarm.py` 说明），活体全流程由 `scripts/test-workflow-e2e.py` 验证。
|
||||
|
||||
每个系统的产出物用 **fixture 留出测试**在 Group B 沙箱里评分得 `TestPassRate`（权威，非自评）。
|
||||
|
||||
## 3. 执行后端（公平网关，`runners/backend.py`）
|
||||
|
||||
- **OfflineBackend**（默认）：确定性、无 key、无网络。回显 fixture 的 `reference_solution/`，按调用数计名义成本。**对所有系统完全相同** → 管线可验证，但**不可能也不应显示蜂群质量优势**（防造假）。
|
||||
- **OpenAIBackend**：真实 OpenAI 兼容生成（需 `OPENAI_API_KEY`），产出真实文件 + token/成本。**真实 `G_E` 由它产生。**
|
||||
|
||||
## 4. 当前能得到什么 / 不能得到什么
|
||||
|
||||
**能**（Offline，CI 每次跑）：完整管线、5 份非 NaN 记录、`G_E`/`G_E,c`/coverage/confidence、回放归档、Markdown 报告。
|
||||
|
||||
**不能**（诚实）：
|
||||
- Offline `G_E=0`、`swarm_valid=False`——同一参考解，无差异。**这是正确的反造假结果，不是 bug。**
|
||||
- 真实 `G_E>0` 需 `--backend openai` + **足量冻结任务集**（当前仅 1 个示例任务）+ **多次运行求方差**（当前 confidence=none/low）。
|
||||
- `CodeReview`/`UserAcceptance` 未采集（掩码忽略）。
|
||||
- 完整 `Benchmark_Agent` 需**活体 swarm run 的 SwarmMetrics**（S_swarm/Reward/Observability，来自 Group A/B 的 collector）与本套件 `G_E` 的合成——报告里标为 NOT AVAILABLE，**不猜数**。
|
||||
|
||||
## 5. 与验收（#13/Issue #2）的关系
|
||||
|
||||
本套件是 #13 验收的**必要管线**，但**本身不构成验收**。验收要求：真实后端、冻结任务集、多次运行、
|
||||
对**全部** A–D 满足 `G_E>0` 且 `G_E,c>0`、并能排除「靠堆 Agent/Token 变强」。在拿到这些真实结果前，
|
||||
**不得关闭 Issue #2/#13，不得宣称 Agent Swarm 正式验收通过。**
|
||||
|
||||
## 6. 测试
|
||||
|
||||
- `scripts/test-benchmark-runners.py`：5 runner 产出有效记录、offline 质量一致（G_E=0 反造假）、
|
||||
报告/coverage/confidence、回放归档（hermetic）。
|
||||
- `scripts/run-benchmark-suite.py`：CI offline smoke gate（防回归）。
|
||||
@@ -0,0 +1,65 @@
|
||||
"""Benchmark suite harness (#13/#21/#22): run a task set through all 5 systems → report.
|
||||
|
||||
Default backend is OFFLINE (hermetic, key-free) — it validates the whole pipeline but, by design,
|
||||
shows NO swarm quality advantage (every system gets the same reference solution). REAL G_E numbers
|
||||
require `--backend openai` with OPENAI_API_KEY set and a sufficiently large, frozen task set.
|
||||
|
||||
Grading executes generated code in the fail-closed sandbox, so HEICODE_SANDBOX_ISOLATED must be
|
||||
set (only inside an isolated pod / ephemeral CI runner — see security-boundary §8.1).
|
||||
|
||||
Usage:
|
||||
HEICODE_SANDBOX_ISOLATED=1 python scripts/run-benchmark-suite.py --taskset coding-set-1 \
|
||||
[--backend offline|openai] [--archive runs/<label>] [--run-label <label>]
|
||||
"""
|
||||
import argparse
|
||||
import sys
|
||||
from pathlib import Path
|
||||
|
||||
# Markdown rendering uses ✅/🔴/⚠️; force UTF-8 so it prints on a Windows GBK console too.
|
||||
try:
|
||||
sys.stdout.reconfigure(encoding="utf-8")
|
||||
except Exception:
|
||||
pass
|
||||
|
||||
sys.path.insert(0, str(Path(__file__).resolve().parents[1]))
|
||||
|
||||
from benchmark.tasksets import load_taskset, available_tasksets
|
||||
from benchmark.runners import run_all, OfflineBackend, OpenAIBackend
|
||||
from benchmark.reports import build_report, render_markdown
|
||||
from benchmark.replay import save_archive
|
||||
|
||||
|
||||
def main() -> int:
|
||||
ap = argparse.ArgumentParser(description="Run the benchmark suite (swarm + 4 baselines).")
|
||||
ap.add_argument("--taskset", default="coding-set-1", help=f"one of: {available_tasksets()}")
|
||||
ap.add_argument("--backend", choices=["offline", "openai"], default="offline")
|
||||
ap.add_argument("--n-runs", type=int, default=1)
|
||||
ap.add_argument("--archive", default=None, help="directory to write the replay archive into")
|
||||
ap.add_argument("--run-label", default="run-1", help="archive subdir name (deterministic)")
|
||||
args = ap.parse_args()
|
||||
|
||||
taskset = load_taskset(args.taskset)
|
||||
|
||||
if args.backend == "openai":
|
||||
backend = OpenAIBackend(price_per_1k_tokens=0.0006)
|
||||
strong_backend = OpenAIBackend(model="gpt-4o", price_per_1k_tokens=0.005) # bigger model
|
||||
else:
|
||||
backend = OfflineBackend(cost_per_call=0.002)
|
||||
strong_backend = OfflineBackend(cost_per_call=0.01) # "stronger" = costlier
|
||||
|
||||
records = run_all(taskset, backend, strong_backend=strong_backend)
|
||||
report = build_report(records, backend_name=args.backend, n_runs=args.n_runs)
|
||||
|
||||
print(render_markdown(report))
|
||||
if args.archive:
|
||||
path = save_archive(args.archive, args.run_label, records=records, report=report)
|
||||
print(f"\narchived → {path}")
|
||||
|
||||
if args.backend == "offline":
|
||||
print("\nNOTE: offline backend — pipeline validation only, NOT a real comparison "
|
||||
"(no swarm advantage is or should be shown). Use --backend openai for real G_E.")
|
||||
return 0
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
sys.exit(main())
|
||||
@@ -0,0 +1,91 @@
|
||||
"""Test the Group C benchmark runners + report + replay (offline, hermetic).
|
||||
|
||||
Proves the PIPELINE (taskset → 5 runners → records → evaluate → report → archive) end-to-end and,
|
||||
crucially, that offline runs do NOT fabricate a swarm advantage. Real G_E needs a model key.
|
||||
Run from agent_swarm_v6:
|
||||
pip install -r orchestrator/requirements.txt -r agent/requirements.txt
|
||||
python scripts/test-benchmark-runners.py
|
||||
(grading executes code in the sandbox; this test self-confirms isolation, like test-sandbox.py.)
|
||||
"""
|
||||
import math
|
||||
import os
|
||||
import sys
|
||||
import tempfile
|
||||
from pathlib import Path
|
||||
|
||||
# The runners grade generated code in the fail-closed sandbox (orchestrator/sandbox.py). Confirm
|
||||
# isolation for this hermetic test (CI runner is ephemeral) — same discipline as test-sandbox.py.
|
||||
os.environ.setdefault("HEICODE_SANDBOX_ISOLATED", "1")
|
||||
sys.path.insert(0, str(Path(__file__).resolve().parents[1]))
|
||||
|
||||
from benchmark.tasksets import load_taskset
|
||||
from benchmark.runners import run_all, RUNNERS, OfflineBackend
|
||||
from benchmark.baselines import quality, BenchmarkRunRecord
|
||||
from benchmark.reports import build_report, render_markdown
|
||||
from benchmark.replay import save_archive
|
||||
|
||||
failures = []
|
||||
|
||||
|
||||
def check(name, cond):
|
||||
print(("PASS" if cond else "FAIL"), "-", name)
|
||||
if not cond:
|
||||
failures.append(name)
|
||||
|
||||
|
||||
taskset = load_taskset("coding-set-1")
|
||||
backend = OfflineBackend(cost_per_call=0.002)
|
||||
strong_backend = OfflineBackend(cost_per_call=0.01)
|
||||
records = run_all(taskset, backend, strong_backend=strong_backend)
|
||||
|
||||
# --- all 5 systems produced a valid record ---
|
||||
check("5 systems ran", set(records.keys()) == set(RUNNERS.keys()))
|
||||
check("all records are BenchmarkRunRecord", all(isinstance(r, BenchmarkRunRecord) for r in records.values()))
|
||||
check("all share the task_set_id", all(r.task_set_id == "coding-set-1" for r in records.values()))
|
||||
|
||||
# --- quality is REAL (graded by held-out tests), non-NaN ---
|
||||
check("quality non-NaN for every system",
|
||||
all(not math.isnan(quality(r)) for r in records.values()))
|
||||
# offline: reference solution passes the held-out tests -> 100 for all
|
||||
check("offline quality = 100 (held-out tests pass)",
|
||||
all(quality(r) == 100.0 for r in records.values()))
|
||||
|
||||
# --- ANTI-FABRICATION: offline shows NO swarm quality advantage ---
|
||||
check("offline G_E == 0 across baselines (no fabricated gain)",
|
||||
all(quality(records["swarm"]) - quality(records[b]) == 0.0 for b in ("single", "strong", "chain", "sub")))
|
||||
|
||||
# --- topology really differs (cost rises with agents/calls) ---
|
||||
check("swarm costs >= single (more calls)", records["swarm"].actual_cost_usd >= records["single"].actual_cost_usd)
|
||||
check("n_agent differs by topology",
|
||||
records["single"].n_agent == 1 and records["chain"].n_agent == 3 and records["sub"].n_agent == 4)
|
||||
|
||||
# --- report computes G_E / G_E,c / coverage / confidence honestly ---
|
||||
report = build_report(records, backend_name="offline", n_runs=1)
|
||||
check("report has a comparison per baseline", len(report["comparisons"]) == 4)
|
||||
check("swarm_valid is False offline (honest, not fabricated)", report["swarm_valid"] is False)
|
||||
check("G_E computed (not NaN) for each baseline",
|
||||
all(not math.isnan(c["g_e"]) for c in report["comparisons"]))
|
||||
check("confidence = none offline", report["confidence"]["level"] == "none")
|
||||
check("coverage: test_pass real, review/acceptance masked",
|
||||
report["coverage"]["quality.test_pass_rate"] is True
|
||||
and report["coverage"]["quality.code_review_score"] is False
|
||||
and report["coverage"]["quality.user_acceptance"] is False)
|
||||
check("Benchmark_Agent reported NOT available (needs live metrics + real backend)",
|
||||
report["benchmark_agent"]["available"] is False)
|
||||
|
||||
# --- replay archive writes the artifacts ---
|
||||
with tempfile.TemporaryDirectory() as d:
|
||||
path = save_archive(d, "run-1", records=records, report=report)
|
||||
p = Path(path)
|
||||
check("archive wrote records/report/md/meta",
|
||||
(p / "records.json").exists() and (p / "report.json").exists()
|
||||
and (p / "report.md").exists() and (p / "meta.json").exists())
|
||||
|
||||
# --- markdown renders without error ---
|
||||
check("markdown renders", "Benchmark report" in render_markdown(report))
|
||||
|
||||
print()
|
||||
if failures:
|
||||
print(f"{len(failures)} runner check(s) FAILED: {failures}")
|
||||
sys.exit(1)
|
||||
print("all Group C benchmark runner checks passed")
|
||||
Reference in New Issue
Block a user