benchmark Group C：基线运行器 + 统一 BenchmarkRunRecord + 报告/回放（Closes #21）
在去中心化重构之上落地 benchmark 对比管线：5 个系统（single/strong/chain/sub_agent/swarm） 跑同一任务集、同一执行后端，产出统一 BenchmarkRunRecord → 评估器算 G_E/G_E,c → 报告 + 回放。 - benchmark/runners/：backend（Offline 确定性 / OpenAI 真实）+ base + 5 个 runner。各 runner 用 held-out fixture 测试在 Group B 沙箱里评分得 TestPassRate（权威，非自评）。 - benchmark/tasksets/：统一任务集 + 加载器（coding-set-1，1 个 fixture）。 - benchmark/reports/、benchmark/replay/：G_E/G_E,c/coverage/confidence + 归档。 - benchmark/baselines/comparison.py：BenchmarkRunRecord 的 CodeReview/UserAcceptance 改为 Optional（掩码归一，未采集即 None，规则 #9）。 - scripts/run-benchmark-suite.py harness + scripts/test-benchmark-runners.py。 与去中心化重构对齐：swarm runner 拓扑已**重指向去中心化流程**（种子→自选→自主分解→竞争→ 同伴交叉评审→收敛，calls=6/review=1），非旧 Master「分解→派发→单评审」。仍用同一离线后端 建模以保证公平对比（驱动活体编排器会换后端→记录不可比；活体全流程由 test-workflow-e2e 验证）。 沙箱适配：runner 评分走 fail-closed 沙箱（#24），故 test + CI 步骤设 HEICODE_SANDBOX_ISOLATED=1 （仅 CI/隔离 Pod）。 影响范围：agent_swarm（benchmark 层 + 测试 + docs + CI）。不碰 orchestrator 编排逻辑、 不改 Manager↔Swarm 契约、不影响 Client/计费/密钥/审计/发布链路。 诚实边界： - **离线后端只验证管线**：所有系统拿同一参考解 → quality 相同 → G_E=0、swarm_valid=False， 刻意不显示蜂群优势（反造假）。真实 G_E>0 需 --backend openai + 足量冻结任务集 + 多次运行。 - 故 Closes #21（运行器 + 统一记录已落地并产出合规非 NaN 记录）；Refs #20（仅 1/5 场景）、 Refs #22（评估器/报告/回放已建，但 Quality 仅 TestPassRate，CodeReview/UserAcceptance 缺）、 Refs #13（验收 EPIC，需真实 run 证明 Swarm>baselines，未满足）。 Closes #21 Refs #20 Refs #22 Refs #13 Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This commit is contained in:
Songhaoz666
co-authored by
Claude Opus 4.8
Claude Opus 4.8
parent
8b5eea296a
commit
baa67350e6
+9
-4
@@ -10,12 +10,17 @@
|
||||
| `collectors/` | base `SwarmMetricsCollector` + `SwarmRunMetricsCollector`（无条件计算 `s_completion`/`s_collaboration`/`s_robustness`，有条件计算 `s_cost`/`s_governance`/`s_communication`/`reward`/`tau`/`eta`/`p_decision`；其余标记 NaN + `coverage=False`） | 🟡 部分落地（单次 run 最多 10/15 字段真实可算） |
|
||||
| `fixtures/` | 留出（held-out）任务 fixture + 加载器；`reward` 的 `Q_quality` 评分源 | 🟡 1 个示例 fixture（`add_function`），统一任务集待扩充 |
|
||||
| 决策引擎 | `orchestrator/decision_engine.py`（τ trail + η 评分 + ε-greedy 采样，`ENABLE_ACO_DISPATCH` 门控；见 docs/benchmark/decision-engine.md） | 🟡 Option A 单边已落地；Option B 与决策质量验证待 Group C |
|
||||
| `baselines/` | Single / Chain / Sub-Agent / Strong 基线运行器 | 🔴 未落地 |
|
||||
| `replay/` | 执行回放 | 🔴 未落地 |
|
||||
| `baselines/` | `BenchmarkRunRecord` + `compare`/`evaluate`（G_E/G_E,c） | ✅ 已实现 |
|
||||
| `tasksets/` | 统一基准任务集（`coding-set-1`） | 🟡 1 个示例任务集（#20） |
|
||||
| `runners/` | single/strong/chain/sub_agent/swarm + 共享 backend（Offline/OpenAI） | 🟡 管线已落地；真实数值需 key（#21） |
|
||||
| `reports/` | G_E/G_E,c/coverage/confidence + Markdown | ✅ 已实现（#22 部分） |
|
||||
| `replay/` | 记录/报告归档（records/report/md/meta） | 🟡 最小回放（#22 部分） |
|
||||
| `leaderboard/` | 排行榜聚合 | 🔴 未落地 |
|
||||
|
||||
> 公式可计算 ≠ 能自证。**数据采集（collectors）与基线（baselines）未落地前，不得宣称已具备
|
||||
> Agent Swarm 量化自证能力**（不能证明 `Swarm > Single/Chain/Sub-Agent/Strong`，也无法输出完整 `Benchmark_Agent`）。
|
||||
> 公式可计算 ≠ 能自证。基线运行管线（runners/tasksets/reports/replay）已落地，但 **Offline 仅验证管线
|
||||
> （无偏、G_E=0），真实 `G_E>0` 需 `--backend openai` + 冻结任务集 + 多次运行**。在拿到真实结果前，
|
||||
> **不得宣称已具备 Agent Swarm 量化自证能力**（不能证明 `Swarm > Single/Chain/Sub/Strong`，也无法输出完整
|
||||
> `Benchmark_Agent`）。入口 `scripts/run-benchmark-suite.py`，说明见 `docs/benchmark/baseline-runners.md`。
|
||||
|
||||
## 待落地（按依赖顺序）
|
||||
1. `collectors/`：接入事件流 / Prometheus / OTel / 任务与用量（telemetry-architecture）。
|
||||
|
||||
@@ -9,6 +9,7 @@ docs/benchmark/baseline-record-schema.md). No record → no comparison (we never
|
||||
from __future__ import annotations
|
||||
|
||||
from dataclasses import dataclass
|
||||
from typing import Optional
|
||||
|
||||
from ..metrics import (
|
||||
quality_score, completion_score, speed_score, cost_score, robustness_score,
|
||||
@@ -17,19 +18,25 @@ from ..metrics import (
|
||||
|
||||
SYSTEMS = {"swarm", "single", "chain", "sub", "strong"}
|
||||
|
||||
# Quality sub-inputs that may legitimately be absent (masked-renormalized in quality_score, rule #9).
|
||||
_OPTIONAL_FIELDS = {"code_review_score", "user_acceptance"}
|
||||
|
||||
|
||||
@dataclass
|
||||
class BenchmarkRunRecord:
|
||||
"""Unified per-run record (v2.0 §9 collection). Both swarm and baselines emit this."""
|
||||
"""Unified per-run record (v2.0 §9 collection). Both swarm and baselines emit this.
|
||||
|
||||
CodeReview/UserAcceptance are Optional: a runner that can only measure TestPassRate (the
|
||||
common case today — no reviewer/acceptance signal wired) leaves them None, and quality()
|
||||
renormalizes over the present inputs. None ≠ 0 (never fabricate a score, rule #9).
|
||||
"""
|
||||
system: str # one of SYSTEMS
|
||||
scenario: str # coding | refactoring | architecture | devops | bugfix
|
||||
task_set_id: str # identifies the shared task set (fairness: same id across systems)
|
||||
n_agent: int # agent count used by this system
|
||||
completed_tasks: int
|
||||
total_tasks: int
|
||||
test_pass_rate: float # 0..100 (Quality input)
|
||||
code_review_score: float # 0..100 (Quality input)
|
||||
user_acceptance: float # 0..100 (Quality input)
|
||||
test_pass_rate: float # 0..100 (Quality input — held-out fixture tests)
|
||||
budget_usd: float # planned cost
|
||||
actual_cost_usd: float # measured model cost
|
||||
model_tokens: int
|
||||
@@ -37,15 +44,18 @@ class BenchmarkRunRecord:
|
||||
actual_time_s: float # measured time
|
||||
recovered_failures: int
|
||||
total_failures: int
|
||||
code_review_score: Optional[float] = None # 0..100, None when not collected
|
||||
user_acceptance: Optional[float] = None # 0..100, None when not collected
|
||||
|
||||
@classmethod
|
||||
def from_dict(cls, data: dict) -> "BenchmarkRunRecord":
|
||||
missing = [f for f in cls.__dataclass_fields__ if f not in data]
|
||||
required = [f for f in cls.__dataclass_fields__ if f not in _OPTIONAL_FIELDS]
|
||||
missing = [f for f in required if f not in data]
|
||||
if missing:
|
||||
raise ValueError(f"benchmark record missing required fields: {missing}")
|
||||
if data["system"] not in SYSTEMS:
|
||||
raise ValueError(f"unknown system '{data['system']}' (expected one of {sorted(SYSTEMS)})")
|
||||
return cls(**{f: data[f] for f in cls.__dataclass_fields__})
|
||||
return cls(**{f: data[f] for f in cls.__dataclass_fields__ if f in data})
|
||||
|
||||
|
||||
# --- derived quantities (from a single record) ---
|
||||
|
||||
@@ -86,3 +86,18 @@ def load_fixture(fixture_id: str) -> Fixture:
|
||||
target_time_seconds=meta.get("target_time_seconds"),
|
||||
test_files=tests,
|
||||
)
|
||||
|
||||
|
||||
def load_reference_solution(fixture_id: str) -> List[FixtureTest]:
|
||||
"""Offline reference solution files (path/content), used ONLY by the offline backend.
|
||||
|
||||
Returns [] if the fixture has no reference_solution/ dir. These files are NOT the swarm's
|
||||
work — they exist so hermetic, key-free runs can validate the benchmark pipeline.
|
||||
"""
|
||||
ref_dir = fixture_dir(fixture_id) / "reference_solution"
|
||||
if not ref_dir.is_dir():
|
||||
return []
|
||||
return [
|
||||
FixtureTest(path=p.name, content=p.read_text(encoding="utf-8"))
|
||||
for p in sorted(ref_dir.glob("*.py"))
|
||||
]
|
||||
|
||||
@@ -0,0 +1,11 @@
|
||||
"""Offline reference solution for the add_function fixture.
|
||||
|
||||
Used ONLY by the offline execution backend (benchmark/runners/backend.py) for hermetic,
|
||||
key-free pipeline validation. It is NOT the swarm's work and is NOT used when a real model
|
||||
backend runs. Every system gets the same reference output offline, so offline runs prove the
|
||||
benchmark pipeline (records → G_E → report, no NaN) WITHOUT fabricating any swarm advantage.
|
||||
"""
|
||||
|
||||
|
||||
def add(a, b):
|
||||
return a + b
|
||||
@@ -1,5 +1,40 @@
|
||||
"""Benchmark replay — 回放一次 swarm/benchmark 执行。
|
||||
"""Benchmark replay archive (#22, minimal): persist a run's records + report for later inspection.
|
||||
|
||||
状态：**未落地（骨架）**。事件流已按 swarm_id 顺序持久化，可作为回放数据源；
|
||||
需定义快照格式与回放接口。见 docs/integration/audit-trace-schema.md §4。
|
||||
Writes, under <out_dir>/<run_label>/:
|
||||
- records.json : every system's BenchmarkRunRecord (the raw, comparable inputs)
|
||||
- report.json : the computed report (G_E / G_E,c / coverage / confidence)
|
||||
- report.md : human-readable rendering
|
||||
- meta.json : run metadata (backend, task_set_id, run_label, + any extra passed in)
|
||||
|
||||
Honest scope: this archives the BENCHMARK layer (records/report/metadata). A full audit replay
|
||||
(prompts, model versions, per-step traces) belongs to docs/integration/audit-trace-schema.md and
|
||||
is NOT claimed here. No timestamps are generated internally — the caller passes run_label so the
|
||||
archive is deterministic and testable.
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
import json
|
||||
from dataclasses import asdict
|
||||
from pathlib import Path
|
||||
from typing import Dict, Optional
|
||||
|
||||
from ..baselines import BenchmarkRunRecord
|
||||
from ..reports import render_markdown
|
||||
|
||||
|
||||
def save_archive(out_dir: str, run_label: str, *, records: Dict[str, BenchmarkRunRecord],
|
||||
report: dict, meta: Optional[dict] = None) -> str:
|
||||
base = Path(out_dir) / run_label
|
||||
base.mkdir(parents=True, exist_ok=True)
|
||||
(base / "records.json").write_text(
|
||||
json.dumps({s: asdict(r) for s, r in records.items()}, indent=2), encoding="utf-8")
|
||||
(base / "report.json").write_text(json.dumps(report, indent=2), encoding="utf-8")
|
||||
(base / "report.md").write_text(render_markdown(report), encoding="utf-8")
|
||||
(base / "meta.json").write_text(json.dumps({
|
||||
"run_label": run_label,
|
||||
"task_set_id": report.get("task_set_id"),
|
||||
"backend": report.get("backend"),
|
||||
"n_runs": report.get("n_runs"),
|
||||
**(meta or {}),
|
||||
}, indent=2), encoding="utf-8")
|
||||
return str(base)
|
||||
|
||||
@@ -0,0 +1,126 @@
|
||||
"""Benchmark report (#22): turn per-system BenchmarkRunRecords into G_E / G_E,c / coverage /
|
||||
confidence, plus a human-readable rendering.
|
||||
|
||||
What the report HONESTLY claims:
|
||||
- G_E, G_E,c per baseline + swarm_valid (from benchmark.baselines.evaluate).
|
||||
- per-system unified metrics (completion / quality / cost-efficiency / speed / robustness).
|
||||
- coverage: which Quality inputs were real (TestPassRate) vs masked-absent (CodeReview/UserAccept).
|
||||
- confidence: LOW unless many runs on a real backend (offline/1-run = pipeline validation only).
|
||||
|
||||
What it does NOT fabricate:
|
||||
- The full `Benchmark_Agent` aggregate needs S_swarm (with communication/governance), Reward,
|
||||
Observability — these come from a LIVE swarm run's SwarmMetrics, not from these topology
|
||||
records. So `benchmark_agent` is reported as NOT-AVAILABLE here, with the reason, rather than
|
||||
a guessed number. Wiring the live collector + multi-run + a real backend is the remaining step.
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
import math
|
||||
from typing import Dict, List
|
||||
|
||||
from ..baselines import BenchmarkRunRecord, evaluate, quality, unified_metrics
|
||||
|
||||
|
||||
def _coverage(swarm: BenchmarkRunRecord) -> Dict[str, bool]:
|
||||
return {
|
||||
"completion": True,
|
||||
"quality.test_pass_rate": True,
|
||||
"quality.code_review_score": swarm.code_review_score is not None,
|
||||
"quality.user_acceptance": swarm.user_acceptance is not None,
|
||||
"cost_efficiency": swarm.actual_cost_usd > 0,
|
||||
"robustness": True,
|
||||
"g_e": True, # computed vs baselines below
|
||||
"g_e_cost": True,
|
||||
# Requires a LIVE swarm run's SwarmMetrics (S_swarm components, Reward, Observability) —
|
||||
# not derivable from topology records alone.
|
||||
"benchmark_agent": False,
|
||||
}
|
||||
|
||||
|
||||
def _confidence(backend_name: str, n_runs: int, swarm: BenchmarkRunRecord) -> Dict[str, object]:
|
||||
reasons = []
|
||||
level = "high"
|
||||
if backend_name != "openai":
|
||||
level = "none"
|
||||
reasons.append("offline backend: reference solution echoed, no real model differentiation")
|
||||
if n_runs < 3:
|
||||
level = "low" if level != "none" else level
|
||||
reasons.append(f"only {n_runs} run(s); no variance/significance (need ≥3)")
|
||||
if swarm.code_review_score is None or swarm.user_acceptance is None:
|
||||
reasons.append("Quality partial: CodeReview/UserAcceptance not collected (masked)")
|
||||
return {"level": level, "reasons": reasons}
|
||||
|
||||
|
||||
def build_report(records: Dict[str, BenchmarkRunRecord], *, symmetric_cost: bool = True,
|
||||
backend_name: str = "offline", n_runs: int = 1) -> dict:
|
||||
if "swarm" not in records:
|
||||
raise ValueError("records must include a 'swarm' record")
|
||||
swarm = records["swarm"]
|
||||
baselines = [r for s, r in records.items() if s != "swarm"]
|
||||
ev = evaluate(swarm, baselines, symmetric_cost=symmetric_cost)
|
||||
|
||||
per_system = {}
|
||||
for system, rec in records.items():
|
||||
m = unified_metrics(rec)
|
||||
per_system[system] = {
|
||||
"n_agent": rec.n_agent,
|
||||
"completed": f"{rec.completed_tasks}/{rec.total_tasks}",
|
||||
"quality": round(quality(rec), 4),
|
||||
"actual_cost_usd": rec.actual_cost_usd,
|
||||
"cost_efficiency": round(m["cost_efficiency"], 4),
|
||||
"speed": round(m["speed"], 4),
|
||||
"robustness": round(m["robustness"], 4),
|
||||
}
|
||||
|
||||
return {
|
||||
"task_set_id": swarm.task_set_id,
|
||||
"scenario": swarm.scenario,
|
||||
"backend": backend_name,
|
||||
"n_runs": n_runs,
|
||||
"cost_mode": ev["cost_mode"],
|
||||
"per_system": per_system,
|
||||
"comparisons": ev["comparisons"],
|
||||
"swarm_valid": ev["swarm_valid"],
|
||||
"coverage": _coverage(swarm),
|
||||
"confidence": _confidence(backend_name, n_runs, swarm),
|
||||
"benchmark_agent": {
|
||||
"available": False,
|
||||
"reason": "needs live swarm SwarmMetrics (S_swarm/Reward/Observability) + multi-run on a real backend",
|
||||
},
|
||||
}
|
||||
|
||||
|
||||
def render_markdown(report: dict) -> str:
|
||||
lines: List[str] = []
|
||||
a = lines.append
|
||||
a(f"# Benchmark report — {report['scenario']} / `{report['task_set_id']}`")
|
||||
a("")
|
||||
a(f"- backend: **{report['backend']}** · runs: **{report['n_runs']}** · cost mode: {report['cost_mode']}")
|
||||
a(f"- **swarm_valid: {report['swarm_valid']}** · confidence: **{report['confidence']['level']}**")
|
||||
if report["confidence"]["reasons"]:
|
||||
for r in report["confidence"]["reasons"]:
|
||||
a(f" - ⚠️ {r}")
|
||||
a("")
|
||||
a("## Per-system")
|
||||
a("| system | n_agent | completed | quality | cost($) | cost_eff | speed | robustness |")
|
||||
a("|---|---|---|---|---|---|---|---|")
|
||||
for s, v in report["per_system"].items():
|
||||
a(f"| {s} | {v['n_agent']} | {v['completed']} | {v['quality']} | {v['actual_cost_usd']} "
|
||||
f"| {v['cost_efficiency']} | {v['speed']} | {v['robustness']} |")
|
||||
a("")
|
||||
a("## Swarm vs baselines")
|
||||
a("| baseline | G_E | G_E,c | G_E>0 | G_E,c>0 |")
|
||||
a("|---|---|---|---|---|")
|
||||
for c in report["comparisons"]:
|
||||
a(f"| {c['base_system']} | {round(c['g_e'], 4)} | {round(c['g_e_cost'], 4)} "
|
||||
f"| {c['raw_gain_positive']} | {c['cost_normalized_positive']} |")
|
||||
a("")
|
||||
ba = report["benchmark_agent"]
|
||||
a(f"## Benchmark_Agent: {'available' if ba['available'] else 'NOT AVAILABLE'}")
|
||||
if not ba["available"]:
|
||||
a(f"> {ba['reason']}")
|
||||
a("")
|
||||
a("## Coverage")
|
||||
for k, v in report["coverage"].items():
|
||||
a(f"- {'✅' if v else '🔴'} {k}")
|
||||
return "\n".join(lines)
|
||||
@@ -0,0 +1,44 @@
|
||||
"""Benchmark runners (#21): run a shared task set through each system → BenchmarkRunRecord.
|
||||
|
||||
Same backend + same task set for all systems (fairness); only topology differs. `run_all` is the
|
||||
convenience entry the harness uses; `strong_backend` lets the Strong baseline use a costlier/larger
|
||||
model than the others (its definitional difference).
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
from typing import Dict, List, Optional
|
||||
|
||||
from ..baselines import BenchmarkRunRecord
|
||||
from ..tasksets import TaskSet
|
||||
from .backend import ExecutionBackend, GenerationResult, OfflineBackend, OpenAIBackend
|
||||
from .base import BaseRunner, Topology
|
||||
from .single import SingleRunner
|
||||
from .strong import StrongRunner
|
||||
from .chain import ChainRunner
|
||||
from .sub_agent import SubAgentRunner
|
||||
from .swarm import SwarmRunner
|
||||
|
||||
RUNNERS = {
|
||||
"single": SingleRunner,
|
||||
"strong": StrongRunner,
|
||||
"chain": ChainRunner,
|
||||
"sub": SubAgentRunner,
|
||||
"swarm": SwarmRunner,
|
||||
}
|
||||
|
||||
|
||||
def run_all(taskset: TaskSet, backend: ExecutionBackend, *,
|
||||
strong_backend: Optional[ExecutionBackend] = None) -> Dict[str, BenchmarkRunRecord]:
|
||||
"""Run every system on the task set; return {system: record}. swarm + 4 baselines."""
|
||||
out: Dict[str, BenchmarkRunRecord] = {}
|
||||
for system, runner_cls in RUNNERS.items():
|
||||
be = strong_backend if (system == "strong" and strong_backend is not None) else backend
|
||||
out[system] = runner_cls().run(taskset, be)
|
||||
return out
|
||||
|
||||
|
||||
__all__ = [
|
||||
"BenchmarkRunRecord", "ExecutionBackend", "GenerationResult", "OfflineBackend", "OpenAIBackend",
|
||||
"BaseRunner", "Topology", "SingleRunner", "StrongRunner", "ChainRunner", "SubAgentRunner",
|
||||
"SwarmRunner", "RUNNERS", "run_all",
|
||||
]
|
||||
@@ -0,0 +1,115 @@
|
||||
"""Execution backends for benchmark runners — the SHARED model gateway all systems run through.
|
||||
|
||||
Fairness (standard §9.1): swarm and every baseline must use the same backend on the same task
|
||||
set; only the topology (call pattern) differs. Two backends:
|
||||
|
||||
- OfflineBackend: deterministic, no network, no key. Echoes the fixture's offline reference
|
||||
solution and charges a notional per-call cost. Used for hermetic CI + pipeline validation.
|
||||
It is IDENTICAL across systems, so offline runs can prove the pipeline (records → G_E →
|
||||
report, no NaN) but CANNOT show a swarm quality advantage — by design (anti-fabrication).
|
||||
|
||||
- OpenAIBackend: real OpenAI-compatible generation (needs OPENAI_API_KEY / OPENAI_BASE_URL).
|
||||
Produces real files + real token/cost. This is what yields REAL G_E numbers.
|
||||
|
||||
`generate()` returns the produced implementation files plus measured token/cost/time.
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
import json
|
||||
import os
|
||||
import re
|
||||
from abc import ABC, abstractmethod
|
||||
from dataclasses import dataclass, field
|
||||
from typing import List, Optional, Tuple
|
||||
|
||||
# (path, content) pairs.
|
||||
Files = List[Tuple[str, str]]
|
||||
|
||||
|
||||
@dataclass
|
||||
class GenerationResult:
|
||||
files: Files = field(default_factory=list)
|
||||
tokens: int = 0
|
||||
cost_usd: float = 0.0
|
||||
elapsed_s: float = 0.0
|
||||
|
||||
|
||||
class ExecutionBackend(ABC):
|
||||
name: str = "abstract"
|
||||
|
||||
@abstractmethod
|
||||
def generate(self, *, objective: str, reference_files: Files, max_tokens: int) -> GenerationResult:
|
||||
"""Produce implementation files for one objective."""
|
||||
|
||||
|
||||
class OfflineBackend(ExecutionBackend):
|
||||
"""Deterministic, key-free. Echoes the fixture reference solution; charges a notional cost.
|
||||
|
||||
cost_per_call / tokens_per_call / time_per_call let the harness model a 'stronger' (costlier)
|
||||
configuration without a real model — see runners/strong.py.
|
||||
"""
|
||||
name = "offline"
|
||||
|
||||
def __init__(self, *, cost_per_call: float = 0.002, tokens_per_call: int = 800,
|
||||
time_per_call: float = 0.5):
|
||||
self.cost_per_call = cost_per_call
|
||||
self.tokens_per_call = tokens_per_call
|
||||
self.time_per_call = time_per_call
|
||||
|
||||
def generate(self, *, objective: str, reference_files: Files, max_tokens: int) -> GenerationResult:
|
||||
# Deterministic: same reference output regardless of system/topology (no swarm bias).
|
||||
return GenerationResult(
|
||||
files=list(reference_files),
|
||||
tokens=self.tokens_per_call,
|
||||
cost_usd=self.cost_per_call,
|
||||
elapsed_s=self.time_per_call,
|
||||
)
|
||||
|
||||
|
||||
class OpenAIBackend(ExecutionBackend):
|
||||
"""Real OpenAI-compatible generation. Lazy client; raises clearly if no key is configured."""
|
||||
name = "openai"
|
||||
|
||||
def __init__(self, *, model: Optional[str] = None, price_per_1k_tokens: float = 0.0):
|
||||
self.model = model or os.getenv("OPENAI_MODEL", "gpt-4o-mini")
|
||||
self.price_per_1k_tokens = price_per_1k_tokens
|
||||
self._client = None
|
||||
|
||||
def _get_client(self):
|
||||
if self._client is None:
|
||||
from openai import OpenAI # lazy: keep offline/CI import-light
|
||||
if not os.getenv("OPENAI_API_KEY"):
|
||||
raise RuntimeError("OpenAIBackend requires OPENAI_API_KEY (use OfflineBackend for hermetic runs)")
|
||||
self._client = OpenAI(base_url=os.getenv("OPENAI_BASE_URL") or None)
|
||||
return self._client
|
||||
|
||||
@staticmethod
|
||||
def _parse_files(content: str) -> Files:
|
||||
# Accept a JSON {"files":[{"path","content"}]} (same shape the agent uses), else empty.
|
||||
try:
|
||||
text = re.sub(r"^```(json)?|```$", "", content.strip(), flags=re.MULTILINE).strip()
|
||||
data = json.loads(text)
|
||||
return [(f["path"], f["content"]) for f in data.get("files", []) if f.get("path")]
|
||||
except Exception:
|
||||
return []
|
||||
|
||||
def generate(self, *, objective: str, reference_files: Files, max_tokens: int) -> GenerationResult:
|
||||
import time as _time
|
||||
client = self._get_client()
|
||||
prompt = (
|
||||
f"{objective}\n\nReturn ONLY JSON: "
|
||||
'{"files":[{"path":"relative/path.py","content":"complete file content"}]}'
|
||||
)
|
||||
start = _time.time()
|
||||
resp = client.chat.completions.create(
|
||||
model=self.model,
|
||||
messages=[{"role": "user", "content": prompt}],
|
||||
max_tokens=max_tokens,
|
||||
)
|
||||
elapsed = _time.time() - start
|
||||
content = resp.choices[0].message.content or ""
|
||||
usage = getattr(resp, "usage", None)
|
||||
tokens = int(getattr(usage, "total_tokens", 0) or 0)
|
||||
cost = tokens / 1000.0 * self.price_per_1k_tokens
|
||||
return GenerationResult(files=self._parse_files(content), tokens=tokens,
|
||||
cost_usd=cost, elapsed_s=elapsed)
|
||||
@@ -0,0 +1,125 @@
|
||||
"""Shared runner: run a task set through a backend with a given topology → BenchmarkRunRecord.
|
||||
|
||||
Each system (single/strong/chain/sub_agent/swarm) is the same pipeline with a different topology
|
||||
(how many model calls, how many agents, whether a review round happens). All systems grade the
|
||||
SAME generated deliverable against the fixture's HELD-OUT tests in the Group B sandbox, so quality
|
||||
is measured identically and fairly.
|
||||
|
||||
The deliverable graded is the LAST generation's implementation files (test files the system may
|
||||
emit are excluded — held-out fixture tests are the grader). completed = produced gradeable impl
|
||||
files; test_pass_rate = mean over tasks; CodeReview/UserAcceptance are NOT collected (None →
|
||||
masked in quality()).
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
import os
|
||||
from dataclasses import dataclass
|
||||
|
||||
from orchestrator.sandbox import SandboxFile, run_tests
|
||||
|
||||
from ..baselines import BenchmarkRunRecord
|
||||
from ..fixtures import load_reference_solution
|
||||
from ..tasksets import TaskSet
|
||||
from .backend import ExecutionBackend, Files
|
||||
|
||||
|
||||
def _is_test_file(path: str) -> bool:
|
||||
base = os.path.basename(path or "")
|
||||
return base.startswith("test_") and base.endswith(".py")
|
||||
|
||||
|
||||
@dataclass
|
||||
class Topology:
|
||||
"""How a system uses the backend (the only thing that differs between systems)."""
|
||||
system: str # swarm | single | chain | sub | strong
|
||||
n_agent: int # agents this system uses (for cost-normalized gain)
|
||||
calls_per_task: int # backend generations per task (drives cost/time/tokens)
|
||||
review_rounds: int = 0 # extra grade+regenerate rounds (swarm/strong-ish); 0 = none
|
||||
|
||||
|
||||
class BaseRunner:
|
||||
"""Run a task set under one topology and emit a unified BenchmarkRunRecord."""
|
||||
|
||||
topology: Topology
|
||||
|
||||
def __init__(self, max_tokens: int = 1500):
|
||||
self.max_tokens = max_tokens
|
||||
|
||||
def run(self, taskset: TaskSet, backend: ExecutionBackend) -> BenchmarkRunRecord:
|
||||
topo = self.topology
|
||||
total = len(taskset.tasks)
|
||||
completed = 0
|
||||
total_tokens = 0
|
||||
total_cost = 0.0
|
||||
total_time = 0.0
|
||||
target_time = 0.0
|
||||
pass_rates: list[float] = []
|
||||
recovered = 0
|
||||
failures = 0
|
||||
|
||||
for item in taskset.tasks:
|
||||
fixture = item.fixture
|
||||
target_time += float(fixture.target_time_seconds or 0.0)
|
||||
reference: Files = [(f.path, f.content) for f in load_reference_solution(fixture.id)]
|
||||
test_files = [SandboxFile(t.path, t.content) for t in fixture.test_files]
|
||||
|
||||
# Topology: N generation calls (sequential stages / sub-agents / specialists). The last
|
||||
# call's files are the deliverable; earlier calls still cost tokens/time/$.
|
||||
gen = None
|
||||
for _ in range(max(1, topo.calls_per_task)):
|
||||
gen = backend.generate(objective=fixture.objective, reference_files=reference,
|
||||
max_tokens=self.max_tokens)
|
||||
total_tokens += gen.tokens
|
||||
total_cost += gen.cost_usd
|
||||
total_time += gen.elapsed_s
|
||||
|
||||
impl = [SandboxFile(p, c) for p, c in (gen.files if gen else []) if not _is_test_file(p)]
|
||||
pass_rate = 0.0
|
||||
if impl and test_files:
|
||||
result = run_tests(impl, test_files)
|
||||
pass_rate = result.pass_rate if result.pass_rate is not None else 0.0
|
||||
if gen and gen.files:
|
||||
completed += 1
|
||||
pass_rates.append(pass_rate)
|
||||
|
||||
# Review rounds (swarm): if the deliverable failed, grade-and-regenerate up to N times.
|
||||
# Offline the reference always passes, so this is exercised mainly under a real backend;
|
||||
# each round is counted as a recovered failure when it lifts the pass rate.
|
||||
rounds = 0
|
||||
while rounds < topo.review_rounds and pass_rate < 100.0:
|
||||
failures += 1
|
||||
gen = backend.generate(objective=fixture.objective, reference_files=reference,
|
||||
max_tokens=self.max_tokens)
|
||||
total_tokens += gen.tokens
|
||||
total_cost += gen.cost_usd
|
||||
total_time += gen.elapsed_s
|
||||
impl = [SandboxFile(p, c) for p, c in gen.files if not _is_test_file(p)]
|
||||
if impl and test_files:
|
||||
new_rate = run_tests(impl, test_files).pass_rate or 0.0
|
||||
if new_rate > pass_rate:
|
||||
recovered += 1
|
||||
pass_rates[-1] = new_rate
|
||||
pass_rate = new_rate
|
||||
rounds += 1
|
||||
|
||||
test_pass_rate = sum(pass_rates) / len(pass_rates) if pass_rates else 0.0
|
||||
# Budget口径: a flat planned budget per call across topology (fair: same per-call budget).
|
||||
budget = max(total_cost, topo.calls_per_task * total * 0.0) or 1.0
|
||||
return BenchmarkRunRecord(
|
||||
system=topo.system,
|
||||
scenario=taskset.scenario,
|
||||
task_set_id=taskset.task_set_id,
|
||||
n_agent=topo.n_agent,
|
||||
completed_tasks=completed,
|
||||
total_tasks=total,
|
||||
test_pass_rate=test_pass_rate,
|
||||
code_review_score=None, # not collected (no reviewer wired)
|
||||
user_acceptance=None, # not collected (external signal)
|
||||
budget_usd=round(budget * 1.5, 6), # planned = 1.5× measured (headroom), same rule for all
|
||||
actual_cost_usd=round(total_cost, 6),
|
||||
model_tokens=total_tokens,
|
||||
target_time_s=target_time or 1.0,
|
||||
actual_time_s=round(total_time, 6) or 1.0,
|
||||
recovered_failures=recovered,
|
||||
total_failures=failures,
|
||||
)
|
||||
@@ -0,0 +1,9 @@
|
||||
"""Baseline B — Chain Agent: sequential pipeline (implement → test → document), NO swarm
|
||||
collaboration and NO review loop (standard §9.1). 3 sequential stages = 3 agents, 3 calls."""
|
||||
from __future__ import annotations
|
||||
|
||||
from .base import BaseRunner, Topology
|
||||
|
||||
|
||||
class ChainRunner(BaseRunner):
|
||||
topology = Topology(system="chain", n_agent=3, calls_per_task=3, review_rounds=0)
|
||||
@@ -0,0 +1,8 @@
|
||||
"""Baseline A — Single Agent: one agent, one generation per task, no review (standard §9.1)."""
|
||||
from __future__ import annotations
|
||||
|
||||
from .base import BaseRunner, Topology
|
||||
|
||||
|
||||
class SingleRunner(BaseRunner):
|
||||
topology = Topology(system="single", n_agent=1, calls_per_task=1, review_rounds=0)
|
||||
@@ -0,0 +1,14 @@
|
||||
"""Baseline D — Strong Agent: a single high-capability agent (bigger model / longer context).
|
||||
|
||||
Same topology as Single (1 agent, 1 call, no review); the difference is the BACKEND — the harness
|
||||
gives StrongRunner a stronger, costlier backend (real: a larger model; offline: higher per-call
|
||||
cost). So it should reach similar/better quality at higher cost than Single — the hardest baseline
|
||||
to beat (reference coefficient ×0.95).
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
from .base import BaseRunner, Topology
|
||||
|
||||
|
||||
class StrongRunner(BaseRunner):
|
||||
topology = Topology(system="strong", n_agent=1, calls_per_task=1, review_rounds=0)
|
||||
@@ -0,0 +1,11 @@
|
||||
"""Baseline C — Sub-Agent: supervisor delegates to sub-agents (hierarchical), NO peer
|
||||
collaboration and NO review/redo loop (standard §9.1). 1 supervisor (decompose) + 3 sub-agents
|
||||
= 4 agents, 4 calls. Reference coefficient ×0.90 — the closest baseline to the swarm topology, so
|
||||
the most important to out-perform on G_E."""
|
||||
from __future__ import annotations
|
||||
|
||||
from .base import BaseRunner, Topology
|
||||
|
||||
|
||||
class SubAgentRunner(BaseRunner):
|
||||
topology = Topology(system="sub", n_agent=4, calls_per_task=4, review_rounds=0)
|
||||
@@ -0,0 +1,30 @@
|
||||
"""Experimental — Swarm: the DECENTRALIZED self-organizing flow (post-rework).
|
||||
|
||||
Models the live swarm's call/cost structure (see docs/swarm/decentralized-rework-plan.md):
|
||||
seed (1) → agents self-select + bottom-up DECOMPOSE into subtasks (proposals) → specialists
|
||||
EXECUTE → peer CROSS-REVIEW (>=2 reviewers) → CONVERGE → synthesize.
|
||||
Per task-set item: 1 seed-perceive + ~3 proposed specialist subtask generations + a synthesis,
|
||||
plus 1 cross-review pass (>=2 independent reviewers). That is materially MORE model work than a
|
||||
single agent — the swarm trades cost for collaboration/validation, which is exactly what G_E,c is
|
||||
meant to weigh.
|
||||
|
||||
`calls_per_task=6` (seed + ~3 proposed specialist subtasks + synthesis + a cross-review pass) and
|
||||
`review_rounds=1` (the cross-review redo cycle); `n_agent=3` specialists. These reflect the
|
||||
DECENTRALIZED topology, NOT the deleted Master decompose→dispatch→single-critic flow.
|
||||
|
||||
WHY a topology model and not the live orchestrator: a FAIR G_E requires every system (this swarm +
|
||||
all baselines) to run the SAME task set through the SAME execution backend; only the topology may
|
||||
differ. Driving the live orchestrator would execute via the WS agents (a different backend), making
|
||||
its record incomparable to the offline baselines. A live-orchestrator + SwarmRunMetricsCollector
|
||||
record is faithful but only comparable against OTHER live runs — tracked as a follow-on. The
|
||||
end-to-end decentralized flow itself is exercised by scripts/test-workflow-e2e.py.
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
from .base import BaseRunner, Topology
|
||||
|
||||
|
||||
class SwarmRunner(BaseRunner):
|
||||
# Decentralized flow: seed + bottom-up decomposition + specialist execution + peer cross-review
|
||||
# + synthesis — more calls than any baseline (the cost of collaboration/validation).
|
||||
topology = Topology(system="swarm", n_agent=3, calls_per_task=6, review_rounds=1)
|
||||
@@ -0,0 +1,24 @@
|
||||
# benchmark/tasksets/ —— 统一基准任务集（#20）
|
||||
|
||||
蜂群与全部基线在**同一 `task_set_id`、同一模型网关**下运行同一组任务，才可比（标准 §9.1）。
|
||||
每个任务引用一个 fixture（目标 + 留出验收测试 + 离线参考解）。
|
||||
|
||||
## 实现状态（诚实覆盖）
|
||||
|
||||
| 场景 scenario | task set | 状态 |
|
||||
|---|---|---|
|
||||
| coding | `coding-set-1`（`add_function`） | 🟡 1 个真实任务（示例规模） |
|
||||
| refactoring | — | 🔴 未建（缺 fixture + 留出测试） |
|
||||
| architecture | — | 🔴 未建 |
|
||||
| devops | — | 🔴 未建 |
|
||||
| bugfix | — | 🔴 未建 |
|
||||
|
||||
> 只有带真实 fixture 的场景才可加载；**不发布空任务集**（空集会「静默通过」，违反规则 #9）。
|
||||
> 扩充 = 新增 `benchmark/fixtures/<id>/`（含 `tests/test_*.py` 留出测试 + 可选 `reference_solution/`）
|
||||
> 后，在对应 `benchmark/tasksets/<set>/taskset.json` 引用。
|
||||
|
||||
## 与验收的关系
|
||||
|
||||
`G_E>0` / `Benchmark_Agent` 验收要求在**冻结的、足够规模的**任务集上、用**真实模型**多次运行
|
||||
（见 `docs/benchmark/baseline-comparison.md`）。当前任务集为**示例规模**，仅用于打通管线，
|
||||
**不构成验收**。
|
||||
@@ -0,0 +1,68 @@
|
||||
"""Unified benchmark task sets (#20) — the shared, frozen tasks every system runs.
|
||||
|
||||
Fairness rule (standard §9.1 / baseline-comparison §3): swarm and every baseline run the SAME
|
||||
task_set_id under the SAME model gateway. A task set groups tasks of one scenario; each task
|
||||
references a fixture (objective + HELD-OUT acceptance tests + offline reference solution).
|
||||
|
||||
Layout: benchmark/tasksets/<task_set_id>/taskset.json
|
||||
|
||||
`taskset.json`:
|
||||
{
|
||||
"task_set_id": "coding-set-1",
|
||||
"scenario": "coding", # coding | refactoring | architecture | devops | bugfix
|
||||
"tasks": [ { "id": "add_function", "fixture": "add_function" } ]
|
||||
}
|
||||
|
||||
Coverage is honest: only scenarios with real fixtures are loadable. The other four scenarios
|
||||
(refactoring/architecture/devops/bugfix) are declared TODO in this package's README until their
|
||||
fixtures + held-out tests exist — we do not ship empty task sets that would silently pass.
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
import json
|
||||
from dataclasses import dataclass, field
|
||||
from pathlib import Path
|
||||
from typing import List
|
||||
|
||||
from ..fixtures import Fixture, load_fixture
|
||||
|
||||
_TASKSET_ROOT = Path(__file__).resolve().parent
|
||||
SCENARIOS = {"coding", "refactoring", "architecture", "devops", "bugfix"}
|
||||
|
||||
|
||||
@dataclass
|
||||
class TaskSetItem:
|
||||
id: str
|
||||
fixture_id: str
|
||||
|
||||
@property
|
||||
def fixture(self) -> Fixture:
|
||||
return load_fixture(self.fixture_id)
|
||||
|
||||
|
||||
@dataclass
|
||||
class TaskSet:
|
||||
task_set_id: str
|
||||
scenario: str
|
||||
tasks: List[TaskSetItem] = field(default_factory=list)
|
||||
|
||||
|
||||
def available_tasksets() -> List[str]:
|
||||
return sorted(
|
||||
p.name for p in _TASKSET_ROOT.iterdir()
|
||||
if p.is_dir() and (p / "taskset.json").exists()
|
||||
)
|
||||
|
||||
|
||||
def load_taskset(task_set_id: str) -> TaskSet:
|
||||
meta_path = _TASKSET_ROOT / task_set_id / "taskset.json"
|
||||
if not meta_path.exists():
|
||||
raise FileNotFoundError(f"unknown task set: {task_set_id}")
|
||||
meta = json.loads(meta_path.read_text(encoding="utf-8"))
|
||||
scenario = meta.get("scenario", "")
|
||||
if scenario not in SCENARIOS:
|
||||
raise ValueError(f"task set {task_set_id} has unknown scenario '{scenario}'")
|
||||
tasks = [TaskSetItem(id=t["id"], fixture_id=t["fixture"]) for t in meta.get("tasks", [])]
|
||||
if not tasks:
|
||||
raise ValueError(f"task set {task_set_id} has no tasks (refusing to ship an empty set)")
|
||||
return TaskSet(task_set_id=meta["task_set_id"], scenario=scenario, tasks=tasks)
|
||||
@@ -0,0 +1,7 @@
|
||||
{
|
||||
"task_set_id": "coding-set-1",
|
||||
"scenario": "coding",
|
||||
"tasks": [
|
||||
{ "id": "add_function", "fixture": "add_function" }
|
||||
]
|
||||
}
|
||||
Reference in New Issue
Block a user