Songhaoz666andClaude Opus 4.8
baa67350e6
在去中心化重构之上落地 benchmark 对比管线：5 个系统（single/strong/chain/sub_agent/swarm） 跑同一任务集、同一执行后端，产出统一 BenchmarkRunRecord → 评估器算 G_E/G_E,c → 报告 + 回放。 - benchmark/runners/：backend（Offline 确定性 / OpenAI 真实）+ base + 5 个 runner。各 runner 用 held-out fixture 测试在 Group B 沙箱里评分得 TestPassRate（权威，非自评）。 - benchmark/tasksets/：统一任务集 + 加载器（coding-set-1，1 个 fixture）。 - benchmark/reports/、benchmark/replay/：G_E/G_E,c/coverage/confidence + 归档。 - benchmark/baselines/comparison.py：BenchmarkRunRecord 的 CodeReview/UserAcceptance 改为 Optional（掩码归一，未采集即 None，规则 #9）。 - scripts/run-benchmark-suite.py harness + scripts/test-benchmark-runners.py。 与去中心化重构对齐：swarm runner 拓扑已**重指向去中心化流程**（种子→自选→自主分解→竞争→ 同伴交叉评审→收敛，calls=6/review=1），非旧 Master「分解→派发→单评审」。仍用同一离线后端 建模以保证公平对比（驱动活体编排器会换后端→记录不可比；活体全流程由 test-workflow-e2e 验证）。 沙箱适配：runner 评分走 fail-closed 沙箱（#24），故 test + CI 步骤设 HEICODE_SANDBOX_ISOLATED=1 （仅 CI/隔离 Pod）。 影响范围：agent_swarm（benchmark 层 + 测试 + docs + CI）。不碰 orchestrator 编排逻辑、 不改 Manager↔Swarm 契约、不影响 Client/计费/密钥/审计/发布链路。 诚实边界： - **离线后端只验证管线**：所有系统拿同一参考解 → quality 相同 → G_E=0、swarm_valid=False， 刻意不显示蜂群优势（反造假）。真实 G_E>0 需 --backend openai + 足量冻结任务集 + 多次运行。 - 故 Closes #21（运行器 + 统一记录已落地并产出合规非 NaN 记录）；Refs #20（仅 1/5 场景）、 Refs #22（评估器/报告/回放已建，但 Quality 仅 TestPassRate，CodeReview/UserAcceptance 缺）、 Refs #13（验收 EPIC，需真实 run 证明 Swarm>baselines，未满足）。 Closes #21 Refs #20 Refs #22 Refs #13 Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
66 lines
2.8 KiB
Python
66 lines
2.8 KiB
Python
"""Benchmark suite harness (#13/#21/#22): run a task set through all 5 systems → report.
|
|
|
|
Default backend is OFFLINE (hermetic, key-free) — it validates the whole pipeline but, by design,
|
|
shows NO swarm quality advantage (every system gets the same reference solution). REAL G_E numbers
|
|
require `--backend openai` with OPENAI_API_KEY set and a sufficiently large, frozen task set.
|
|
|
|
Grading executes generated code in the fail-closed sandbox, so HEICODE_SANDBOX_ISOLATED must be
|
|
set (only inside an isolated pod / ephemeral CI runner — see security-boundary §8.1).
|
|
|
|
Usage:
|
|
HEICODE_SANDBOX_ISOLATED=1 python scripts/run-benchmark-suite.py --taskset coding-set-1 \
|
|
[--backend offline|openai] [--archive runs/<label>] [--run-label <label>]
|
|
"""
|
|
import argparse
|
|
import sys
|
|
from pathlib import Path
|
|
|
|
# Markdown rendering uses ✅/🔴/⚠️; force UTF-8 so it prints on a Windows GBK console too.
|
|
try:
|
|
sys.stdout.reconfigure(encoding="utf-8")
|
|
except Exception:
|
|
pass
|
|
|
|
sys.path.insert(0, str(Path(__file__).resolve().parents[1]))
|
|
|
|
from benchmark.tasksets import load_taskset, available_tasksets
|
|
from benchmark.runners import run_all, OfflineBackend, OpenAIBackend
|
|
from benchmark.reports import build_report, render_markdown
|
|
from benchmark.replay import save_archive
|
|
|
|
|
|
def main() -> int:
|
|
ap = argparse.ArgumentParser(description="Run the benchmark suite (swarm + 4 baselines).")
|
|
ap.add_argument("--taskset", default="coding-set-1", help=f"one of: {available_tasksets()}")
|
|
ap.add_argument("--backend", choices=["offline", "openai"], default="offline")
|
|
ap.add_argument("--n-runs", type=int, default=1)
|
|
ap.add_argument("--archive", default=None, help="directory to write the replay archive into")
|
|
ap.add_argument("--run-label", default="run-1", help="archive subdir name (deterministic)")
|
|
args = ap.parse_args()
|
|
|
|
taskset = load_taskset(args.taskset)
|
|
|
|
if args.backend == "openai":
|
|
backend = OpenAIBackend(price_per_1k_tokens=0.0006)
|
|
strong_backend = OpenAIBackend(model="gpt-4o", price_per_1k_tokens=0.005) # bigger model
|
|
else:
|
|
backend = OfflineBackend(cost_per_call=0.002)
|
|
strong_backend = OfflineBackend(cost_per_call=0.01) # "stronger" = costlier
|
|
|
|
records = run_all(taskset, backend, strong_backend=strong_backend)
|
|
report = build_report(records, backend_name=args.backend, n_runs=args.n_runs)
|
|
|
|
print(render_markdown(report))
|
|
if args.archive:
|
|
path = save_archive(args.archive, args.run_label, records=records, report=report)
|
|
print(f"\narchived → {path}")
|
|
|
|
if args.backend == "offline":
|
|
print("\nNOTE: offline backend — pipeline validation only, NOT a real comparison "
|
|
"(no swarm advantage is or should be shown). Use --backend openai for real G_E.")
|
|
return 0
|
|
|
|
|
|
if __name__ == "__main__":
|
|
sys.exit(main())
|