benchmark Group C:基线运行器 + 统一 BenchmarkRunRecord + 报告/回放(Closes #21)
在去中心化重构之上落地 benchmark 对比管线:5 个系统(single/strong/chain/sub_agent/swarm) 跑同一任务集、同一执行后端,产出统一 BenchmarkRunRecord → 评估器算 G_E/G_E,c → 报告 + 回放。 - benchmark/runners/:backend(Offline 确定性 / OpenAI 真实)+ base + 5 个 runner。各 runner 用 held-out fixture 测试在 Group B 沙箱里评分得 TestPassRate(权威,非自评)。 - benchmark/tasksets/:统一任务集 + 加载器(coding-set-1,1 个 fixture)。 - benchmark/reports/、benchmark/replay/:G_E/G_E,c/coverage/confidence + 归档。 - benchmark/baselines/comparison.py:BenchmarkRunRecord 的 CodeReview/UserAcceptance 改为 Optional(掩码归一,未采集即 None,规则 #9)。 - scripts/run-benchmark-suite.py harness + scripts/test-benchmark-runners.py。 与去中心化重构对齐:swarm runner 拓扑已**重指向去中心化流程**(种子→自选→自主分解→竞争→ 同伴交叉评审→收敛,calls=6/review=1),非旧 Master「分解→派发→单评审」。仍用同一离线后端 建模以保证公平对比(驱动活体编排器会换后端→记录不可比;活体全流程由 test-workflow-e2e 验证)。 沙箱适配:runner 评分走 fail-closed 沙箱(#24),故 test + CI 步骤设 HEICODE_SANDBOX_ISOLATED=1 (仅 CI/隔离 Pod)。 影响范围:agent_swarm(benchmark 层 + 测试 + docs + CI)。不碰 orchestrator 编排逻辑、 不改 Manager↔Swarm 契约、不影响 Client/计费/密钥/审计/发布链路。 诚实边界: - **离线后端只验证管线**:所有系统拿同一参考解 → quality 相同 → G_E=0、swarm_valid=False, 刻意不显示蜂群优势(反造假)。真实 G_E>0 需 --backend openai + 足量冻结任务集 + 多次运行。 - 故 Closes #21(运行器 + 统一记录已落地并产出合规非 NaN 记录);Refs #20(仅 1/5 场景)、 Refs #22(评估器/报告/回放已建,但 Quality 仅 TestPassRate,CodeReview/UserAcceptance 缺)、 Refs #13(验收 EPIC,需真实 run 证明 Swarm>baselines,未满足)。 Closes #21 Refs #20 Refs #22 Refs #13 Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This commit is contained in:
co-authored by
Claude Opus 4.8
parent
8b5eea296a
commit
baa67350e6
@@ -0,0 +1,65 @@
|
||||
"""Benchmark suite harness (#13/#21/#22): run a task set through all 5 systems → report.
|
||||
|
||||
Default backend is OFFLINE (hermetic, key-free) — it validates the whole pipeline but, by design,
|
||||
shows NO swarm quality advantage (every system gets the same reference solution). REAL G_E numbers
|
||||
require `--backend openai` with OPENAI_API_KEY set and a sufficiently large, frozen task set.
|
||||
|
||||
Grading executes generated code in the fail-closed sandbox, so HEICODE_SANDBOX_ISOLATED must be
|
||||
set (only inside an isolated pod / ephemeral CI runner — see security-boundary §8.1).
|
||||
|
||||
Usage:
|
||||
HEICODE_SANDBOX_ISOLATED=1 python scripts/run-benchmark-suite.py --taskset coding-set-1 \
|
||||
[--backend offline|openai] [--archive runs/<label>] [--run-label <label>]
|
||||
"""
|
||||
import argparse
|
||||
import sys
|
||||
from pathlib import Path
|
||||
|
||||
# Markdown rendering uses ✅/🔴/⚠️; force UTF-8 so it prints on a Windows GBK console too.
|
||||
try:
|
||||
sys.stdout.reconfigure(encoding="utf-8")
|
||||
except Exception:
|
||||
pass
|
||||
|
||||
sys.path.insert(0, str(Path(__file__).resolve().parents[1]))
|
||||
|
||||
from benchmark.tasksets import load_taskset, available_tasksets
|
||||
from benchmark.runners import run_all, OfflineBackend, OpenAIBackend
|
||||
from benchmark.reports import build_report, render_markdown
|
||||
from benchmark.replay import save_archive
|
||||
|
||||
|
||||
def main() -> int:
|
||||
ap = argparse.ArgumentParser(description="Run the benchmark suite (swarm + 4 baselines).")
|
||||
ap.add_argument("--taskset", default="coding-set-1", help=f"one of: {available_tasksets()}")
|
||||
ap.add_argument("--backend", choices=["offline", "openai"], default="offline")
|
||||
ap.add_argument("--n-runs", type=int, default=1)
|
||||
ap.add_argument("--archive", default=None, help="directory to write the replay archive into")
|
||||
ap.add_argument("--run-label", default="run-1", help="archive subdir name (deterministic)")
|
||||
args = ap.parse_args()
|
||||
|
||||
taskset = load_taskset(args.taskset)
|
||||
|
||||
if args.backend == "openai":
|
||||
backend = OpenAIBackend(price_per_1k_tokens=0.0006)
|
||||
strong_backend = OpenAIBackend(model="gpt-4o", price_per_1k_tokens=0.005) # bigger model
|
||||
else:
|
||||
backend = OfflineBackend(cost_per_call=0.002)
|
||||
strong_backend = OfflineBackend(cost_per_call=0.01) # "stronger" = costlier
|
||||
|
||||
records = run_all(taskset, backend, strong_backend=strong_backend)
|
||||
report = build_report(records, backend_name=args.backend, n_runs=args.n_runs)
|
||||
|
||||
print(render_markdown(report))
|
||||
if args.archive:
|
||||
path = save_archive(args.archive, args.run_label, records=records, report=report)
|
||||
print(f"\narchived → {path}")
|
||||
|
||||
if args.backend == "offline":
|
||||
print("\nNOTE: offline backend — pipeline validation only, NOT a real comparison "
|
||||
"(no swarm advantage is or should be shown). Use --backend openai for real G_E.")
|
||||
return 0
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
sys.exit(main())
|
||||
Reference in New Issue
Block a user