Songhaoz666andClaude Opus 4.8
baa67350e6
在去中心化重构之上落地 benchmark 对比管线：5 个系统（single/strong/chain/sub_agent/swarm） 跑同一任务集、同一执行后端，产出统一 BenchmarkRunRecord → 评估器算 G_E/G_E,c → 报告 + 回放。 - benchmark/runners/：backend（Offline 确定性 / OpenAI 真实）+ base + 5 个 runner。各 runner 用 held-out fixture 测试在 Group B 沙箱里评分得 TestPassRate（权威，非自评）。 - benchmark/tasksets/：统一任务集 + 加载器（coding-set-1，1 个 fixture）。 - benchmark/reports/、benchmark/replay/：G_E/G_E,c/coverage/confidence + 归档。 - benchmark/baselines/comparison.py：BenchmarkRunRecord 的 CodeReview/UserAcceptance 改为 Optional（掩码归一，未采集即 None，规则 #9）。 - scripts/run-benchmark-suite.py harness + scripts/test-benchmark-runners.py。 与去中心化重构对齐：swarm runner 拓扑已**重指向去中心化流程**（种子→自选→自主分解→竞争→ 同伴交叉评审→收敛，calls=6/review=1），非旧 Master「分解→派发→单评审」。仍用同一离线后端 建模以保证公平对比（驱动活体编排器会换后端→记录不可比；活体全流程由 test-workflow-e2e 验证）。 沙箱适配：runner 评分走 fail-closed 沙箱（#24），故 test + CI 步骤设 HEICODE_SANDBOX_ISOLATED=1 （仅 CI/隔离 Pod）。 影响范围：agent_swarm（benchmark 层 + 测试 + docs + CI）。不碰 orchestrator 编排逻辑、 不改 Manager↔Swarm 契约、不影响 Client/计费/密钥/审计/发布链路。 诚实边界： - **离线后端只验证管线**：所有系统拿同一参考解 → quality 相同 → G_E=0、swarm_valid=False， 刻意不显示蜂群优势（反造假）。真实 G_E>0 需 --backend openai + 足量冻结任务集 + 多次运行。 - 故 Closes #21（运行器 + 统一记录已落地并产出合规非 NaN 记录）；Refs #20（仅 1/5 场景）、 Refs #22（评估器/报告/回放已建，但 Quality 仅 TestPassRate，CodeReview/UserAcceptance 缺）、 Refs #13（验收 EPIC，需真实 run 证明 Swarm>baselines，未满足）。 Closes #21 Refs #20 Refs #22 Refs #13 Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
41 lines
1.8 KiB
Python
41 lines
1.8 KiB
Python
"""Benchmark replay archive (#22, minimal): persist a run's records + report for later inspection.
|
|
|
|
Writes, under <out_dir>/<run_label>/:
|
|
- records.json : every system's BenchmarkRunRecord (the raw, comparable inputs)
|
|
- report.json : the computed report (G_E / G_E,c / coverage / confidence)
|
|
- report.md : human-readable rendering
|
|
- meta.json : run metadata (backend, task_set_id, run_label, + any extra passed in)
|
|
|
|
Honest scope: this archives the BENCHMARK layer (records/report/metadata). A full audit replay
|
|
(prompts, model versions, per-step traces) belongs to docs/integration/audit-trace-schema.md and
|
|
is NOT claimed here. No timestamps are generated internally — the caller passes run_label so the
|
|
archive is deterministic and testable.
|
|
"""
|
|
from __future__ import annotations
|
|
|
|
import json
|
|
from dataclasses import asdict
|
|
from pathlib import Path
|
|
from typing import Dict, Optional
|
|
|
|
from ..baselines import BenchmarkRunRecord
|
|
from ..reports import render_markdown
|
|
|
|
|
|
def save_archive(out_dir: str, run_label: str, *, records: Dict[str, BenchmarkRunRecord],
|
|
report: dict, meta: Optional[dict] = None) -> str:
|
|
base = Path(out_dir) / run_label
|
|
base.mkdir(parents=True, exist_ok=True)
|
|
(base / "records.json").write_text(
|
|
json.dumps({s: asdict(r) for s, r in records.items()}, indent=2), encoding="utf-8")
|
|
(base / "report.json").write_text(json.dumps(report, indent=2), encoding="utf-8")
|
|
(base / "report.md").write_text(render_markdown(report), encoding="utf-8")
|
|
(base / "meta.json").write_text(json.dumps({
|
|
"run_label": run_label,
|
|
"task_set_id": report.get("task_set_id"),
|
|
"backend": report.get("backend"),
|
|
"n_runs": report.get("n_runs"),
|
|
**(meta or {}),
|
|
}, indent=2), encoding="utf-8")
|
|
return str(base)
|