benchmark Group C:基线运行器 + 统一 BenchmarkRunRecord + 报告/回放(Closes #21)
在去中心化重构之上落地 benchmark 对比管线:5 个系统(single/strong/chain/sub_agent/swarm) 跑同一任务集、同一执行后端,产出统一 BenchmarkRunRecord → 评估器算 G_E/G_E,c → 报告 + 回放。 - benchmark/runners/:backend(Offline 确定性 / OpenAI 真实)+ base + 5 个 runner。各 runner 用 held-out fixture 测试在 Group B 沙箱里评分得 TestPassRate(权威,非自评)。 - benchmark/tasksets/:统一任务集 + 加载器(coding-set-1,1 个 fixture)。 - benchmark/reports/、benchmark/replay/:G_E/G_E,c/coverage/confidence + 归档。 - benchmark/baselines/comparison.py:BenchmarkRunRecord 的 CodeReview/UserAcceptance 改为 Optional(掩码归一,未采集即 None,规则 #9)。 - scripts/run-benchmark-suite.py harness + scripts/test-benchmark-runners.py。 与去中心化重构对齐:swarm runner 拓扑已**重指向去中心化流程**(种子→自选→自主分解→竞争→ 同伴交叉评审→收敛,calls=6/review=1),非旧 Master「分解→派发→单评审」。仍用同一离线后端 建模以保证公平对比(驱动活体编排器会换后端→记录不可比;活体全流程由 test-workflow-e2e 验证)。 沙箱适配:runner 评分走 fail-closed 沙箱(#24),故 test + CI 步骤设 HEICODE_SANDBOX_ISOLATED=1 (仅 CI/隔离 Pod)。 影响范围:agent_swarm(benchmark 层 + 测试 + docs + CI)。不碰 orchestrator 编排逻辑、 不改 Manager↔Swarm 契约、不影响 Client/计费/密钥/审计/发布链路。 诚实边界: - **离线后端只验证管线**:所有系统拿同一参考解 → quality 相同 → G_E=0、swarm_valid=False, 刻意不显示蜂群优势(反造假)。真实 G_E>0 需 --backend openai + 足量冻结任务集 + 多次运行。 - 故 Closes #21(运行器 + 统一记录已落地并产出合规非 NaN 记录);Refs #20(仅 1/5 场景)、 Refs #22(评估器/报告/回放已建,但 Quality 仅 TestPassRate,CodeReview/UserAcceptance 缺)、 Refs #13(验收 EPIC,需真实 run 证明 Swarm>baselines,未满足)。 Closes #21 Refs #20 Refs #22 Refs #13 Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This commit is contained in:
co-authored by
Claude Opus 4.8
parent
8b5eea296a
commit
baa67350e6
@@ -122,3 +122,15 @@ jobs:
|
||||
python scripts/test-task-competition.py
|
||||
python scripts/test-cross-review.py
|
||||
python scripts/test-convergence.py
|
||||
|
||||
# --- benchmark baseline runners + suite smoke (Group C, #21) ---
|
||||
# HEICODE_SANDBOX_ISOLATED: the runners grade generated code in the fail-closed sandbox; the
|
||||
# CI runner is ephemeral/isolated, so confirm isolation here (see security-boundary §8.1).
|
||||
- name: Benchmark runners (Group C)
|
||||
env: { HEICODE_SANDBOX_ISOLATED: "1" }
|
||||
run: python scripts/test-benchmark-runners.py
|
||||
|
||||
# Offline = pipeline validation only (deterministic, no real G_E). Guards against regressions.
|
||||
- name: Benchmark suite smoke (offline)
|
||||
env: { HEICODE_SANDBOX_ISOLATED: "1" }
|
||||
run: python scripts/run-benchmark-suite.py --taskset coding-set-1
|
||||
|
||||
Reference in New Issue
Block a user