feat(benchmark): 落地自证采集器(阶段0+1)— S_gain≡G_E 接通 S_swarm + leaderboard

阶段0(定口径,docs/benchmark/emergence-evaluation.md §6 v2.1-impl):
- S_gain ≡ G_E(差值,不强制归一 [0,100],与标准「见涌现增益」字面一致)。
- 聚合 S_gain 取对最强基线(Q_base 最大)的 G_E(最保守,避免挑弱基线虚高)。
- Q ≡ Q_quality;swarm_valid 仍要求对全部基线 G_E>0 且 G_E,c>0。

阶段1(采集器):
- 新增 benchmark/collectors/selfcert_collector.py:把套件 5 份 BenchmarkRunRecord
  (swarm+4基线)+ 可选活体 SwarmMetrics 合流,经 baselines.compare 算 G_E/G_E,c,
  补全 run_collector 无法自算的 s_gain/g_e/g_e_cost/s_swarm,可能时产出 Benchmark_Agent。
- benchmark/leaderboard:实现排行榜聚合+渲染(标准 §11 字段)。
- run-benchmark-suite.py 接入自证 + leaderboard 输出。

诚实纪律(组织规则 #9):缺真实输入一律 NaN+coverage False,不伪造。
- O(可观测性)标准无公式 → 恒 NaN;Gov 计数器未实现 → 无活体治理则 NaN。
- 故完整 Benchmark_Agent 数字仍待 O 公式 + Gov 计数器(阶段2),采集器明列缺口。

验证:新增 test-benchmark-selfcert.py(17 项)+ 现有 benchmark 测试(metrics/
collector/comparison/runners)+ offline suite + 契约冒烟(runtime/merge/freeze)全 PASS。

影响范围:仅 agent_swarm benchmark 模块 + docs;不改 Manager↔Swarm 契约/计费/审计/密钥/发布链路。

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This commit is contained in:
Fasthei
2026-06-12 17:38:19 +08:00
co-authored by Claude Opus 4.8
parent b83638a475
commit b5cc68c977
5 changed files with 392 additions and 6 deletions
+60 -4
View File
@@ -1,6 +1,62 @@
"""Benchmark leaderboard — 排行榜聚合与展示字段。
"""Benchmark leaderboard — 排行榜聚合与展示(标准 §11)。
状态:**未落地(骨架)**。展示字段(标准 §11):Benchmark_Agent / S_swarm / G_E / G_E,c /
Reward / Cost Efficiency;场景:Coding / Refactoring / Architecture / DevOps / Bug Fix。
需定义 leaderboard schema 与持久化。
展示字段:`Benchmark_Agent` / `S_swarm` / `G_E` / `G_E,c` / `Reward` / `Cost Efficiency`;
按场景(Coding / Refactoring / Architecture / DevOps / Bug Fix)分组。
输入为一组 `SelfCertResult`(见 collectors/selfcert_collector.py)。缺失/未覆盖的字段以 `None`
表示并在渲染时显示「—」,**不伪造分值**(组织规则 #9)。排序按 `Benchmark_Agent`(不可用者沉底)。
"""
from __future__ import annotations
import math
from typing import List
LEADERBOARD_FIELDS = ("benchmark_agent", "s_swarm", "g_e", "g_e_cost", "reward", "cost_efficiency")
SCENARIOS = ("coding", "refactoring", "architecture", "devops", "bugfix")
def _num(x):
return None if (x is None or (isinstance(x, float) and math.isnan(x))) else x
def build_leaderboard(results: List) -> dict:
"""Turn SelfCertResult list into leaderboard rows (one per scenario/task_set)."""
rows = []
for r in results:
m = r.metrics
# cost_efficiency = swarm CostEfficiency (= s_cost 口径, 100·Budget/ActualCost)
rows.append({
"scenario": r.scenario,
"task_set_id": r.task_set_id,
"swarm_valid": r.swarm_valid,
"benchmark_available": r.benchmark_available,
"benchmark_agent": _num(m.benchmark),
"s_swarm": _num(m.s_swarm),
"g_e": _num(m.g_e),
"g_e_cost": _num(m.g_e_cost),
"reward": _num(m.reward),
"cost_efficiency": _num(m.s_cost),
})
# Sort by Benchmark_Agent desc; unavailable (None) sink to the bottom.
rows.sort(key=lambda x: (x["benchmark_agent"] is not None, x["benchmark_agent"] or 0.0),
reverse=True)
return {"fields": list(LEADERBOARD_FIELDS), "rows": rows}
def render_markdown(board: dict) -> str:
lines = ["# Benchmark Leaderboard", "",
"| scenario | task_set | valid | Benchmark_Agent | S_swarm | G_E | G_E,c | Reward | CostEff |",
"|---|---|---|---|---|---|---|---|---|"]
def cell(v):
return "—" if v is None else (round(v, 4) if isinstance(v, float) else v)
for r in board["rows"]:
lines.append(
f"| {r['scenario']} | `{r['task_set_id']}` | {'✅' if r['swarm_valid'] else '🔴'} "
f"| {cell(r['benchmark_agent'])} | {cell(r['s_swarm'])} | {cell(r['g_e'])} "
f"| {cell(r['g_e_cost'])} | {cell(r['reward'])} | {cell(r['cost_efficiency'])} |")
lines.append("")
lines.append("> `—` = 未覆盖/不可用(缺真实输入,未伪造)。`Benchmark_Agent` 不可用通常因 "
"`O`(标准无公式)或 `Gov`(计数器未实现)——见各 run 的 gaps。")
return "\n".join(lines)