feat(benchmark): 落地自证采集器(阶段0+1)— S_gain≡G_E 接通 S_swarm + leaderboard
阶段0(定口径,docs/benchmark/emergence-evaluation.md §6 v2.1-impl): - S_gain ≡ G_E(差值,不强制归一 [0,100],与标准「见涌现增益」字面一致)。 - 聚合 S_gain 取对最强基线(Q_base 最大)的 G_E(最保守,避免挑弱基线虚高)。 - Q ≡ Q_quality;swarm_valid 仍要求对全部基线 G_E>0 且 G_E,c>0。 阶段1(采集器): - 新增 benchmark/collectors/selfcert_collector.py:把套件 5 份 BenchmarkRunRecord (swarm+4基线)+ 可选活体 SwarmMetrics 合流,经 baselines.compare 算 G_E/G_E,c, 补全 run_collector 无法自算的 s_gain/g_e/g_e_cost/s_swarm,可能时产出 Benchmark_Agent。 - benchmark/leaderboard:实现排行榜聚合+渲染(标准 §11 字段)。 - run-benchmark-suite.py 接入自证 + leaderboard 输出。 诚实纪律(组织规则 #9):缺真实输入一律 NaN+coverage False,不伪造。 - O(可观测性)标准无公式 → 恒 NaN;Gov 计数器未实现 → 无活体治理则 NaN。 - 故完整 Benchmark_Agent 数字仍待 O 公式 + Gov 计数器(阶段2),采集器明列缺口。 验证:新增 test-benchmark-selfcert.py(17 项)+ 现有 benchmark 测试(metrics/ collector/comparison/runners)+ offline suite + 契约冒烟(runtime/merge/freeze)全 PASS。 影响范围:仅 agent_swarm benchmark 模块 + docs;不改 Manager↔Swarm 契约/计费/审计/密钥/发布链路。 Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This commit is contained in:
co-authored by
Claude Opus 4.8
parent
b83638a475
commit
b5cc68c977
@@ -1,6 +1,62 @@
|
||||
"""Benchmark leaderboard — 排行榜聚合与展示字段。
|
||||
"""Benchmark leaderboard — 排行榜聚合与展示(标准 §11)。
|
||||
|
||||
状态:**未落地(骨架)**。展示字段(标准 §11):Benchmark_Agent / S_swarm / G_E / G_E,c /
|
||||
Reward / Cost Efficiency;场景:Coding / Refactoring / Architecture / DevOps / Bug Fix。
|
||||
需定义 leaderboard schema 与持久化。
|
||||
展示字段:`Benchmark_Agent` / `S_swarm` / `G_E` / `G_E,c` / `Reward` / `Cost Efficiency`;
|
||||
按场景(Coding / Refactoring / Architecture / DevOps / Bug Fix)分组。
|
||||
|
||||
输入为一组 `SelfCertResult`(见 collectors/selfcert_collector.py)。缺失/未覆盖的字段以 `None`
|
||||
表示并在渲染时显示「—」,**不伪造分值**(组织规则 #9)。排序按 `Benchmark_Agent`(不可用者沉底)。
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
import math
|
||||
from typing import List
|
||||
|
||||
LEADERBOARD_FIELDS = ("benchmark_agent", "s_swarm", "g_e", "g_e_cost", "reward", "cost_efficiency")
|
||||
SCENARIOS = ("coding", "refactoring", "architecture", "devops", "bugfix")
|
||||
|
||||
|
||||
def _num(x):
|
||||
return None if (x is None or (isinstance(x, float) and math.isnan(x))) else x
|
||||
|
||||
|
||||
def build_leaderboard(results: List) -> dict:
|
||||
"""Turn SelfCertResult list into leaderboard rows (one per scenario/task_set)."""
|
||||
rows = []
|
||||
for r in results:
|
||||
m = r.metrics
|
||||
# cost_efficiency = swarm CostEfficiency (= s_cost 口径, 100·Budget/ActualCost)
|
||||
rows.append({
|
||||
"scenario": r.scenario,
|
||||
"task_set_id": r.task_set_id,
|
||||
"swarm_valid": r.swarm_valid,
|
||||
"benchmark_available": r.benchmark_available,
|
||||
"benchmark_agent": _num(m.benchmark),
|
||||
"s_swarm": _num(m.s_swarm),
|
||||
"g_e": _num(m.g_e),
|
||||
"g_e_cost": _num(m.g_e_cost),
|
||||
"reward": _num(m.reward),
|
||||
"cost_efficiency": _num(m.s_cost),
|
||||
})
|
||||
# Sort by Benchmark_Agent desc; unavailable (None) sink to the bottom.
|
||||
rows.sort(key=lambda x: (x["benchmark_agent"] is not None, x["benchmark_agent"] or 0.0),
|
||||
reverse=True)
|
||||
return {"fields": list(LEADERBOARD_FIELDS), "rows": rows}
|
||||
|
||||
|
||||
def render_markdown(board: dict) -> str:
|
||||
lines = ["# Benchmark Leaderboard", "",
|
||||
"| scenario | task_set | valid | Benchmark_Agent | S_swarm | G_E | G_E,c | Reward | CostEff |",
|
||||
"|---|---|---|---|---|---|---|---|---|"]
|
||||
|
||||
def cell(v):
|
||||
return "—" if v is None else (round(v, 4) if isinstance(v, float) else v)
|
||||
|
||||
for r in board["rows"]:
|
||||
lines.append(
|
||||
f"| {r['scenario']} | `{r['task_set_id']}` | {'✅' if r['swarm_valid'] else '🔴'} "
|
||||
f"| {cell(r['benchmark_agent'])} | {cell(r['s_swarm'])} | {cell(r['g_e'])} "
|
||||
f"| {cell(r['g_e_cost'])} | {cell(r['reward'])} | {cell(r['cost_efficiency'])} |")
|
||||
lines.append("")
|
||||
lines.append("> `—` = 未覆盖/不可用(缺真实输入,未伪造)。`Benchmark_Agent` 不可用通常因 "
|
||||
"`O`(标准无公式)或 `Gov`(计数器未实现)——见各 run 的 gaps。")
|
||||
return "\n".join(lines)
|
||||
|
||||
Reference in New Issue
Block a user