在去中心化重构之上落地 benchmark 对比管线:5 个系统(single/strong/chain/sub_agent/swarm) 跑同一任务集、同一执行后端,产出统一 BenchmarkRunRecord → 评估器算 G_E/G_E,c → 报告 + 回放。 - benchmark/runners/:backend(Offline 确定性 / OpenAI 真实)+ base + 5 个 runner。各 runner 用 held-out fixture 测试在 Group B 沙箱里评分得 TestPassRate(权威,非自评)。 - benchmark/tasksets/:统一任务集 + 加载器(coding-set-1,1 个 fixture)。 - benchmark/reports/、benchmark/replay/:G_E/G_E,c/coverage/confidence + 归档。 - benchmark/baselines/comparison.py:BenchmarkRunRecord 的 CodeReview/UserAcceptance 改为 Optional(掩码归一,未采集即 None,规则 #9)。 - scripts/run-benchmark-suite.py harness + scripts/test-benchmark-runners.py。 与去中心化重构对齐:swarm runner 拓扑已**重指向去中心化流程**(种子→自选→自主分解→竞争→ 同伴交叉评审→收敛,calls=6/review=1),非旧 Master「分解→派发→单评审」。仍用同一离线后端 建模以保证公平对比(驱动活体编排器会换后端→记录不可比;活体全流程由 test-workflow-e2e 验证)。 沙箱适配:runner 评分走 fail-closed 沙箱(#24),故 test + CI 步骤设 HEICODE_SANDBOX_ISOLATED=1 (仅 CI/隔离 Pod)。 影响范围:agent_swarm(benchmark 层 + 测试 + docs + CI)。不碰 orchestrator 编排逻辑、 不改 Manager↔Swarm 契约、不影响 Client/计费/密钥/审计/发布链路。 诚实边界: - **离线后端只验证管线**:所有系统拿同一参考解 → quality 相同 → G_E=0、swarm_valid=False, 刻意不显示蜂群优势(反造假)。真实 G_E>0 需 --backend openai + 足量冻结任务集 + 多次运行。 - 故 Closes #21(运行器 + 统一记录已落地并产出合规非 NaN 记录);Refs #20(仅 1/5 场景)、 Refs #22(评估器/报告/回放已建,但 Quality 仅 TestPassRate,CodeReview/UserAcceptance 缺)、 Refs #13(验收 EPIC,需真实 run 证明 Swarm>baselines,未满足)。 Closes #21 Refs #20 Refs #22 Refs #13 Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
137 lines
4.7 KiB
YAML
137 lines
4.7 KiB
YAML
name: CI
|
|
|
|
on:
|
|
push:
|
|
branches: ["**"]
|
|
pull_request:
|
|
branches: ["**"]
|
|
|
|
jobs:
|
|
guardrails:
|
|
runs-on: ubuntu-latest
|
|
steps:
|
|
- uses: actions/checkout@v4
|
|
|
|
# Defense in depth: fail if secrets or heavy/generated dirs were ever committed.
|
|
- name: Block secrets & node_modules
|
|
run: |
|
|
if git ls-files | grep -E '(^|/)\.env($|\.)|(^|/)secrets/|\.pem$|\.key$|\.p12$|\.pfx$|(^|/)id_rsa$|(^|/)id_ed25519$'; then
|
|
echo "::error::Secret-like files are tracked — remove them and rotate any exposed credential."; exit 1
|
|
fi
|
|
if git ls-files | grep -E '(^|/)node_modules/'; then
|
|
echo "::error::node_modules is tracked — it must be gitignored."; exit 1
|
|
fi
|
|
|
|
- name: Required standards files present
|
|
run: |
|
|
for f in CLAUDE.md PROJECT_STANDARD.md README.md; do
|
|
test -f "$f" || { echo "::error::Missing required file: $f"; exit 1; }
|
|
done
|
|
|
|
tests:
|
|
runs-on: ubuntu-latest
|
|
steps:
|
|
- uses: actions/checkout@v4
|
|
|
|
- uses: actions/setup-python@v5
|
|
with:
|
|
python-version: "3.13"
|
|
|
|
- name: Install dependencies
|
|
run: |
|
|
python -m pip install --upgrade pip
|
|
pip install -r orchestrator/requirements.txt -r agent/requirements.txt
|
|
|
|
# Hermetic: in-memory store, planner forced offline by the tests — no model key needed.
|
|
- name: Manager contract test
|
|
env: { REDIS_FAKE: "1" }
|
|
run: python scripts/test-runtime-contract.py
|
|
|
|
- name: Workflow mechanism smoke test
|
|
env: { REDIS_FAKE: "1" }
|
|
run: python scripts/test-merge-smoke.py
|
|
|
|
- name: End-to-end workflow test
|
|
env: { REDIS_FAKE: "1" }
|
|
run: python scripts/test-workflow-e2e.py
|
|
|
|
- name: Manager event-contract test
|
|
env: { REDIS_FAKE: "1" }
|
|
run: python scripts/test-contract-events.py
|
|
|
|
- name: Benchmark metric formulas (v2.1)
|
|
env: { REDIS_FAKE: "1" }
|
|
run: python scripts/test-benchmark-metrics.py
|
|
|
|
- name: Benchmark collector
|
|
env: { REDIS_FAKE: "1" }
|
|
run: python scripts/test-benchmark-collector.py
|
|
|
|
- name: Baseline comparison
|
|
env: { REDIS_FAKE: "1" }
|
|
run: python scripts/test-baseline-comparison.py
|
|
|
|
- name: Code sandbox (in-pod test runner)
|
|
run: python scripts/test-sandbox.py
|
|
|
|
- name: Quality instrumentation (Group B)
|
|
env: { REDIS_FAKE: "1" }
|
|
run: python scripts/test-quality.py
|
|
|
|
- name: Decision-engine pheromone library (τ)
|
|
env: { REDIS_FAKE: "1" }
|
|
run: python scripts/test-decision-engine.py
|
|
|
|
- name: Dispatch scoring formulas
|
|
env: { REDIS_FAKE: "1" }
|
|
run: python scripts/test-dispatch-score.py
|
|
|
|
# --- decentralized swarm flow (the only flow; primitives are unconditional) ---
|
|
- name: Swarm seeder (#6)
|
|
run: python scripts/test-swarm-seed.py
|
|
|
|
- name: Swarm self-selection dispatch
|
|
env: { REDIS_FAKE: "1" }
|
|
run: python scripts/test-swarm-dispatch.py
|
|
|
|
- name: Swarm autonomous task generation (#7)
|
|
env: { REDIS_FAKE: "1", AGENT_PROPOSAL_BUDGET: "3" }
|
|
run: python scripts/test-swarm-autonomous.py
|
|
|
|
- name: Swarm task competition (#8)
|
|
env: { REDIS_FAKE: "1" }
|
|
run: python scripts/test-swarm-competition.py
|
|
|
|
- name: Swarm cross-review (#11)
|
|
env: { REDIS_FAKE: "1" }
|
|
run: python scripts/test-swarm-cross-review.py
|
|
|
|
- name: Swarm convergence (#12)
|
|
env: { REDIS_FAKE: "1" }
|
|
run: python scripts/test-swarm-convergence.py
|
|
|
|
- name: Swarm health guard
|
|
env: { REDIS_FAKE: "1" }
|
|
run: python scripts/test-swarm-guard.py
|
|
|
|
# Pure-module unit tests for the swarm primitives (formulas/policies, infra-free).
|
|
- name: Swarm primitive modules (unit)
|
|
env: { REDIS_FAKE: "1" }
|
|
run: |
|
|
python scripts/test-autonomous-tasks.py
|
|
python scripts/test-task-competition.py
|
|
python scripts/test-cross-review.py
|
|
python scripts/test-convergence.py
|
|
|
|
# --- benchmark baseline runners + suite smoke (Group C, #21) ---
|
|
# HEICODE_SANDBOX_ISOLATED: the runners grade generated code in the fail-closed sandbox; the
|
|
# CI runner is ephemeral/isolated, so confirm isolation here (see security-boundary §8.1).
|
|
- name: Benchmark runners (Group C)
|
|
env: { HEICODE_SANDBOX_ISOLATED: "1" }
|
|
run: python scripts/test-benchmark-runners.py
|
|
|
|
# Offline = pipeline validation only (deterministic, no real G_E). Guards against regressions.
|
|
- name: Benchmark suite smoke (offline)
|
|
env: { HEICODE_SANDBOX_ISOLATED: "1" }
|
|
run: python scripts/run-benchmark-suite.py --taskset coding-set-1
|