taijibaga/fengqun
9
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Use external codebase for live agent scenario

Browse Source 
Replace the self-referential S07 code task with a pinned fastapi/fastapi GitHub scenario so the live Agent standard tests evaluate an external complex codebase instead of the local harness project.

Constraint: The user explicitly rejected using this project as the code target for the live scenario.

Rejected: Keeping swarm-minimal as the S07 code target | it would keep validating the harness against itself.

Confidence: high

Scope-risk: moderate

Directive: Keep S07 target files external to this repository unless the user explicitly asks for a local-harness scenario.

Tested: .venv/bin/python -u -B examples/run_standard_scenario_acceptance.py; .venv/bin/python -B -m unittest discover -s tests; .venv/bin/python -B -m py_compile swarm_minimal/*.py examples/*.py tests/*.py; .venv/bin/python -u -B examples/run_academic_standard_evaluation.py; git diff --check; docs secret-pattern scan.

Not-tested: Applying the proposed FastAPI patch inside the external fastapi/fastapi repository was not run; S07 is a live Agent reasoning and evidence-chain test.

Co-authored-by: OmX <omx@oh-my-codex.dev>
This commit is contained in:
gongzhiyong
2026-05-16 15:44:36 +08:00
co-authored by OmX
parent d632fd9f64
commit dc2b166e60
12 changed files with 594 additions and 5313 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/ACADEMIC_STANDARD_TEST_REPORT.zh-CN.md
+10 -8
View File
@@ -14,7 +14,7 @@
| S04 | 依赖边界 | MVP 不把 NATS/Cosmos 当成必需依赖 | `tests/test_standard_scenarios.py` |
| S05 | 最终收敛评分 | 最终 STEP-07 输出必须优于中间步骤 | `tests/test_standard_scenarios.py` |
| S06 | 失败注入 | 失败任务要标记失败、产生负信息素，其他任务仍可收敛 | `tests/test_standard_scenarios.py` |
| S07 | live Azure/NewAPI 连续推理 | 真实连接 Azure PostgreSQL、Redis、Blob 和 NewAPI，跑 7 步连续推理链 | `examples/run_continuous_reasoning_acceptance.py` |
| S07 | live 外部 GitHub 代码推理 | 真实连接 Azure PostgreSQL、Redis、Blob 和 NewAPI，对 `fastapi/fastapi` 固定 commit 跑 7 步代码推理链 | `examples/run_continuous_reasoning_acceptance.py` |
| S08 | 模型 I/O 报告审计 | 验证报告包含场景、输入、输出、接手证据，且无明显真实密钥样式 | `tests/test_model_io_report_audit.py` |
| B01 | 单 Agnet 故障隔离 | 一个 Agnet 崩溃不导致整体失败 | `tests/test_swarm_behavior_academic.py` |
| B02 | 群体涌现 | 多个弱局部信号通过共享状态聚合成更强整体结果 | `tests/test_swarm_behavior_academic.py` |
@@ -54,7 +54,7 @@ python3 -u -B examples/run_consensus_convergence_acceptance.py
python3 -u -B examples/run_academic_standard_evaluation.py
```
完整标准矩阵还包含 S07 live 场景和 S08 模型 I/O 报告审计：
完整标准矩阵还包含 S07 外部 GitHub live 代码场景和 S08 模型 I/O 报告审计：
```bash
python3 -u -B examples/run_standard_scenario_acceptance.py
@@ -67,20 +67,22 @@ S07 需要本地被 git 忽略的 `.env`，其中包含 Azure PostgreSQL、Redis
| 命令 | 本轮结果 | 说明 |
| --- | --- | --- |
| `python3 -B -m py_compile swarm_minimal/*.py examples/*.py tests/*.py` | PASS | 无输出，表示所有模块编译通过 |
| `python3 -B -m unittest discover -s tests` | PASS | 28 个测试通过 |
| `python3 -B -m unittest discover -s tests` | PASS | 29 个测试通过 |
| `python3 -B -m unittest tests.test_markov_process_properties` | PASS | 3 个马尔可夫适配性测试通过 |
| `python3 -B -m unittest tests.test_model_io_report_audit` | PASS | 2 个模型 I/O 报告审计测试通过 |
| `python3 -u -B examples/run_swarm_behavior_acceptance.py` | PASS | B01-B04 全部通过 |
| `python3 -u -B examples/run_swarm_vs_traditional_benchmark.py` | PASS | 蜂群总归一化分 0.9175，传统基线 0.1958，ratio 4.69 |
| `python3 -u -B examples/run_consensus_convergence_acceptance.py` | PASS | 2 轮收敛，接受 `lease_based_pg_queue` |
| `python3 -u -B examples/run_academic_standard_evaluation.py` | PASS | A01-A05 本地学术化门禁全部通过，检测到 `examples/.env` 可用于 live 测试 |
| `python3 -u -B examples/run_standard_scenario_acceptance.py` | PASS | S01-S08 全部通过；S07 完成 7 个 live 任务，12 个检查无失败，S08 报告审计通过 |
| `python3 -u -B examples/run_standard_scenario_acceptance.py` | PASS | S01-S08 全部通过；S07 外部 FastAPI 代码场景完成 7 个 live 任务，12 个检查无失败，S08 报告审计通过 |
最新 S07 live 证据：
最新 S07 外部 GitHub live 证据：
```json
{
"run_id": "04c641d170fe4ea7aa3d882d9df37cca",
"run_id": "7ba01d0cc4ad402793938835654fbca5",
"target_repo": "fastapi/fastapi",
"target_commit": "ecace740f3eaccb1aba152cf1de79477095c56f4",
"completed_tasks": 7,
"accepted_score": 1.0,
"selected_models": [
@@ -90,7 +92,7 @@ S07 需要本地被 git 忽略的 `.env`，其中包含 Azure PostgreSQL、Redis
],
"failed_checks": [],
"check_count": 12,
"artifact_path": "swarm-runs/04c641d170fe4ea7aa3d882d9df37cca/result.json"
"artifact_path": "swarm-runs/7ba01d0cc4ad402793938835654fbca5/result.json"
}
```
@@ -100,7 +102,7 @@ S07 需要本地被 git 忽略的 `.env`，其中包含 Azure PostgreSQL、Redis
| --- | --- |
| 本地学术化门禁 A01-A05 | 可通过 |
| 标准矩阵 S01-S06 | 可通过 |
| 标准矩阵 S07 live Azure/NewAPI | 已通过 |
| 标准矩阵 S07 live 外部 GitHub 代码推理 | 已通过 |
| 标准矩阵 S08 模型 I/O 报告审计 | 已通过 |
| 生产级标准化验收 | 当前最小标准已满足 |
docs/AGENT_SWARM_QUALITY_STANDARD.zh-CN.md
+18 -12
View File
@@ -43,7 +43,7 @@
| AQS-08 可观测性 | run_id、task status、score、observation、artifact、stream event 可查 | S07 PostgreSQL/Redis/Blob 证据 |
| AQS-09 错误可解释 | 失败任务必须记录 error、failed observation 和负信息素 | S06/B01/C01 |
| AQS-10 交接准备度 | 输出要给下一个 Agent 留出摘要、风险和下一步 | S07 prompt/output 中强制“下一步/交接” |
| AQS-11 外部依赖真实性 | live 测试必须真实连 PostgreSQL、Redis、Blob、NewAPI | S07 live integration PASS |
| AQS-11 外部依赖真实性 | live 测试必须真实连 PostgreSQL、Redis、Blob、NewAPI，并把代码任务指向外部 GitHub 项目 | S07 external GitHub live PASS |
| AQS-12 人类审计友好 | 最终报告必须能回答：任务、输入、输出、接手、结果、未满足项 | S08 + `MODEL_AGNET_IO_REPORT.zh-CN.md` 和本文件 |
## 4. 蜂群 Agent 质量标准 SW-AQS
@@ -71,20 +71,24 @@
总任务：
```text
连续性长推理场景：为 swarm-minimal 设计可恢复的大规模代码任务推理链
外部 GitHub 代码场景：审查 fastapi/fastapi 响应序列化与 OpenAPI 依赖链
```
目标不是让模型自由聊天，而是让 7 个连续 Agent 阶段接力完成一个工程推理链：
目标不是让模型自由聊天，也不是拿本仓库验证自己闭环，而是让 7 个连续 Agent 阶段接力审查一个外部复杂 GitHub 项目：
- 目标仓库：`https://github.com/fastapi/fastapi`
- 固定 commit：`ecace740f3eaccb1aba152cf1de79477095c56f4`
- 代码范围：`fastapi/routing.py`、`fastapi/dependencies/utils.py`、`fastapi/openapi/utils.py`、`fastapi/params.py`、`fastapi/encoders.py`、`fastapi/applications.py`、`tests/test_serialize_response_model.py`、`tests/test_response_model_data_filter.py`
| 步骤 | 分配任务 | 主要验证点 |
| --- | --- | --- |
| STEP-01 | 界定问题和不可变约束 | 建立目标、输入输出、不变量和 Azure 资源边界 |
| STEP-02 | 建立依赖图和状态模型 | 承接 STEP-01，给出任务依赖图、共享状态、租约、状态转移 |
| STEP-03 | 设计连续调度算法 | 承接 STEP-02，设计调度、信息素更新、收敛算法和复杂度 |
| STEP-04 | 构造反例和失败场景 | 承接 STEP-03，构造慢模型、重复任务、状态倒退、分数误导 |
| STEP-05 | 修正算法和恢复策略 | 承接 STEP-04，加入幂等、重试、死信、outbox、重连恢复 |
| STEP-06 | 落到文件级实现计划 | 承接 STEP-05，引用真实文件和测试文件 |
| STEP-07 | 最终收敛和验收判定 | 承接 STEP-06，给出命令、指标、失败判定和上线前结论 |
| STEP-01 | 界定问题和不可变约束 | 建立 FastAPI 外部代码审查目标、输入输出、不变量和禁止自测边界 |
| STEP-02 | 建立依赖图和状态模型 | 承接 STEP-01，给出路由、依赖注入、OpenAPI、响应序列化和测试文件依赖图 |
| STEP-03 | 定位跨文件风险路径 | 承接 STEP-02，定位 response_model、Depends、参数 metadata、jsonable_encoder 与 OpenAPI schema 的漂移风险 |
| STEP-04 | 构造反例和失败场景 | 承接 STEP-03，构造响应过滤、默认值、nullable、依赖参数和 schema 不一致反例 |
| STEP-05 | 修正算法和恢复策略 | 承接 STEP-04，给出模块修正策略、兼容性和 Starlette/Pydantic 交互边界 |
| STEP-06 | 落到文件级实现计划 | 承接 STEP-05，引用 fastapi/fastapi 真实源码文件和测试文件 |
| STEP-07 | 最终收敛和验收判定 | 承接 STEP-06，给出 FastAPI 仓库内可执行验收命令、指标和可合并结论 |
模型分配来自动态发现，不写死 `NEWAPI_MODEL`：
@@ -122,11 +126,13 @@
## 7. 本轮判定
最新标准矩阵 run（S01-S08 全部通过，以下为 S07 live 证据）：
最新标准矩阵 run（S01-S08 全部通过，以下为 S07 外部 GitHub live 证据）：
```json
{
"run_id": "04c641d170fe4ea7aa3d882d9df37cca",
"run_id": "7ba01d0cc4ad402793938835654fbca5",
"target_repo": "fastapi/fastapi",
"target_commit": "ecace740f3eaccb1aba152cf1de79477095c56f4",
"completed_tasks": 7,
"accepted_score": 1.0,
"check_count": 12,
docs/INDUSTRY_STANDARD_AGNET_TEST_PLAN.md
+4 -2
View File
@@ -28,11 +28,13 @@
3. 模型输入输出能被审计：知道每个 Agent 被分配了什么任务、收到了什么输入、输出了什么、下一个 Agent 如何接手。
4. live 测试真实连接 Azure PostgreSQL、Redis、Blob 和 NewAPI，且不输出任何真实密钥。
最新一次标准矩阵（S01-S08）已经通过，以下为 S07 live 证据：
最新一次标准矩阵（S01-S08）已经通过，以下为 S07 外部 GitHub live 证据：
```json
{
"run_id": "04c641d170fe4ea7aa3d882d9df37cca",
"run_id": "7ba01d0cc4ad402793938835654fbca5",
"target_repo": "fastapi/fastapi",
"target_commit": "ecace740f3eaccb1aba152cf1de79477095c56f4",
"completed_tasks": 7,
"accepted_score": 1.0,
"check_count": 12,
docs/MODEL_AGNET_IO_REPORT.zh-CN.md
+422 -5235
View File
File diff suppressed because it is too large Load Diff
docs/STANDARD_TEST_MATRIX.md
+2 -2
View File
@@ -12,7 +12,7 @@ result means more than "one long run completed".
| Unit | Existing closed-loop and NewAPI mock behavior | `./.venv/bin/python -B -m unittest discover -s tests` |
| Deterministic scenarios | No-network Given/When/Then cases for continuity, policy, final scoring, and failure injection | `./.venv/bin/python -B -m unittest tests.test_standard_scenarios` |
| Academic Markov fit | No-network checks for Markov-style transition behavior and non-MDP limitations | `./.venv/bin/python -B -m unittest tests.test_markov_process_properties` |
| Live integration | Real NewAPI + Azure PostgreSQL/Redis/Blob continuous reasoning chain | `./.venv/bin/python -u -B examples/run_continuous_reasoning_acceptance.py` |
| Live integration | Real NewAPI + Azure PostgreSQL/Redis/Blob continuous reasoning chain against external GitHub code | `./.venv/bin/python -u -B examples/run_continuous_reasoning_acceptance.py` |
| Report audit | Human-auditable model I/O report with scenario, input, output, handoff and secret-safety evidence | `./.venv/bin/python -B -m unittest tests.test_model_io_report_audit` |
## Required Scenarios
@@ -25,7 +25,7 @@ result means more than "one long run completed".
| S04 | Dependency boundary | Text mentions NATS/Cosmos | Use negative and positive dependency wording | Negative wording passes; required dependency wording fails |
| S05 | Final convergence scoring | STEP-06 and STEP-07 both produce rich outputs | Score both outputs | STEP-07 is strictly selected as final convergence |
| S06 | Failure injection | One deterministic worker raises | Coordinator runs to convergence | Failed task is marked failed, pheromone is negative, completed task still converges |
| S07 | Live resource closure | Real Azure/NewAPI env is loaded | Run seven-step continuous reasoning | PostgreSQL, Redis, Blob, stream events, model discovery, and final artifact all pass |
| S07 | External GitHub code reasoning | Real Azure/NewAPI env is loaded and target is `fastapi/fastapi` at a pinned commit | Run seven-step code reasoning against external GitHub files | PostgreSQL, Redis, Blob, stream events, model discovery, external file references, and final artifact all pass |
| S08 | Model I/O report audit | Generated model/Agnet I/O report | Audit scenario coverage, task input/output, handoff evidence, and obvious secret patterns | Report is human-auditable and contains no obvious secret values |
| M01 | Markov-style claim transition | Same current tasks and pheromone values but different prior paths | Claim next task | The same high-pheromone task is claimed |
| M02 | Markov-style score transition | Same current task, agent, output, and score but different prior paths | Complete task | Task status, output, score, and pheromone update match |