Establish agent swarm quality evidence
Define Agent and swarm-specific acceptance evidence, move the reports under docs, and make the homepage point to the current standard, live run, model I/O, and handoff evidence. Constraint: Agent quality standards are configured from industry AI and agent risk references because there is no single accepted swarm-Agent certification standard. Rejected: Treating py_compile or unittest as the primary quality standard | they are evidence collection tools, not the Agent quality standard itself. Confidence: high Scope-risk: moderate Directive: Keep future standard reports under docs/ and keep secrets in ignored local .env files only. Tested: git diff --cached --check; python -B -m py_compile swarm_minimal/*.py examples/*.py tests/*.py; python -B -m unittest discover -s tests; python -u -B examples/run_academic_standard_evaluation.py Not-tested: Did not rerun the full live Azure/NewAPI S07 scenario after moving docs; previous live run 3e8e58ae4e084bc8b90cf5c46f8992f3 passed before the docs relocation. Co-authored-by: OmX <omx@oh-my-codex.dev>
This commit is contained in:
@@ -1,45 +0,0 @@
|
||||
# Industry Standard Agnet Test Plan
|
||||
|
||||
There is no single universal "Agnet" certification standard. This plan maps the
|
||||
minimal swarm to widely used global standards and frameworks that are relevant
|
||||
to autonomous AI agents, software quality, testing, security, risk management,
|
||||
and observability.
|
||||
|
||||
## Source Standards
|
||||
|
||||
| Standard / framework | What it contributes here | Local evidence |
|
||||
| --- | --- | --- |
|
||||
| ISO/IEC/IEEE 29119 Software Testing | Test process, test design, traceable scenarios, PASS/FAIL evidence | `STANDARD_TEST_MATRIX.md`, `examples/run_standard_scenario_acceptance.py` |
|
||||
| ISO/IEC 25010 Product Quality Model | Functional suitability, reliability, security, maintainability, portability, performance efficiency, compatibility, usability | `tests/test_industry_standard_mapping.py` |
|
||||
| NIST AI RMF 1.0 | Govern, Map, Measure, Manage risk lifecycle | Standard control mapping in this file |
|
||||
| NIST AI 600-1 GenAI Profile | GenAI-specific risk measurement and management expectations | Secret redaction, model discovery, output policy, live checks |
|
||||
| OWASP LLM Top 10 / Agentic Skills guidance | Prompt injection, excessive agency, sensitive information disclosure, supply chain, tool boundary controls | Deterministic security/policy tests |
|
||||
| MITRE ATLAS | Adversarial AI threat framing | Failure and policy scenarios linked to threat classes |
|
||||
| OpenTelemetry | Vendor-neutral observability expectations for traces, metrics, and logs | Event, score, artifact, and run-id evidence |
|
||||
|
||||
## Control Matrix
|
||||
|
||||
| Control ID | Standard mapping | Requirement | Test evidence |
|
||||
| --- | --- | --- | --- |
|
||||
| AGT-29119-TRACE | ISO/IEC/IEEE 29119 | Every scenario has ID, Given, When, Then, command, and evidence | `run_standard_scenario_acceptance.py` output |
|
||||
| AGT-25010-FUNC | ISO/IEC 25010 functional suitability | Goal submission, task execution, scoring, and convergence work end to end | Unit and live tests |
|
||||
| AGT-25010-REL | ISO/IEC 25010 reliability | Failed agent does not prevent successful convergence of completed tasks | `test_failed_agent_records_failed_task_and_negative_pheromone` |
|
||||
| AGT-25010-SEC | ISO/IEC 25010 security | Secrets are redacted; NATS/Cosmos are not required MVP dependencies | config and policy tests |
|
||||
| AGT-25010-MAINT | ISO/IEC 25010 maintainability | Test matrix and source-file traceability are explicit | docs and deterministic tests |
|
||||
| AGT-NIST-GOVERN | NIST AI RMF Govern | Scope, dependency boundaries, and acceptance rules are documented | this file + matrix |
|
||||
| AGT-NIST-MAP | NIST AI RMF Map | Intended use, resources, actors, and failure modes are enumerated | scenario matrix |
|
||||
| AGT-NIST-MEASURE | NIST AI RMF Measure | Scores, run status, stream event deltas, artifact existence, and failures are measured | live report |
|
||||
| AGT-NIST-MANAGE | NIST AI RMF Manage | Failure handling and dependency rejection have explicit stop rules | deterministic tests |
|
||||
| AGT-OWASP-PI | OWASP LLM / Agentic | Prompt content cannot redefine MVP dependency boundary or request secrets | policy tests |
|
||||
| AGT-OWASP-EXCESS | OWASP Agentic | Agent outputs cannot create external dependency requirements outside approved Azure resources | policy tests |
|
||||
| AGT-MITRE-FAIL | MITRE ATLAS style threat modeling | Timeout/failure and misleading score scenarios are represented | failure injection and final scoring tests |
|
||||
| AGT-OTEL-OBS | OpenTelemetry-style observability | Live run returns run_id, selected models, event delta, score evidence, artifact path | live integration report |
|
||||
|
||||
## Minimum Passing Bar
|
||||
|
||||
The industry-standard acceptance is PASS only when:
|
||||
|
||||
1. The local standard scenario matrix passes.
|
||||
2. Every required control above has at least one local evidence source.
|
||||
3. No control has an unhandled critical gap.
|
||||
4. The final report preserves run evidence without printing secrets.
|
||||
@@ -2,6 +2,30 @@
|
||||
|
||||
这是一个独立的最小蜂群闭环原型,不接入 Heicode Manager 主项目,也不修改现有 API。
|
||||
|
||||
## 当前状态
|
||||
|
||||
本仓库已经补齐最小 Agent / 蜂群 Agent 标准化测试闭环。这里的标准不是普通软件开发质量标准,而是围绕 Agent 的任务理解、上下文承接、模型选择、工具边界、敏感信息保护、可观测性、handoff、信息素协作、收敛和马尔可夫式状态转移建立的 AQS / SW-AQS v1。
|
||||
|
||||
最新 live 标准矩阵已通过:
|
||||
|
||||
```json
|
||||
{
|
||||
"run_id": "3e8e58ae4e084bc8b90cf5c46f8992f3",
|
||||
"completed_tasks": 7,
|
||||
"accepted_score": 1.0,
|
||||
"check_count": 12,
|
||||
"failed_checks": []
|
||||
}
|
||||
```
|
||||
|
||||
主要报告入口都放在 `docs/`,完整索引见 `docs/README.md`:
|
||||
|
||||
- `docs/AGENT_SWARM_QUALITY_STANDARD.zh-CN.md`:Agent / 蜂群 Agent 质量标准、测试标准来源、S07 任务分配和接手机制。
|
||||
- `docs/ACADEMIC_STANDARD_TEST_REPORT.zh-CN.md`:测试场景、测试过程、测试结果、算法清单和马尔可夫过程判定。
|
||||
- `docs/MODEL_AGNET_IO_REPORT.zh-CN.md`:每个模型 / Agnet 的任务、输入、输出、评分和交接过程。
|
||||
- `docs/STANDARD_TEST_MATRIX.md`:S01-S07 标准矩阵。
|
||||
- `docs/INDUSTRY_STANDARD_AGNET_TEST_PLAN.md`:行业参考框架到本项目标准的映射。
|
||||
|
||||
目标是先把蜂群的四个共享资源跑通:
|
||||
|
||||
1. 任务池:记录待执行、执行中、完成和失败的任务。
|
||||
@@ -44,11 +68,24 @@ MVP 不使用 NATS,也不使用 Cosmos DB。
|
||||
## 本地运行
|
||||
|
||||
```bash
|
||||
cd swarm-minimal
|
||||
python3 -B examples/run_demo.py
|
||||
python3 -B -m unittest discover -s tests
|
||||
```
|
||||
|
||||
标准化测试入口:
|
||||
|
||||
```bash
|
||||
python3 -B -m py_compile swarm_minimal/*.py examples/*.py tests/*.py
|
||||
python3 -B -m unittest discover -s tests
|
||||
python3 -u -B examples/run_academic_standard_evaluation.py
|
||||
```
|
||||
|
||||
完整 live 标准矩阵需要本机被忽略的 `.env` 中提供 Azure PostgreSQL、Redis、Blob 和 NewAPI 测试凭据:
|
||||
|
||||
```bash
|
||||
python3 -u -B examples/run_standard_scenario_acceptance.py
|
||||
```
|
||||
|
||||
## Agnet / NewAPI 测试
|
||||
|
||||
NewAPI 连接使用 OpenAI-compatible `/v1/chat/completions` 形式。真实 key 只通过环境变量注入:
|
||||
@@ -62,7 +99,7 @@ export NEWAPI_API_KEY=<newapi-api-key>
|
||||
python3 -B examples/run_newapi_agnet_demo.py
|
||||
```
|
||||
|
||||
也可以在本机创建 `swarm-minimal/.env`,demo 会自动读取;该文件已被 `.gitignore` 忽略:
|
||||
也可以在本机创建项目根目录 `.env`,demo 会自动读取;该文件已被 `.gitignore` 忽略:
|
||||
|
||||
```bash
|
||||
NEWAPI_BASE_URL=<newapi-base-url>
|
||||
@@ -113,7 +150,7 @@ export SWARM_BLOB_CONTAINER=swarm-artifacts
|
||||
python3 -B examples/run_azure_demo.py
|
||||
```
|
||||
|
||||
`examples/run_azure_demo.py`、`examples/run_newapi_agnet_demo.py`、`examples/run_three_newapi_agnets.py` 和 `examples/run_full_live_test.py` 都会自动读取本地私有 `swarm-minimal/.env`。
|
||||
`examples/run_azure_demo.py`、`examples/run_newapi_agnet_demo.py`、`examples/run_three_newapi_agnets.py` 和 `examples/run_full_live_test.py` 都会自动读取本地私有 `.env`;优先读取项目根目录 `.env`,没有时读取 `examples/.env`。
|
||||
|
||||
如果连接字符串、数据库密码、Redis key 或 Storage account key 曾经出现在聊天、日志、截图或提交记录里,应在 Azure 侧轮换后再长期使用。
|
||||
|
||||
@@ -121,7 +158,7 @@ python3 -B examples/run_azure_demo.py
|
||||
|
||||
- `swarm_minimal/core.py`:最小蜂群闭环逻辑。
|
||||
- `swarm_minimal/config.py`:从环境变量读取 PostgreSQL、Redis、Blob 配置,并只输出脱敏摘要。
|
||||
- `swarm_minimal/local_env.py`:读取本地私有 `.env`,不依赖第三方 dotenv 包。
|
||||
- `swarm_minimal/local_env.py`:读取本地私有 `.env` 或 `examples/.env`,不依赖第三方 dotenv 包。
|
||||
- `swarm_minimal/azure_store.py`:可选 Azure 后端,使用 PostgreSQL、Redis Streams / Sorted Set 和 Blob Storage。
|
||||
- `swarm_minimal/azure_resources.py`:Azure 资源需求清单。
|
||||
- `swarm_minimal/newapi_agnet.py`:NewAPI / OpenAI-compatible Agnet 适配器。
|
||||
@@ -133,3 +170,7 @@ python3 -B examples/run_azure_demo.py
|
||||
- `examples/run_full_live_test_interactive.py`:本机交互式输入密钥并运行完整 live 测试。
|
||||
- `tests/test_minimal_swarm.py`:闭环和资源约束测试。
|
||||
- `tests/test_newapi_agnet.py`:NewAPI Agnet 的 mock 测试。
|
||||
- `tests/test_markov_process_properties.py`:马尔可夫式状态转移判定测试。
|
||||
- `examples/run_academic_standard_evaluation.py`:学术化标准测试汇总入口。
|
||||
- `docs/ACADEMIC_STANDARD_TEST_REPORT.zh-CN.md`:标准来源、测试过程、结果、算法和马尔可夫过程结论。
|
||||
- `docs/AGENT_SWARM_QUALITY_STANDARD.zh-CN.md`:Agent / 蜂群 Agent 质量标准、S07 任务分配、模型接手机制和当前结论。
|
||||
|
||||
@@ -0,0 +1,130 @@
|
||||
# 学术化标准测试报告
|
||||
|
||||
**对象**: `swarm-minimal` 最小蜂群原型
|
||||
**日期**: 2026-05-16
|
||||
**范围**: 本地确定性测试、蜂群行为验收、传统 Agnet 对比、多轮共识、马尔可夫过程适配性判断
|
||||
|
||||
## 1. 已实现的场景测试
|
||||
|
||||
| ID | 场景 | 测试目的 | 证据入口 |
|
||||
| --- | --- | --- | --- |
|
||||
| S01 | 静态编译 | 所有 Python 模块可编译,无语法错误 | `python3 -B -m py_compile swarm_minimal/*.py examples/*.py tests/*.py` |
|
||||
| S02 | 单元回归 | 内存蜂群闭环、Azure 资源边界、配置脱敏、NewAPI mock 行为 | `python3 -B -m unittest discover -s tests` |
|
||||
| S03 | 连续推理链路 | 每一步必须承接上一步 marker,断链必须失败 | `tests/test_standard_scenarios.py` |
|
||||
| S04 | 依赖边界 | MVP 不把 NATS/Cosmos 当成必需依赖 | `tests/test_standard_scenarios.py` |
|
||||
| S05 | 最终收敛评分 | 最终 STEP-07 输出必须优于中间步骤 | `tests/test_standard_scenarios.py` |
|
||||
| S06 | 失败注入 | 失败任务要标记失败、产生负信息素,其他任务仍可收敛 | `tests/test_standard_scenarios.py` |
|
||||
| B01 | 单 Agnet 故障隔离 | 一个 Agnet 崩溃不导致整体失败 | `tests/test_swarm_behavior_academic.py` |
|
||||
| B02 | 群体涌现 | 多个弱局部信号通过共享状态聚合成更强整体结果 | `tests/test_swarm_behavior_academic.py` |
|
||||
| B03 | 信息素间接协作 | 信息素影响任务 claim 顺序并形成正反馈 | `tests/test_swarm_behavior_academic.py` |
|
||||
| B04 | handoff 连续性 | active agent、目标 agent、payload 在交接中保留 | `tests/test_swarm_behavior_academic.py` |
|
||||
| C01-C04 | 传统 Agnet 对比 | 和单路线/FIFO/无共享状态/无上下文 handoff 的基线比较 | `examples/run_swarm_vs_traditional_benchmark.py` |
|
||||
| M01-M03 | 马尔可夫过程适配性 | 判断当前实现是马尔可夫式状态机,但不是严格 MDP | `tests/test_markov_process_properties.py` |
|
||||
|
||||
## 2. 测试标准来源
|
||||
|
||||
本项目没有使用“某一个 Agnet 认证标准”,因为行业目前还没有统一答案。当前采用 `AGENT_SWARM_QUALITY_STANDARD.zh-CN.md` 中定义的 AQS / SW-AQS v1:以行业 Agent 风险框架为参考,配置本项目自己的 Agent 和蜂群 Agent 质量标准。
|
||||
|
||||
| 来源 | 本项目采用方式 |
|
||||
| --- | --- |
|
||||
| NIST AI RMF 1.0 | Govern / Map / Measure / Manage 的 Agent 风险管理结构 |
|
||||
| NIST AI 600-1 | 生成式 AI 的幻觉、隐私、信息安全、组件集成、预部署测试和事件披露 |
|
||||
| OWASP LLM Top 10 | prompt injection、敏感信息泄露、过度代理、不安全输出处理、过度信任 |
|
||||
| OWASP Agentic Skills Top 10 | Agent 技能/工具执行层的权限、隔离、审计、运行时安全 |
|
||||
| MITRE ATLAS | AI 攻击/误用视角下的失败、越权、诱导和异常场景 |
|
||||
| OpenTelemetry | Agent 过程的 traces、metrics、logs、events 和 artifact 证据 |
|
||||
| LangGraph handoff 参考 | `transfer_to_<agent>`、active-agent 路由和上下文传递语义 |
|
||||
| 蜂群模式归纳标准 | 去中心化、自组织、局部感知、间接协作、涌现、鲁棒性和收敛 |
|
||||
| Markov property 定义 | 给定当前状态后,未来状态不再依赖历史路径 |
|
||||
|
||||
`py_compile` 和 `unittest` 只作为证据采集工具,不作为最终质量标准本身。完整链接在 `swarm_minimal/academic_evaluation.py` 的 `ACADEMIC_STANDARD_SOURCES` 中维护。
|
||||
|
||||
## 3. 测试过程
|
||||
|
||||
本地确定性验收按以下顺序执行:
|
||||
|
||||
```bash
|
||||
python3 -B -m py_compile swarm_minimal/*.py examples/*.py tests/*.py
|
||||
python3 -B -m unittest discover -s tests
|
||||
python3 -u -B examples/run_swarm_behavior_acceptance.py
|
||||
python3 -u -B examples/run_swarm_vs_traditional_benchmark.py
|
||||
python3 -u -B examples/run_consensus_convergence_acceptance.py
|
||||
python3 -u -B examples/run_academic_standard_evaluation.py
|
||||
```
|
||||
|
||||
完整标准矩阵还包含 S07 live 场景:
|
||||
|
||||
```bash
|
||||
python3 -u -B examples/run_standard_scenario_acceptance.py
|
||||
```
|
||||
|
||||
S07 需要本地被 git 忽略的 `.env`,其中包含 Azure PostgreSQL、Redis、Blob 和 NewAPI 测试凭据。脚本优先读取项目根目录 `.env`,没有时读取 `examples/.env`。没有可用 `.env` 时,本地确定性门禁可以通过,但完整标准化验收不能判定为通过。
|
||||
|
||||
## 4. 本轮实际执行结果
|
||||
|
||||
| 命令 | 本轮结果 | 说明 |
|
||||
| --- | --- | --- |
|
||||
| `python3 -B -m py_compile swarm_minimal/*.py examples/*.py tests/*.py` | PASS | 无输出,表示所有模块编译通过 |
|
||||
| `python3 -B -m unittest discover -s tests` | PASS | 26 个测试通过 |
|
||||
| `python3 -B -m unittest tests.test_markov_process_properties` | PASS | 3 个马尔可夫适配性测试通过 |
|
||||
| `python3 -u -B examples/run_swarm_behavior_acceptance.py` | PASS | B01-B04 全部通过 |
|
||||
| `python3 -u -B examples/run_swarm_vs_traditional_benchmark.py` | PASS | 蜂群总归一化分 0.9175,传统基线 0.1958,ratio 4.69 |
|
||||
| `python3 -u -B examples/run_consensus_convergence_acceptance.py` | PASS | 2 轮收敛,接受 `lease_based_pg_queue` |
|
||||
| `python3 -u -B examples/run_academic_standard_evaluation.py` | PASS | A01-A05 本地学术化门禁全部通过,检测到 `examples/.env` 可用于 live 测试 |
|
||||
| `python3 -u -B examples/run_standard_scenario_acceptance.py` | PASS | S01-S07 全部通过;S07 完成 7 个 live 任务,12 个检查无失败 |
|
||||
|
||||
最新 S07 live 证据:
|
||||
|
||||
```json
|
||||
{
|
||||
"run_id": "3e8e58ae4e084bc8b90cf5c46f8992f3",
|
||||
"completed_tasks": 7,
|
||||
"accepted_score": 1.0,
|
||||
"selected_models": [
|
||||
"deepseek-v4-flash",
|
||||
"claude-haiku-4-5-20251001",
|
||||
"claude-sonnet-4-6"
|
||||
],
|
||||
"failed_checks": [],
|
||||
"check_count": 12,
|
||||
"artifact_path": "swarm-runs/3e8e58ae4e084bc8b90cf5c46f8992f3/result.json"
|
||||
}
|
||||
```
|
||||
|
||||
## 5. 达标判定
|
||||
|
||||
| 门禁 | 当前结论 |
|
||||
| --- | --- |
|
||||
| 本地学术化门禁 A01-A05 | 可通过 |
|
||||
| 标准矩阵 S01-S06 | 可通过 |
|
||||
| 标准矩阵 S07 live Azure/NewAPI | 已通过 |
|
||||
| 生产级标准化验收 | 当前最小标准已满足 |
|
||||
|
||||
结论:当前项目已经具备可执行的标准化测试,且本地确定性蜂群行为测试与真实 Azure/NewAPI live 集成测试均已通过。该结论限定在本仓库定义的最小蜂群标准矩阵内,不等同于大规模生产压测或第三方认证。
|
||||
|
||||
## 6. 算法清单
|
||||
|
||||
| 算法 / 机制 | 位置 | 说明 |
|
||||
| --- | --- | --- |
|
||||
| 能力匹配任务 claim | `InMemorySwarmStore.claim_next` | Agnet 只领取能力匹配的 pending 任务 |
|
||||
| 信息素排序 | `InMemorySwarmStore.claim_next` | pending 任务按 pheromone score 从高到低排序 |
|
||||
| 正/负反馈更新 | `complete_task` / `fail_task` | 成功加分,失败扣分 |
|
||||
| 最高分收敛 | `converge` | 在完成任务中选择最高 score 作为最终输出 |
|
||||
| 多轮加权共识 | `ConsensusSwarm.run` | 按 agent weight 和 confidence 累积分数 |
|
||||
| 分数蒸发 | `ConsensusSwarm._evaporate_scores` | 每轮按 evaporation 衰减历史候选分 |
|
||||
| 动态模型发现 | `discover_newapi_models` | 从 NewAPI 兼容端点发现模型 |
|
||||
| 去重模型选择 | `select_distinct_models` | 多 Agnet 测试时选择不同模型 |
|
||||
|
||||
## 7. 马尔可夫过程结论
|
||||
|
||||
当前实现满足“工程意义上的马尔可夫式状态机”:
|
||||
|
||||
- 如果把 `tasks`、`pheromones`、`shared_state`、`observations`、agent policy 和共识轮次一起视为完整当前状态,则下一步 claim、score 更新和收敛选择只依赖当前状态。
|
||||
- 新增测试 `tests/test_markov_process_properties.py` 验证了等价当前任务/信息素状态下,不同历史路径会得到相同 claim 与 score 更新。
|
||||
|
||||
但它不满足严格数学意义的 Markov process / Markov Decision Process:
|
||||
|
||||
- 没有定义状态转移概率核 `P(s_next | s_current)`。
|
||||
- 没有正式 action space、reward function 和策略优化目标。
|
||||
- UUID、时间戳、外部 NewAPI/LLM 调用没有建模为随机变量。
|
||||
- 部分输出保留历史观测作为审计证据,这属于工程可追溯性,不是形式化随机过程。
|
||||
@@ -0,0 +1,141 @@
|
||||
# Agent / 蜂群 Agent 质量标准 v1
|
||||
|
||||
**版本**: AQS/SW-AQS v1
|
||||
**对象**: `swarm-minimal` 最小蜂群原型
|
||||
**目的**: 用 Agent 行业通用风险与能力框架,加上本项目自定义蜂群 Agent 标准,评价这个系统是否像一个可控、多 Agent、可交接、可观测、可收敛的 Agent 系统。
|
||||
|
||||
## 1. 边界说明
|
||||
|
||||
这份标准不是普通软件开发质量标准。
|
||||
|
||||
`py_compile`、`unittest`、场景脚本只作为“证据采集工具”,不作为最终质量标准本身。最终评价对象是:
|
||||
|
||||
- 单个 Agent 是否能理解任务、保持上下文、使用工具、产出可验证结果。
|
||||
- 多个 Agent 是否能通过共享环境、信息素、handoff 和收敛机制形成蜂群行为。
|
||||
- Agent 运行过程是否能被审计、复现、拒绝越权、避免明文密钥泄露。
|
||||
|
||||
## 2. 行业参考来源
|
||||
|
||||
当前行业没有统一的“Agent 质量认证”。本项目采用“行业参考 + 本项目配置标准”的方式:
|
||||
|
||||
| 来源 | 本项目采用的 Agent 质量含义 |
|
||||
| --- | --- |
|
||||
| NIST AI RMF 1.0 | 用 Govern / Map / Measure / Manage 管理 Agent 风险、可控性和可度量性 |
|
||||
| NIST AI 600-1 Generative AI Profile | 关注生成式 AI 的幻觉、数据隐私、信息安全、供应链/组件、预部署测试和事件披露 |
|
||||
| OWASP LLM Top 10 | 覆盖 prompt injection、敏感信息泄露、过度代理、过度信任、不安全输出处理 |
|
||||
| OWASP Agentic Skills Top 10 | 覆盖 Agent 技能/工具执行层的权限、隔离、审计、供应链和运行时安全 |
|
||||
| MITRE ATLAS | 用 AI 攻击/误用视角设计失败、越权、诱导和异常场景 |
|
||||
| OpenTelemetry | 用 traces、metrics、logs、events 的思想要求 Agent 过程可观测 |
|
||||
| LangGraph handoff 参考 | 用 active-agent / `transfer_to_<agent>` 语义约束 Agent 交接 |
|
||||
| 蜂群模式文献/文章归纳 | 用去中心化、自组织、局部感知、间接协作、涌现、鲁棒性和收敛作为蜂群标准 |
|
||||
|
||||
## 3. 通用 Agent 质量标准 AQS
|
||||
|
||||
| ID | 质量项 | 合格标准 | 当前证据 |
|
||||
| --- | --- | --- | --- |
|
||||
| AQS-01 任务理解 | Agent 输出必须回应分配给它的具体子任务,不泛泛回答 | S07 每步 task.input 明确当前步骤和要求 |
|
||||
| AQS-02 指令遵循 | 输出必须包含指定 marker、约束、文件路径、验收命令或风险项 | `score_output()` 和 S07 12 项检查 |
|
||||
| AQS-03 上下文忠实 | 除第一步外,Agent 必须引用上一阶段 marker 和 summary | S07 `step_markers_and_previous_links` |
|
||||
| AQS-04 可验证输出 | 输出必须能被规则检查,不能只有自然语言主张 | S03/S05/S07 checks |
|
||||
| AQS-05 模型选择可控 | 多 Agent 不写死 `NEWAPI_MODEL`,必须从模型列表发现并选择 | S07 `three_distinct_models_from_discovery` |
|
||||
| AQS-06 工具/资源边界 | 不把 NATS/Cosmos 等未批准依赖变成 MVP 必需项 | S04/S07 `no_required_nats_or_cosmos` |
|
||||
| AQS-07 敏感信息保护 | `.env`、密钥、连接串不进入 Git、日志、报告或模型输出 | 配置脱敏测试、报告只输出 redacted/路径 |
|
||||
| AQS-08 可观测性 | run_id、task status、score、observation、artifact、stream event 可查 | S07 PostgreSQL/Redis/Blob 证据 |
|
||||
| AQS-09 错误可解释 | 失败任务必须记录 error、failed observation 和负信息素 | S06/B01/C01 |
|
||||
| AQS-10 交接准备度 | 输出要给下一个 Agent 留出摘要、风险和下一步 | S07 prompt/output 中强制“下一步/交接” |
|
||||
| AQS-11 外部依赖真实性 | live 测试必须真实连 PostgreSQL、Redis、Blob、NewAPI | S07 live integration PASS |
|
||||
| AQS-12 人类审计友好 | 最终报告必须能回答:任务、输入、输出、接手、结果、未满足项 | `MODEL_AGNET_IO_REPORT.zh-CN.md` 和本文件 |
|
||||
|
||||
## 4. 蜂群 Agent 质量标准 SW-AQS
|
||||
|
||||
| ID | 蜂群质量项 | 合格标准 | 当前证据 / 缺口 |
|
||||
| --- | --- | --- | --- |
|
||||
| SW-AQS-01 多角色能力 | 至少 3 个不同 Agent/模型参与,角色或步骤不同 | S07 3 个模型、7 个连续步骤 |
|
||||
| SW-AQS-02 共享任务池 | 任务进入统一 task pool,Agent 从池中 claim | PostgreSQL `swarm_tasks` |
|
||||
| SW-AQS-03 局部感知 | Agent 输入包含当前任务、上一阶段 summary 或 shared_state keys | S07 user prompt shape |
|
||||
| SW-AQS-04 自主 claim | Agent 根据 capability/任务状态领取任务 | `claim_next()`;live 路径仍是 coordinator 顺序驱动,部分满足 |
|
||||
| SW-AQS-05 信息素/分数 | 每个任务完成后写入正分,失败写入负分 | `swarm_pheromones` + Redis sorted set |
|
||||
| SW-AQS-06 间接协作 | 后续 Agent 通过 shared_state/summary/pheromone 感知前序结果 | S07 chain summary 和 cursor |
|
||||
| SW-AQS-07 handoff 连续性 | 交接必须有 from->to/previous->current edge,并保留 payload | B04 与 S07 `chain_edge` |
|
||||
| SW-AQS-08 广播/事件 | claim、done、converged 等事件进入 stream/outbox | S07 Redis Stream 事件数检查 |
|
||||
| SW-AQS-09 收敛条件 | 不能只说“跑完”;必须检查任务、分数、状态、artifact、事件和内容质量 | S07 12 项 checks |
|
||||
| SW-AQS-10 鲁棒性 | 单个 Agent 失败不应吞掉整体状态,失败要可观测 | B01/C01/S06 |
|
||||
| SW-AQS-11 涌现性 | 群体聚合结果能超过单个局部强信号 | B02/C02 |
|
||||
| SW-AQS-12 传统基线对比 | 必须和单 Agent/FIFO/无共享状态基线比较 | C01-C04 |
|
||||
| SW-AQS-13 live 外部闭环 | 不能只 mock,至少一次真实资源闭环 | S07 PASS |
|
||||
| SW-AQS-14 马尔可夫式状态 | 给定完整当前状态,下一步工程转移由当前状态决定 | M01-M03,非严格 MDP |
|
||||
| SW-AQS-15 扩缩容与并发 | 3/5/7 Agent 并发自主 claim 下仍稳定 | 仍待补充 |
|
||||
|
||||
## 5. S07 live 测试任务定义
|
||||
|
||||
总任务:
|
||||
|
||||
```text
|
||||
连续性长推理场景:为 swarm-minimal 设计可恢复的大规模代码任务推理链
|
||||
```
|
||||
|
||||
目标不是让模型自由聊天,而是让 7 个连续 Agent 阶段接力完成一个工程推理链:
|
||||
|
||||
| 步骤 | 分配任务 | 主要验证点 |
|
||||
| --- | --- | --- |
|
||||
| STEP-01 | 界定问题和不可变约束 | 建立目标、输入输出、不变量和 Azure 资源边界 |
|
||||
| STEP-02 | 建立依赖图和状态模型 | 承接 STEP-01,给出任务依赖图、共享状态、租约、状态转移 |
|
||||
| STEP-03 | 设计连续调度算法 | 承接 STEP-02,设计调度、信息素更新、收敛算法和复杂度 |
|
||||
| STEP-04 | 构造反例和失败场景 | 承接 STEP-03,构造慢模型、重复任务、状态倒退、分数误导 |
|
||||
| STEP-05 | 修正算法和恢复策略 | 承接 STEP-04,加入幂等、重试、死信、outbox、重连恢复 |
|
||||
| STEP-06 | 落到文件级实现计划 | 承接 STEP-05,引用真实文件和测试文件 |
|
||||
| STEP-07 | 最终收敛和验收判定 | 承接 STEP-06,给出命令、指标、失败判定和上线前结论 |
|
||||
|
||||
模型分配来自动态发现,不写死 `NEWAPI_MODEL`:
|
||||
|
||||
| 步骤 | Agnet | primary / used model | 交接边 |
|
||||
| --- | --- | --- | --- |
|
||||
| STEP-01 | `continuous-agnet-1` | `deepseek-v4-flash` | `START->STEP-01` |
|
||||
| STEP-02 | `continuous-agnet-2` | `claude-haiku-4-5-20251001` | `STEP-01->STEP-02` |
|
||||
| STEP-03 | `continuous-agnet-3` | `claude-sonnet-4-6` | `STEP-02->STEP-03` |
|
||||
| STEP-04 | `continuous-agnet-4` | `deepseek-v4-flash` | `STEP-03->STEP-04` |
|
||||
| STEP-05 | `continuous-agnet-5` | `claude-haiku-4-5-20251001` | `STEP-04->STEP-05` |
|
||||
| STEP-06 | `continuous-agnet-6` | `claude-sonnet-4-6` | `STEP-05->STEP-06` |
|
||||
| STEP-07 | `continuous-agnet-7` | `deepseek-v4-flash` | `STEP-06->STEP-07` |
|
||||
|
||||
## 6. 模型输入输出与接手机制
|
||||
|
||||
完整输入输出见 `MODEL_AGNET_IO_REPORT.zh-CN.md`。核心机制如下:
|
||||
|
||||
1. wrapper 给当前 Agnet 的 user prompt 注入:
|
||||
- `Previous marker`
|
||||
- `Previous summary`
|
||||
- `Task kind`
|
||||
- 完整 `Task input`
|
||||
2. 当前 Agnet 输出必须带:
|
||||
- `chain_edge=<previous>-><current>`
|
||||
- `primary_model`
|
||||
- `used_model`
|
||||
- `model_selection=discovered_models_not_NEWAPI_MODEL`
|
||||
3. 当前 Agnet 完成后,系统写入:
|
||||
- `chain:{run_id}:{STEP}:summary`
|
||||
- `chain:{run_id}:cursor`
|
||||
- `chain:{run_id}:edge:<previous>-><current>=done`
|
||||
4. 下一个 Agnet 读取上一步 summary 和 marker 后继续执行。
|
||||
|
||||
因此“接手”不是人工解释,也不是模型凭空猜测,而是通过 PostgreSQL shared_state 中的 summary、cursor、edge 字段完成。
|
||||
|
||||
## 7. 本轮判定
|
||||
|
||||
最新标准矩阵 run:
|
||||
|
||||
```json
|
||||
{
|
||||
"run_id": "3e8e58ae4e084bc8b90cf5c46f8992f3",
|
||||
"completed_tasks": 7,
|
||||
"accepted_score": 1.0,
|
||||
"check_count": 12,
|
||||
"failed_checks": []
|
||||
}
|
||||
```
|
||||
|
||||
结论:
|
||||
|
||||
- 按 AQS 通用 Agent 标准:当前最小测试通过。
|
||||
- 按 SW-AQS 蜂群 Agent 标准:关键闭环通过,但“并发自主 claim、广播驱动决策、3/5/7 扩缩容压测”仍是下一阶段。
|
||||
- 所以它满足“最小蜂群 Agent 质量标准 v1”,还不等于完整生产级蜂群平台认证。
|
||||
@@ -0,0 +1,50 @@
|
||||
# Agent 行业参考标准映射
|
||||
|
||||
当前行业没有统一的 “Agnet/Agent 质量认证标准”。本项目不再用普通软件产品质量标准作为主标准,而是采用:
|
||||
|
||||
- `AGENT_SWARM_QUALITY_STANDARD.zh-CN.md`:主标准,定义通用 Agent 质量标准 AQS 和蜂群 Agent 质量标准 SW-AQS。
|
||||
- `ACADEMIC_STANDARD_TEST_REPORT.zh-CN.md`:执行报告,记录本轮测试过程、结果、算法和马尔可夫过程判定。
|
||||
- `MODEL_AGNET_IO_REPORT.zh-CN.md`:输入输出报告,记录模型任务、prompt 结构、实际输出和接手机制。
|
||||
|
||||
## 行业参考框架
|
||||
|
||||
| 来源 | 在本项目中的用途 |
|
||||
| --- | --- |
|
||||
| NIST AI RMF 1.0 | 管理 Agent 风险、可控性、可度量性和治理闭环 |
|
||||
| NIST AI 600-1 | 处理生成式 AI 的幻觉、隐私、信息安全、组件集成和预部署测试 |
|
||||
| OWASP LLM Top 10 | 覆盖 prompt injection、敏感信息泄露、过度代理、不安全输出处理、过度信任 |
|
||||
| OWASP Agentic Skills Top 10 | 覆盖 Agent 工具/技能执行层的权限、隔离、审计和运行时安全 |
|
||||
| MITRE ATLAS | 从对抗性 AI 和误用角度设计失败、越权、诱导和异常场景 |
|
||||
| OpenTelemetry | 约束 run_id、task、event、metric、log、artifact 等可观测证据 |
|
||||
| LangGraph handoff | 参考 active-agent 和 `transfer_to_<agent>` 式交接语义 |
|
||||
| 蜂群模式归纳标准 | 约束去中心化、自组织、局部感知、信息素、涌现、鲁棒性和收敛 |
|
||||
|
||||
## 最小通过条件
|
||||
|
||||
本项目当前最小通过条件不是“代码质量通过”,而是:
|
||||
|
||||
1. AQS 通用 Agent 标准中,任务理解、上下文、模型选择、资源边界、敏感信息、可观测性、错误可解释、交接准备度均有证据。
|
||||
2. SW-AQS 蜂群 Agent 标准中,任务池、共享状态、信息素、handoff、事件流、收敛、鲁棒性、涌现性、传统基线对比和 live 外部闭环均有证据。
|
||||
3. 模型输入输出能被审计:知道每个 Agent 被分配了什么任务、收到了什么输入、输出了什么、下一个 Agent 如何接手。
|
||||
4. live 测试真实连接 Azure PostgreSQL、Redis、Blob 和 NewAPI,且不输出任何真实密钥。
|
||||
|
||||
最新一次标准矩阵已经通过:
|
||||
|
||||
```json
|
||||
{
|
||||
"run_id": "3e8e58ae4e084bc8b90cf5c46f8992f3",
|
||||
"completed_tasks": 7,
|
||||
"accepted_score": 1.0,
|
||||
"check_count": 12,
|
||||
"failed_checks": []
|
||||
}
|
||||
```
|
||||
|
||||
## 保留缺口
|
||||
|
||||
当前满足“最小蜂群 Agent 质量标准 v1”,但还不是完整生产级蜂群平台认证。仍需补:
|
||||
|
||||
- 并发 worker 自主 claim,而不是 coordinator 顺序驱动。
|
||||
- Redis Stream 事件被其他 Agent 消费并改变决策。
|
||||
- 3/5/7 Agent 扩缩容和稳定性压测。
|
||||
- 收敛速度、Agent 利用率、响应时间、信息素分布等统一 metrics。
|
||||
File diff suppressed because it is too large
Load Diff
@@ -0,0 +1,15 @@
|
||||
# 文档索引
|
||||
|
||||
## Agent / 蜂群 Agent 测试
|
||||
|
||||
- `AGENT_SWARM_QUALITY_STANDARD.zh-CN.md`:AQS / SW-AQS v1 主标准,说明 Agent 和蜂群 Agent 的质量项、S07 任务分配、模型交接和当前结论。
|
||||
- `ACADEMIC_STANDARD_TEST_REPORT.zh-CN.md`:测试场景、测试过程、测试结果、算法清单和马尔可夫过程判定。
|
||||
- `MODEL_AGNET_IO_REPORT.zh-CN.md`:从 live run 导出的模型 / Agnet 任务、输入、输出、评分和交接过程。
|
||||
- `STANDARD_TEST_MATRIX.md`:S01-S07 标准化测试矩阵。
|
||||
- `INDUSTRY_STANDARD_AGNET_TEST_PLAN.md`:行业参考框架到本项目 Agent 质量标准的映射。
|
||||
|
||||
## 交付计划
|
||||
|
||||
- `01-ten-day-delivery-plan.md`:10 天交付计划。
|
||||
- `02-person-task-interface-matrix.md`:人员、任务和接口矩阵。
|
||||
- `03-minimal-validation-and-acceptance.md`:最小验证和验收流程。
|
||||
@@ -11,6 +11,7 @@ result means more than "one long run completed".
|
||||
| Static | Python syntax and importability | `./.venv/bin/python -B -m py_compile swarm_minimal/*.py examples/*.py tests/*.py` |
|
||||
| Unit | Existing closed-loop and NewAPI mock behavior | `./.venv/bin/python -B -m unittest discover -s tests` |
|
||||
| Deterministic scenarios | No-network Given/When/Then cases for continuity, policy, final scoring, and failure injection | `./.venv/bin/python -B -m unittest tests.test_standard_scenarios` |
|
||||
| Academic Markov fit | No-network checks for Markov-style transition behavior and non-MDP limitations | `./.venv/bin/python -B -m unittest tests.test_markov_process_properties` |
|
||||
| Live integration | Real NewAPI + Azure PostgreSQL/Redis/Blob continuous reasoning chain | `./.venv/bin/python -u -B examples/run_continuous_reasoning_acceptance.py` |
|
||||
|
||||
## Required Scenarios
|
||||
@@ -24,6 +25,9 @@ result means more than "one long run completed".
|
||||
| S05 | Final convergence scoring | STEP-06 and STEP-07 both produce rich outputs | Score both outputs | STEP-07 is strictly selected as final convergence |
|
||||
| S06 | Failure injection | One deterministic worker raises | Coordinator runs to convergence | Failed task is marked failed, pheromone is negative, completed task still converges |
|
||||
| S07 | Live resource closure | Real Azure/NewAPI env is loaded | Run seven-step continuous reasoning | PostgreSQL, Redis, Blob, stream events, model discovery, and final artifact all pass |
|
||||
| M01 | Markov-style claim transition | Same current tasks and pheromone values but different prior paths | Claim next task | The same high-pheromone task is claimed |
|
||||
| M02 | Markov-style score transition | Same current task, agent, output, and score but different prior paths | Complete task | Task status, output, score, and pheromone update match |
|
||||
| M03 | Formal MDP limitation | Project metadata is evaluated | Assess Markov fit | Prototype is classified as Markov-style state machine, not a formal Markov process or MDP |
|
||||
|
||||
## Stop Rule
|
||||
|
||||
@@ -10,24 +10,26 @@ sys.path.insert(0, str(ROOT))
|
||||
|
||||
from swarm_minimal.azure_store import PostgresRedisBlobSwarmStore
|
||||
from swarm_minimal.config import SwarmConfig
|
||||
from swarm_minimal.local_env import load_env_file
|
||||
from swarm_minimal.local_env import load_project_env
|
||||
|
||||
|
||||
RUN_IDS = [
|
||||
"3e8e58ae4e084bc8b90cf5c46f8992f3",
|
||||
"78f189ccd1924ed0a4fb0a0a447ad449",
|
||||
"623b5f6e5cc24cc7967fd9577f9c224b",
|
||||
"42688e7b6245466dab8398dfe4790456",
|
||||
]
|
||||
|
||||
OUTPUT_PATH = ROOT / "MODEL_AGNET_IO_REPORT.zh-CN.md"
|
||||
OUTPUT_PATH = ROOT / "docs" / "MODEL_AGNET_IO_REPORT.zh-CN.md"
|
||||
|
||||
|
||||
def main() -> None:
|
||||
load_env_file(ROOT / ".env")
|
||||
load_project_env(ROOT)
|
||||
store = PostgresRedisBlobSwarmStore(SwarmConfig.from_env())
|
||||
try:
|
||||
store.ensure_schema()
|
||||
report = build_report(store)
|
||||
OUTPUT_PATH.parent.mkdir(exist_ok=True)
|
||||
OUTPUT_PATH.write_text(report, encoding="utf-8")
|
||||
print(str(OUTPUT_PATH))
|
||||
finally:
|
||||
@@ -50,7 +52,7 @@ def build_report(store: PostgresRedisBlobSwarmStore) -> str:
|
||||
task_ids = [item["task_id"] for item in convergence["observations"]]
|
||||
tasks = fetch_tasks(store, task_ids)
|
||||
sections.extend(render_run(convergence, tasks))
|
||||
return "\n".join(sections).rstrip() + "\n"
|
||||
return "\n".join(line.rstrip() for line in sections).rstrip() + "\n"
|
||||
|
||||
|
||||
def render_run(convergence: dict[str, object], tasks: dict[str, dict[str, object]]) -> list[str]:
|
||||
@@ -94,6 +96,10 @@ def render_run(convergence: dict[str, object], tasks: dict[str, dict[str, object
|
||||
user_prompt_shape_for_goal(goal),
|
||||
"```",
|
||||
"",
|
||||
"#### 接手 / 交接机制",
|
||||
"",
|
||||
handoff_description(goal, run_id, task, output),
|
||||
"",
|
||||
"#### 本次任务输入 task.input",
|
||||
"",
|
||||
"```text",
|
||||
@@ -111,6 +117,37 @@ def render_run(convergence: dict[str, object], tasks: dict[str, dict[str, object
|
||||
return lines
|
||||
|
||||
|
||||
def handoff_description(goal: str, run_id: str, task: dict[str, object], output: str) -> str:
|
||||
if goal.startswith("连续性长推理场景"):
|
||||
match = re.search(r"chain_edge=([^;\\n]+)", output)
|
||||
edge = match.group(1).strip() if match else "<missing>"
|
||||
marker = edge.split("->")[-1] if "->" in edge else str(task.get("kind") or "<unknown>")
|
||||
return "\n".join(
|
||||
[
|
||||
f"- 当前输出前缀记录 `chain_edge={edge}`,证明本 Agnet 承接了上一阶段。",
|
||||
"- 调用前,wrapper 会把 `Previous marker` 和 `Previous summary` 放入 user prompt。",
|
||||
f"- 执行后,wrapper 把输出摘要写入 `chain:{run_id}:{marker}:summary`。",
|
||||
f"- 同时推进 `chain:{run_id}:cursor`,并写入 `chain:{run_id}:edge:{edge}=done`。",
|
||||
"- 下一个 Agnet 读取这个 summary 和 edge 后继续执行,所以接手不是靠口头描述,而是靠共享状态字段完成。",
|
||||
]
|
||||
)
|
||||
if goal.startswith("真实全面场景"):
|
||||
return "\n".join(
|
||||
[
|
||||
"- 每个 Agnet 从 PostgreSQL 任务池 claim 自己的子任务。",
|
||||
"- 共享状态键列表作为上下文输入,让后续任务能看到已有 run/task/agent 状态。",
|
||||
"- 完成后写回 task output、score、observation 和 Redis Stream 事件,供收敛阶段读取。",
|
||||
]
|
||||
)
|
||||
return "\n".join(
|
||||
[
|
||||
"- Agnet 从共享任务池 claim 与自身 capability 匹配的任务。",
|
||||
"- 完成后写回 output、score、observation、heartbeat 和 pheromone 分数。",
|
||||
"- 后续 Agnet 通过 shared state、task status 和 pheromone score 感知前序执行结果。",
|
||||
]
|
||||
)
|
||||
|
||||
|
||||
def system_prompt_for_goal(goal: str) -> str:
|
||||
if goal.startswith("full live test"):
|
||||
return (
|
||||
|
||||
@@ -0,0 +1,174 @@
|
||||
from __future__ import annotations
|
||||
|
||||
from pathlib import Path
|
||||
import json
|
||||
import subprocess
|
||||
import sys
|
||||
|
||||
|
||||
ROOT = Path(__file__).resolve().parents[1]
|
||||
sys.path.insert(0, str(ROOT))
|
||||
|
||||
from swarm_minimal.academic_evaluation import ( # noqa: E402
|
||||
ACADEMIC_STANDARD_SOURCES,
|
||||
ALGORITHMS_USED,
|
||||
assess_markov_process_fit,
|
||||
)
|
||||
from swarm_minimal.local_env import find_project_env # noqa: E402
|
||||
|
||||
|
||||
CHECKS = [
|
||||
{
|
||||
"id": "A01",
|
||||
"name": "static_compile",
|
||||
"layer": "static",
|
||||
"command": [
|
||||
sys.executable,
|
||||
"-B",
|
||||
"-m",
|
||||
"py_compile",
|
||||
*sorted(str(path.relative_to(ROOT)) for path in ROOT.glob("swarm_minimal/*.py")),
|
||||
*sorted(str(path.relative_to(ROOT)) for path in ROOT.glob("examples/*.py")),
|
||||
*sorted(str(path.relative_to(ROOT)) for path in ROOT.glob("tests/*.py")),
|
||||
],
|
||||
},
|
||||
{
|
||||
"id": "A02",
|
||||
"name": "unit_and_deterministic_scenarios",
|
||||
"layer": "unit-scenario",
|
||||
"command": [sys.executable, "-B", "-m", "unittest", "discover", "-s", "tests"],
|
||||
},
|
||||
{
|
||||
"id": "A03",
|
||||
"name": "swarm_behavior_acceptance",
|
||||
"layer": "behavior",
|
||||
"command": [sys.executable, "-u", "-B", "examples/run_swarm_behavior_acceptance.py"],
|
||||
"parse_json": True,
|
||||
},
|
||||
{
|
||||
"id": "A04",
|
||||
"name": "swarm_vs_traditional_benchmark",
|
||||
"layer": "benchmark",
|
||||
"command": [sys.executable, "-u", "-B", "examples/run_swarm_vs_traditional_benchmark.py"],
|
||||
"parse_json": True,
|
||||
},
|
||||
{
|
||||
"id": "A05",
|
||||
"name": "consensus_convergence_acceptance",
|
||||
"layer": "consensus",
|
||||
"command": [sys.executable, "-u", "-B", "examples/run_consensus_convergence_acceptance.py"],
|
||||
"parse_json": True,
|
||||
},
|
||||
]
|
||||
|
||||
|
||||
def main() -> None:
|
||||
results = [run_check(check) for check in CHECKS]
|
||||
env_path = find_project_env(ROOT)
|
||||
live_ready = env_path is not None
|
||||
markov = assess_markov_process_fit()
|
||||
report = {
|
||||
"standard": "academic-standard-evaluation-v1",
|
||||
"status": "PASS" if all(item["passed"] for item in results) else "FAIL",
|
||||
"scope": {
|
||||
"deterministic_local": True,
|
||||
"live_azure_newapi": "ready" if live_ready else "blocked_missing_.env",
|
||||
"env_path": str(env_path.relative_to(ROOT)) if env_path else None,
|
||||
"live_note": (
|
||||
"S07 live integration requires Azure PostgreSQL, Redis, Blob and NewAPI credentials in an ignored .env."
|
||||
),
|
||||
},
|
||||
"standards": ACADEMIC_STANDARD_SOURCES,
|
||||
"checks": results,
|
||||
"algorithms_used": ALGORITHMS_USED,
|
||||
"markov_process_assessment": {
|
||||
"markov_style_state_machine": markov.markov_style_state_machine,
|
||||
"formal_markov_process": markov.formal_markov_process,
|
||||
"formal_markov_decision_process": markov.formal_markov_decision_process,
|
||||
"sufficient_state": markov.sufficient_state,
|
||||
"limiting_factors": markov.limiting_factors,
|
||||
"conclusion": markov.conclusion,
|
||||
},
|
||||
"pass_condition": {
|
||||
"local_academic_gate": "all A01-A05 checks pass",
|
||||
"full_standard_gate": "local_academic_gate plus S07 live Azure/NewAPI scenario",
|
||||
},
|
||||
}
|
||||
print(json.dumps(report, ensure_ascii=False, indent=2))
|
||||
if report["status"] != "PASS":
|
||||
raise SystemExit(1)
|
||||
|
||||
|
||||
def run_check(check: dict[str, object]) -> dict[str, object]:
|
||||
command = check["command"]
|
||||
assert isinstance(command, list)
|
||||
completed = subprocess.run(
|
||||
command,
|
||||
cwd=ROOT,
|
||||
text=True,
|
||||
capture_output=True,
|
||||
timeout=900,
|
||||
)
|
||||
parsed = parse_last_json(completed.stdout) if check.get("parse_json") else None
|
||||
evidence: dict[str, object] = {"tail": summarize_process_output(completed.stdout, completed.stderr)}
|
||||
if parsed:
|
||||
evidence = summarize_json(parsed)
|
||||
return {
|
||||
"id": check["id"],
|
||||
"name": check["name"],
|
||||
"layer": check["layer"],
|
||||
"command": " ".join(command),
|
||||
"passed": completed.returncode == 0,
|
||||
"evidence": evidence,
|
||||
}
|
||||
|
||||
|
||||
def summarize_process_output(stdout: str, stderr: str) -> str:
|
||||
combined = "\n".join(part.strip() for part in [stdout, stderr] if part.strip())
|
||||
return combined[-1200:] if combined else "<no output>"
|
||||
|
||||
|
||||
def parse_last_json(text: str) -> dict[str, object] | None:
|
||||
stripped = text.strip()
|
||||
if not stripped:
|
||||
return None
|
||||
decoder = json.JSONDecoder()
|
||||
last = None
|
||||
index = 0
|
||||
while index < len(stripped):
|
||||
brace = stripped.find("{", index)
|
||||
if brace == -1:
|
||||
break
|
||||
try:
|
||||
value, end = decoder.raw_decode(stripped[brace:])
|
||||
except json.JSONDecodeError:
|
||||
index = brace + 1
|
||||
continue
|
||||
if isinstance(value, dict):
|
||||
last = value
|
||||
index = brace + end
|
||||
return last
|
||||
|
||||
|
||||
def summarize_json(value: dict[str, object]) -> dict[str, object]:
|
||||
summary: dict[str, object] = {"status": value.get("status")}
|
||||
if "standard" in value:
|
||||
summary["standard"] = value["standard"]
|
||||
if "overall_normalized_score" in value:
|
||||
summary["overall_normalized_score"] = value["overall_normalized_score"]
|
||||
if "accepted_candidate" in value:
|
||||
summary["accepted_candidate"] = value["accepted_candidate"]
|
||||
if "rounds" in value and isinstance(value["rounds"], list):
|
||||
summary["round_count"] = len(value["rounds"])
|
||||
if "scenarios" in value and isinstance(value["scenarios"], list):
|
||||
summary["scenario_count"] = len(value["scenarios"])
|
||||
summary["failed_scenarios"] = [
|
||||
item.get("id")
|
||||
for item in value["scenarios"]
|
||||
if isinstance(item, dict) and not item.get("passed", False)
|
||||
]
|
||||
return summary
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
@@ -9,11 +9,11 @@ sys.path.insert(0, str(ROOT))
|
||||
from swarm_minimal.azure_store import PostgresRedisBlobSwarmStore
|
||||
from swarm_minimal.config import SwarmConfig
|
||||
from swarm_minimal.core import SwarmCoordinator, default_agents
|
||||
from swarm_minimal.local_env import load_env_file
|
||||
from swarm_minimal.local_env import load_project_env
|
||||
|
||||
|
||||
def main() -> None:
|
||||
load_env_file(ROOT / ".env")
|
||||
load_project_env(ROOT)
|
||||
config = SwarmConfig.from_env()
|
||||
print(json.dumps(config.redacted_summary(), ensure_ascii=False, indent=2))
|
||||
|
||||
|
||||
@@ -10,7 +10,7 @@ sys.path.insert(0, str(ROOT))
|
||||
from swarm_minimal.azure_store import PostgresRedisBlobSwarmStore
|
||||
from swarm_minimal.config import SwarmConfig
|
||||
from swarm_minimal.core import Agent, SwarmCoordinator, Task
|
||||
from swarm_minimal.local_env import load_env_file
|
||||
from swarm_minimal.local_env import load_project_env
|
||||
from swarm_minimal.newapi_agnet import (
|
||||
NewApiAgnet,
|
||||
NewApiChannelConfig,
|
||||
@@ -103,7 +103,7 @@ ACCEPTANCE_CRITERIA = [
|
||||
|
||||
|
||||
def main() -> None:
|
||||
load_env_file(ROOT / ".env")
|
||||
load_project_env(ROOT)
|
||||
azure_config = SwarmConfig.from_env()
|
||||
newapi_config = NewApiChannelConfig.from_env()
|
||||
if newapi_config.timeout_seconds < 360:
|
||||
|
||||
@@ -9,7 +9,7 @@ sys.path.insert(0, str(ROOT))
|
||||
from swarm_minimal.azure_store import PostgresRedisBlobSwarmStore
|
||||
from swarm_minimal.config import SwarmConfig
|
||||
from swarm_minimal.core import SwarmCoordinator, Task
|
||||
from swarm_minimal.local_env import load_env_file
|
||||
from swarm_minimal.local_env import load_project_env
|
||||
from swarm_minimal.newapi_agnet import (
|
||||
NewApiChannelConfig,
|
||||
build_model_test_agnets,
|
||||
@@ -19,7 +19,7 @@ from swarm_minimal.newapi_agnet import (
|
||||
|
||||
|
||||
def main() -> None:
|
||||
load_env_file(ROOT / ".env")
|
||||
load_project_env(ROOT)
|
||||
azure_config = SwarmConfig.from_env()
|
||||
newapi_config = NewApiChannelConfig.from_env()
|
||||
print("azure_config:")
|
||||
|
||||
@@ -10,7 +10,7 @@ sys.path.insert(0, str(ROOT))
|
||||
from swarm_minimal.azure_store import PostgresRedisBlobSwarmStore
|
||||
from swarm_minimal.config import SwarmConfig
|
||||
from swarm_minimal.core import Agent, SwarmCoordinator, Task
|
||||
from swarm_minimal.local_env import load_env_file
|
||||
from swarm_minimal.local_env import load_project_env
|
||||
from swarm_minimal.newapi_agnet import (
|
||||
NewApiAgnet,
|
||||
NewApiChannelConfig,
|
||||
@@ -36,7 +36,7 @@ Return a compact JSON-like answer. Do not include secrets.
|
||||
|
||||
|
||||
def main() -> None:
|
||||
load_env_file(ROOT / ".env")
|
||||
load_project_env(ROOT)
|
||||
azure_config = SwarmConfig.from_env()
|
||||
newapi_config = NewApiChannelConfig.from_env()
|
||||
if newapi_config.timeout_seconds < 120:
|
||||
|
||||
@@ -10,7 +10,7 @@ sys.path.insert(0, str(ROOT))
|
||||
from swarm_minimal.azure_store import PostgresRedisBlobSwarmStore
|
||||
from swarm_minimal.config import SwarmConfig
|
||||
from swarm_minimal.core import Agent, SwarmCoordinator, Task
|
||||
from swarm_minimal.local_env import load_env_file
|
||||
from swarm_minimal.local_env import load_project_env
|
||||
from swarm_minimal.newapi_agnet import (
|
||||
NewApiAgnet,
|
||||
NewApiChannelConfig,
|
||||
@@ -94,7 +94,7 @@ ACCEPTANCE_CRITERIA = [
|
||||
|
||||
|
||||
def main() -> None:
|
||||
load_env_file(ROOT / ".env")
|
||||
load_project_env(ROOT)
|
||||
azure_config = SwarmConfig.from_env()
|
||||
newapi_config = NewApiChannelConfig.from_env()
|
||||
if newapi_config.timeout_seconds < 360:
|
||||
|
||||
@@ -7,7 +7,7 @@ ROOT = Path(__file__).resolve().parents[1]
|
||||
sys.path.insert(0, str(ROOT))
|
||||
|
||||
from swarm_minimal.core import InMemorySwarmStore, SwarmCoordinator, default_agents
|
||||
from swarm_minimal.local_env import load_env_file
|
||||
from swarm_minimal.local_env import load_project_env
|
||||
from swarm_minimal.newapi_agnet import (
|
||||
NewApiAgnet,
|
||||
NewApiChannelConfig,
|
||||
@@ -17,7 +17,7 @@ from swarm_minimal.newapi_agnet import (
|
||||
|
||||
|
||||
def main() -> None:
|
||||
load_env_file(ROOT / ".env")
|
||||
load_project_env(ROOT)
|
||||
config = NewApiChannelConfig.from_env()
|
||||
if not config.model:
|
||||
model = select_distinct_models(discover_newapi_models(config), count=1)[0]
|
||||
|
||||
@@ -10,7 +10,7 @@ sys.path.insert(0, str(ROOT))
|
||||
from swarm_minimal.azure_store import PostgresRedisBlobSwarmStore
|
||||
from swarm_minimal.config import SwarmConfig
|
||||
from swarm_minimal.core import Agent, SwarmCoordinator, Task
|
||||
from swarm_minimal.local_env import load_env_file
|
||||
from swarm_minimal.local_env import load_project_env
|
||||
from swarm_minimal.newapi_agnet import (
|
||||
NewApiAgnet,
|
||||
NewApiChannelConfig,
|
||||
@@ -53,7 +53,7 @@ ACCEPTANCE_CRITERIA = [
|
||||
|
||||
|
||||
def main() -> None:
|
||||
load_env_file(ROOT / ".env")
|
||||
load_project_env(ROOT)
|
||||
azure_config = SwarmConfig.from_env()
|
||||
newapi_config = NewApiChannelConfig.from_env()
|
||||
if newapi_config.timeout_seconds < 120:
|
||||
|
||||
@@ -7,7 +7,7 @@ ROOT = Path(__file__).resolve().parents[1]
|
||||
sys.path.insert(0, str(ROOT))
|
||||
|
||||
from swarm_minimal.core import InMemorySwarmStore, SwarmCoordinator, Task
|
||||
from swarm_minimal.local_env import load_env_file
|
||||
from swarm_minimal.local_env import load_project_env
|
||||
from swarm_minimal.newapi_agnet import (
|
||||
NewApiChannelConfig,
|
||||
build_model_test_agnets,
|
||||
@@ -17,7 +17,7 @@ from swarm_minimal.newapi_agnet import (
|
||||
|
||||
|
||||
def main() -> None:
|
||||
load_env_file(ROOT / ".env")
|
||||
load_project_env(ROOT)
|
||||
config = NewApiChannelConfig.from_env()
|
||||
print(json.dumps(config.redacted_summary(), ensure_ascii=False, indent=2))
|
||||
|
||||
|
||||
@@ -0,0 +1,132 @@
|
||||
"""Academic-style evaluation metadata for the minimal swarm prototype."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
from dataclasses import dataclass
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class MarkovProcessAssessment:
|
||||
"""Result of evaluating whether the prototype satisfies Markov assumptions."""
|
||||
|
||||
markov_style_state_machine: bool
|
||||
formal_markov_process: bool
|
||||
formal_markov_decision_process: bool
|
||||
sufficient_state: tuple[str, ...]
|
||||
limiting_factors: tuple[str, ...]
|
||||
conclusion: str
|
||||
|
||||
|
||||
ACADEMIC_STANDARD_SOURCES = (
|
||||
{
|
||||
"id": "NIST-AI-RMF",
|
||||
"name": "NIST AI Risk Management Framework 1.0",
|
||||
"use": "govern, map, measure and manage risk framing for autonomous AI behavior",
|
||||
"url": "https://www.nist.gov/itl/ai-risk-management-framework",
|
||||
},
|
||||
{
|
||||
"id": "NIST-AI-600-1",
|
||||
"name": "NIST AI RMF Generative AI Profile",
|
||||
"use": "generative-AI risks such as confabulation, privacy, information security and component integration",
|
||||
"url": "https://doi.org/10.6028/NIST.AI.600-1",
|
||||
},
|
||||
{
|
||||
"id": "OWASP-LLM",
|
||||
"name": "OWASP Top 10 for Large Language Model Applications",
|
||||
"use": "sensitive information disclosure, excessive agency and tool-boundary checks",
|
||||
"url": "https://owasp.org/www-project-top-10-for-large-language-model-applications/",
|
||||
},
|
||||
{
|
||||
"id": "OWASP-AST10",
|
||||
"name": "OWASP Agentic Skills Top 10",
|
||||
"use": "agentic skill risk checks for autonomous tools and delegated execution boundaries",
|
||||
"url": "https://owasp.org/www-project-agentic-skills-top-10/",
|
||||
},
|
||||
{
|
||||
"id": "MITRE-ATLAS",
|
||||
"name": "MITRE ATLAS",
|
||||
"use": "adversarial-AI and agent misuse framing for failure, abuse and recovery scenarios",
|
||||
"url": "https://atlas.mitre.org/",
|
||||
},
|
||||
{
|
||||
"id": "OTEL",
|
||||
"name": "OpenTelemetry documentation",
|
||||
"use": "observable traces, metrics, logs and event evidence expectations",
|
||||
"url": "https://opentelemetry.io/docs/",
|
||||
},
|
||||
{
|
||||
"id": "LANGGRAPH-HANDOFF",
|
||||
"name": "LangGraph handoff reference",
|
||||
"use": "active-agent handoff and transfer_to_<agent> continuity reference",
|
||||
"url": "https://reference.langchain.com/python/langgraph-swarm/handoff/create_handoff_tool",
|
||||
},
|
||||
)
|
||||
|
||||
|
||||
ALGORITHMS_USED = (
|
||||
{
|
||||
"name": "capability-based task claiming",
|
||||
"location": "swarm_minimal.core.InMemorySwarmStore.claim_next",
|
||||
"description": "agents claim pending tasks matching their capability; ties are ordered by pheromone score",
|
||||
},
|
||||
{
|
||||
"name": "pheromone / score reinforcement",
|
||||
"location": "swarm_minimal.core.InMemorySwarmStore.complete_task and fail_task",
|
||||
"description": "successful task scores add positive feedback; failed tasks receive negative feedback",
|
||||
},
|
||||
{
|
||||
"name": "winner-take-highest-score convergence",
|
||||
"location": "swarm_minimal.core.InMemorySwarmStore.converge",
|
||||
"description": "the highest-scoring completed task becomes the accepted result",
|
||||
},
|
||||
{
|
||||
"name": "weighted multi-round consensus",
|
||||
"location": "swarm_minimal.core.ConsensusSwarm.run",
|
||||
"description": "role-weighted votes accumulate until leader share and margin thresholds are reached",
|
||||
},
|
||||
{
|
||||
"name": "score evaporation",
|
||||
"location": "swarm_minimal.core.ConsensusSwarm._evaporate_scores",
|
||||
"description": "candidate scores decay between rounds before new evidence is added",
|
||||
},
|
||||
{
|
||||
"name": "distinct model discovery and selection",
|
||||
"location": "swarm_minimal.newapi_agnet.discover_newapi_models and select_distinct_models",
|
||||
"description": "NewAPI models are discovered from compatible endpoints and de-duplicated for multi-agent tests",
|
||||
},
|
||||
)
|
||||
|
||||
|
||||
def assess_markov_process_fit() -> MarkovProcessAssessment:
|
||||
"""Classify the prototype against Markov-process requirements.
|
||||
|
||||
The local swarm can be interpreted as a Markov-style state machine if the
|
||||
complete environment state is treated as the state variable. It is not a
|
||||
formal Markov process or MDP because the implementation does not define a
|
||||
transition probability kernel, action/reward tuple, or stochastic model for
|
||||
external LLM/API behavior.
|
||||
"""
|
||||
|
||||
return MarkovProcessAssessment(
|
||||
markov_style_state_machine=True,
|
||||
formal_markov_process=False,
|
||||
formal_markov_decision_process=False,
|
||||
sufficient_state=(
|
||||
"tasks with status, owner, output, score and error",
|
||||
"pheromone score table",
|
||||
"shared_state key-value environment",
|
||||
"observations already emitted",
|
||||
"agent policy functions and current round index for consensus",
|
||||
),
|
||||
limiting_factors=(
|
||||
"no transition probability kernel P(s_next | s_current)",
|
||||
"no formal action space, reward function or policy optimization objective",
|
||||
"uuid/time and external NewAPI/LLM calls are not modeled as stochastic variables",
|
||||
"some acceptance outputs deliberately preserve history as audit evidence",
|
||||
),
|
||||
conclusion=(
|
||||
"满足工程意义上的马尔可夫式状态转移:给定完整当前状态和 agent policy,"
|
||||
"下一步 claim、score 更新和收敛选择由当前状态决定。"
|
||||
"但不满足严格数学意义的 Markov process / MDP 定义。"
|
||||
),
|
||||
)
|
||||
@@ -1,7 +1,9 @@
|
||||
"""Load local environment files without adding a dependency.
|
||||
|
||||
The intended file is ``.env`` under ``swarm-minimal/``. It is ignored by git and
|
||||
should never be committed.
|
||||
The preferred file is ``.env`` under the project root. For local test runs,
|
||||
``examples/.env`` is also accepted so users can keep live-test credentials next
|
||||
to the example entrypoints. Both paths are ignored by git and should never be
|
||||
committed.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
@@ -10,6 +12,24 @@ from pathlib import Path
|
||||
import os
|
||||
|
||||
|
||||
def find_project_env(root: Path) -> Path | None:
|
||||
"""Return the first supported private env file path for a project."""
|
||||
|
||||
for path in (root / ".env", root / "examples" / ".env"):
|
||||
if path.exists():
|
||||
return path
|
||||
return None
|
||||
|
||||
|
||||
def load_project_env(root: Path, *, override: bool = False) -> int:
|
||||
"""Load the supported project env file if one exists."""
|
||||
|
||||
path = find_project_env(root)
|
||||
if path is None:
|
||||
return 0
|
||||
return load_env_file(path, override=override)
|
||||
|
||||
|
||||
def load_env_file(path: Path, *, override: bool = False) -> int:
|
||||
"""Load KEY=VALUE pairs from a local env file.
|
||||
|
||||
|
||||
@@ -0,0 +1,72 @@
|
||||
import unittest
|
||||
|
||||
from swarm_minimal.academic_evaluation import assess_markov_process_fit
|
||||
from swarm_minimal.core import Agent, InMemorySwarmStore, Observation, Task, TaskStatus
|
||||
|
||||
|
||||
def build_equivalent_claim_store(history_label: str) -> InMemorySwarmStore:
|
||||
store = InMemorySwarmStore()
|
||||
low = Task(kind="probe", input="low", id="task-low")
|
||||
high = Task(kind="probe", input="high", id="task-high")
|
||||
store.add_task(low)
|
||||
store.add_task(high)
|
||||
store.pheromones[low.id] = 0.2
|
||||
store.pheromones[high.id] = 0.8
|
||||
store.shared_state["irrelevant_history_label"] = history_label
|
||||
store.observations.append(
|
||||
Observation(
|
||||
task_id=f"past-{history_label}",
|
||||
agent_id="past-agent",
|
||||
signal="past:done",
|
||||
score_delta=0.1,
|
||||
)
|
||||
)
|
||||
return store
|
||||
|
||||
|
||||
class MarkovProcessPropertyTest(unittest.TestCase):
|
||||
def test_claim_transition_uses_current_task_pheromone_state_not_past_path(self) -> None:
|
||||
agent = Agent(id="probe-agent", capability="probe", run=lambda task, _: ("ok", 0.1))
|
||||
first = build_equivalent_claim_store("path-a")
|
||||
second = build_equivalent_claim_store("path-b")
|
||||
|
||||
first_claim = first.claim_next(agent)
|
||||
second_claim = second.claim_next(agent)
|
||||
|
||||
self.assertIsNotNone(first_claim)
|
||||
self.assertIsNotNone(second_claim)
|
||||
self.assertEqual(first_claim.id, "task-high")
|
||||
self.assertEqual(second_claim.id, "task-high")
|
||||
self.assertEqual(first.shared_state["task:task-high:claimed_by"], "probe-agent")
|
||||
self.assertEqual(second.shared_state["task:task-high:claimed_by"], "probe-agent")
|
||||
|
||||
def test_score_update_depends_on_current_task_agent_and_output(self) -> None:
|
||||
agent = Agent(id="probe-agent", capability="probe", run=lambda task, _: ("ok", 0.4))
|
||||
first = build_equivalent_claim_store("path-a")
|
||||
second = build_equivalent_claim_store("path-b")
|
||||
first_task = first.claim_next(agent)
|
||||
second_task = second.claim_next(agent)
|
||||
assert first_task is not None
|
||||
assert second_task is not None
|
||||
|
||||
first.complete_task(first_task, agent, "same output", 0.4)
|
||||
second.complete_task(second_task, agent, "same output", 0.4)
|
||||
|
||||
self.assertEqual(first_task.status, TaskStatus.DONE)
|
||||
self.assertEqual(second_task.status, TaskStatus.DONE)
|
||||
self.assertEqual(first_task.output, second_task.output)
|
||||
self.assertEqual(first_task.score, second_task.score)
|
||||
self.assertEqual(first.pheromones[first_task.id], second.pheromones[second_task.id])
|
||||
|
||||
def test_project_is_markov_style_state_machine_not_formal_mdp(self) -> None:
|
||||
assessment = assess_markov_process_fit()
|
||||
|
||||
self.assertTrue(assessment.markov_style_state_machine)
|
||||
self.assertFalse(assessment.formal_markov_process)
|
||||
self.assertFalse(assessment.formal_markov_decision_process)
|
||||
self.assertIn("pheromone score table", assessment.sufficient_state)
|
||||
self.assertTrue(any("transition probability" in item for item in assessment.limiting_factors))
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
unittest.main()
|
||||
Reference in New Issue
Block a user