Establish agent swarm quality evidence

Define Agent and swarm-specific acceptance evidence, move the reports under docs, and make the homepage point to the current standard, live run, model I/O, and handoff evidence.

Constraint: Agent quality standards are configured from industry AI and agent risk references because there is no single accepted swarm-Agent certification standard.

Rejected: Treating py_compile or unittest as the primary quality standard | they are evidence collection tools, not the Agent quality standard itself.

Confidence: high

Scope-risk: moderate

Directive: Keep future standard reports under docs/ and keep secrets in ignored local .env files only.

Tested: git diff --cached --check; python -B -m py_compile swarm_minimal/*.py examples/*.py tests/*.py; python -B -m unittest discover -s tests; python -u -B examples/run_academic_standard_evaluation.py

Not-tested: Did not rerun the full live Azure/NewAPI S07 scenario after moving docs; previous live run 3e8e58ae4e084bc8b90cf5c46f8992f3 passed before the docs relocation.

Co-authored-by: OmX <omx@oh-my-codex.dev>
This commit is contained in:
gongzhiyong
2026-05-16 14:36:47 +08:00
co-authored by OmX
parent 111be3e435
commit 10a980b0bf
21 changed files with 1994 additions and 135 deletions
-45
View File
@@ -1,45 +0,0 @@
# Industry Standard Agnet Test Plan
There is no single universal "Agnet" certification standard. This plan maps the
minimal swarm to widely used global standards and frameworks that are relevant
to autonomous AI agents, software quality, testing, security, risk management,
and observability.
## Source Standards
| Standard / framework | What it contributes here | Local evidence |
| --- | --- | --- |
| ISO/IEC/IEEE 29119 Software Testing | Test process, test design, traceable scenarios, PASS/FAIL evidence | `STANDARD_TEST_MATRIX.md`, `examples/run_standard_scenario_acceptance.py` |
| ISO/IEC 25010 Product Quality Model | Functional suitability, reliability, security, maintainability, portability, performance efficiency, compatibility, usability | `tests/test_industry_standard_mapping.py` |
| NIST AI RMF 1.0 | Govern, Map, Measure, Manage risk lifecycle | Standard control mapping in this file |
| NIST AI 600-1 GenAI Profile | GenAI-specific risk measurement and management expectations | Secret redaction, model discovery, output policy, live checks |
| OWASP LLM Top 10 / Agentic Skills guidance | Prompt injection, excessive agency, sensitive information disclosure, supply chain, tool boundary controls | Deterministic security/policy tests |
| MITRE ATLAS | Adversarial AI threat framing | Failure and policy scenarios linked to threat classes |
| OpenTelemetry | Vendor-neutral observability expectations for traces, metrics, and logs | Event, score, artifact, and run-id evidence |
## Control Matrix
| Control ID | Standard mapping | Requirement | Test evidence |
| --- | --- | --- | --- |
| AGT-29119-TRACE | ISO/IEC/IEEE 29119 | Every scenario has ID, Given, When, Then, command, and evidence | `run_standard_scenario_acceptance.py` output |
| AGT-25010-FUNC | ISO/IEC 25010 functional suitability | Goal submission, task execution, scoring, and convergence work end to end | Unit and live tests |
| AGT-25010-REL | ISO/IEC 25010 reliability | Failed agent does not prevent successful convergence of completed tasks | `test_failed_agent_records_failed_task_and_negative_pheromone` |
| AGT-25010-SEC | ISO/IEC 25010 security | Secrets are redacted; NATS/Cosmos are not required MVP dependencies | config and policy tests |
| AGT-25010-MAINT | ISO/IEC 25010 maintainability | Test matrix and source-file traceability are explicit | docs and deterministic tests |
| AGT-NIST-GOVERN | NIST AI RMF Govern | Scope, dependency boundaries, and acceptance rules are documented | this file + matrix |
| AGT-NIST-MAP | NIST AI RMF Map | Intended use, resources, actors, and failure modes are enumerated | scenario matrix |
| AGT-NIST-MEASURE | NIST AI RMF Measure | Scores, run status, stream event deltas, artifact existence, and failures are measured | live report |
| AGT-NIST-MANAGE | NIST AI RMF Manage | Failure handling and dependency rejection have explicit stop rules | deterministic tests |
| AGT-OWASP-PI | OWASP LLM / Agentic | Prompt content cannot redefine MVP dependency boundary or request secrets | policy tests |
| AGT-OWASP-EXCESS | OWASP Agentic | Agent outputs cannot create external dependency requirements outside approved Azure resources | policy tests |
| AGT-MITRE-FAIL | MITRE ATLAS style threat modeling | Timeout/failure and misleading score scenarios are represented | failure injection and final scoring tests |
| AGT-OTEL-OBS | OpenTelemetry-style observability | Live run returns run_id, selected models, event delta, score evidence, artifact path | live integration report |
## Minimum Passing Bar
The industry-standard acceptance is PASS only when:
1. The local standard scenario matrix passes.
2. Every required control above has at least one local evidence source.
3. No control has an unhandled critical gap.
4. The final report preserves run evidence without printing secrets.
+45 -4
View File
@@ -2,6 +2,30 @@
这是一个独立的最小蜂群闭环原型,不接入 Heicode Manager 主项目,也不修改现有 API。
## 当前状态
本仓库已经补齐最小 Agent / 蜂群 Agent 标准化测试闭环。这里的标准不是普通软件开发质量标准,而是围绕 Agent 的任务理解、上下文承接、模型选择、工具边界、敏感信息保护、可观测性、handoff、信息素协作、收敛和马尔可夫式状态转移建立的 AQS / SW-AQS v1。
最新 live 标准矩阵已通过:
```json
{
"run_id": "3e8e58ae4e084bc8b90cf5c46f8992f3",
"completed_tasks": 7,
"accepted_score": 1.0,
"check_count": 12,
"failed_checks": []
}
```
主要报告入口都放在 `docs/`,完整索引见 `docs/README.md`:
- `docs/AGENT_SWARM_QUALITY_STANDARD.zh-CN.md`:Agent / 蜂群 Agent 质量标准、测试标准来源、S07 任务分配和接手机制。
- `docs/ACADEMIC_STANDARD_TEST_REPORT.zh-CN.md`:测试场景、测试过程、测试结果、算法清单和马尔可夫过程判定。
- `docs/MODEL_AGNET_IO_REPORT.zh-CN.md`:每个模型 / Agnet 的任务、输入、输出、评分和交接过程。
- `docs/STANDARD_TEST_MATRIX.md`:S01-S07 标准矩阵。
- `docs/INDUSTRY_STANDARD_AGNET_TEST_PLAN.md`:行业参考框架到本项目标准的映射。
目标是先把蜂群的四个共享资源跑通:
1. 任务池:记录待执行、执行中、完成和失败的任务。
@@ -44,11 +68,24 @@ MVP 不使用 NATS,也不使用 Cosmos DB。
## 本地运行
```bash
cd swarm-minimal
python3 -B examples/run_demo.py
python3 -B -m unittest discover -s tests
```
标准化测试入口:
```bash
python3 -B -m py_compile swarm_minimal/*.py examples/*.py tests/*.py
python3 -B -m unittest discover -s tests
python3 -u -B examples/run_academic_standard_evaluation.py
```
完整 live 标准矩阵需要本机被忽略的 `.env` 中提供 Azure PostgreSQL、Redis、Blob 和 NewAPI 测试凭据:
```bash
python3 -u -B examples/run_standard_scenario_acceptance.py
```
## Agnet / NewAPI 测试
NewAPI 连接使用 OpenAI-compatible `/v1/chat/completions` 形式。真实 key 只通过环境变量注入:
@@ -62,7 +99,7 @@ export NEWAPI_API_KEY=<newapi-api-key>
python3 -B examples/run_newapi_agnet_demo.py
```
也可以在本机创建 `swarm-minimal/.env`,demo 会自动读取;该文件已被 `.gitignore` 忽略:
也可以在本机创建项目根目录 `.env`,demo 会自动读取;该文件已被 `.gitignore` 忽略:
```bash
NEWAPI_BASE_URL=<newapi-base-url>
@@ -113,7 +150,7 @@ export SWARM_BLOB_CONTAINER=swarm-artifacts
python3 -B examples/run_azure_demo.py
```
`examples/run_azure_demo.py`、`examples/run_newapi_agnet_demo.py`、`examples/run_three_newapi_agnets.py` 和 `examples/run_full_live_test.py` 都会自动读取本地私有 `swarm-minimal/.env`。
`examples/run_azure_demo.py`、`examples/run_newapi_agnet_demo.py`、`examples/run_three_newapi_agnets.py` 和 `examples/run_full_live_test.py` 都会自动读取本地私有 `.env`;优先读取项目根目录 `.env`,没有时读取 `examples/.env`。
如果连接字符串、数据库密码、Redis key 或 Storage account key 曾经出现在聊天、日志、截图或提交记录里,应在 Azure 侧轮换后再长期使用。
@@ -121,7 +158,7 @@ python3 -B examples/run_azure_demo.py
- `swarm_minimal/core.py`:最小蜂群闭环逻辑。
- `swarm_minimal/config.py`:从环境变量读取 PostgreSQL、Redis、Blob 配置,并只输出脱敏摘要。
- `swarm_minimal/local_env.py`:读取本地私有 `.env`,不依赖第三方 dotenv 包。
- `swarm_minimal/local_env.py`:读取本地私有 `.env` 或 `examples/.env`,不依赖第三方 dotenv 包。
- `swarm_minimal/azure_store.py`:可选 Azure 后端,使用 PostgreSQL、Redis Streams / Sorted Set 和 Blob Storage。
- `swarm_minimal/azure_resources.py`:Azure 资源需求清单。
- `swarm_minimal/newapi_agnet.py`:NewAPI / OpenAI-compatible Agnet 适配器。
@@ -133,3 +170,7 @@ python3 -B examples/run_azure_demo.py
- `examples/run_full_live_test_interactive.py`:本机交互式输入密钥并运行完整 live 测试。
- `tests/test_minimal_swarm.py`:闭环和资源约束测试。
- `tests/test_newapi_agnet.py`:NewAPI Agnet 的 mock 测试。
- `tests/test_markov_process_properties.py`:马尔可夫式状态转移判定测试。
- `examples/run_academic_standard_evaluation.py`:学术化标准测试汇总入口。
- `docs/ACADEMIC_STANDARD_TEST_REPORT.zh-CN.md`:标准来源、测试过程、结果、算法和马尔可夫过程结论。
- `docs/AGENT_SWARM_QUALITY_STANDARD.zh-CN.md`:Agent / 蜂群 Agent 质量标准、S07 任务分配、模型接手机制和当前结论。
+130
View File
@@ -0,0 +1,130 @@
# 学术化标准测试报告
**对象**: `swarm-minimal` 最小蜂群原型
**日期**: 2026-05-16
**范围**: 本地确定性测试、蜂群行为验收、传统 Agnet 对比、多轮共识、马尔可夫过程适配性判断
## 1. 已实现的场景测试
| ID | 场景 | 测试目的 | 证据入口 |
| --- | --- | --- | --- |
| S01 | 静态编译 | 所有 Python 模块可编译,无语法错误 | `python3 -B -m py_compile swarm_minimal/*.py examples/*.py tests/*.py` |
| S02 | 单元回归 | 内存蜂群闭环、Azure 资源边界、配置脱敏、NewAPI mock 行为 | `python3 -B -m unittest discover -s tests` |
| S03 | 连续推理链路 | 每一步必须承接上一步 marker,断链必须失败 | `tests/test_standard_scenarios.py` |
| S04 | 依赖边界 | MVP 不把 NATS/Cosmos 当成必需依赖 | `tests/test_standard_scenarios.py` |
| S05 | 最终收敛评分 | 最终 STEP-07 输出必须优于中间步骤 | `tests/test_standard_scenarios.py` |
| S06 | 失败注入 | 失败任务要标记失败、产生负信息素,其他任务仍可收敛 | `tests/test_standard_scenarios.py` |
| B01 | 单 Agnet 故障隔离 | 一个 Agnet 崩溃不导致整体失败 | `tests/test_swarm_behavior_academic.py` |
| B02 | 群体涌现 | 多个弱局部信号通过共享状态聚合成更强整体结果 | `tests/test_swarm_behavior_academic.py` |
| B03 | 信息素间接协作 | 信息素影响任务 claim 顺序并形成正反馈 | `tests/test_swarm_behavior_academic.py` |
| B04 | handoff 连续性 | active agent、目标 agent、payload 在交接中保留 | `tests/test_swarm_behavior_academic.py` |
| C01-C04 | 传统 Agnet 对比 | 和单路线/FIFO/无共享状态/无上下文 handoff 的基线比较 | `examples/run_swarm_vs_traditional_benchmark.py` |
| M01-M03 | 马尔可夫过程适配性 | 判断当前实现是马尔可夫式状态机,但不是严格 MDP | `tests/test_markov_process_properties.py` |
## 2. 测试标准来源
本项目没有使用“某一个 Agnet 认证标准”,因为行业目前还没有统一答案。当前采用 `AGENT_SWARM_QUALITY_STANDARD.zh-CN.md` 中定义的 AQS / SW-AQS v1:以行业 Agent 风险框架为参考,配置本项目自己的 Agent 和蜂群 Agent 质量标准。
| 来源 | 本项目采用方式 |
| --- | --- |
| NIST AI RMF 1.0 | Govern / Map / Measure / Manage 的 Agent 风险管理结构 |
| NIST AI 600-1 | 生成式 AI 的幻觉、隐私、信息安全、组件集成、预部署测试和事件披露 |
| OWASP LLM Top 10 | prompt injection、敏感信息泄露、过度代理、不安全输出处理、过度信任 |
| OWASP Agentic Skills Top 10 | Agent 技能/工具执行层的权限、隔离、审计、运行时安全 |
| MITRE ATLAS | AI 攻击/误用视角下的失败、越权、诱导和异常场景 |
| OpenTelemetry | Agent 过程的 traces、metrics、logs、events 和 artifact 证据 |
| LangGraph handoff 参考 | `transfer_to_<agent>`、active-agent 路由和上下文传递语义 |
| 蜂群模式归纳标准 | 去中心化、自组织、局部感知、间接协作、涌现、鲁棒性和收敛 |
| Markov property 定义 | 给定当前状态后,未来状态不再依赖历史路径 |
`py_compile` 和 `unittest` 只作为证据采集工具,不作为最终质量标准本身。完整链接在 `swarm_minimal/academic_evaluation.py` 的 `ACADEMIC_STANDARD_SOURCES` 中维护。
## 3. 测试过程
本地确定性验收按以下顺序执行:
```bash
python3 -B -m py_compile swarm_minimal/*.py examples/*.py tests/*.py
python3 -B -m unittest discover -s tests
python3 -u -B examples/run_swarm_behavior_acceptance.py
python3 -u -B examples/run_swarm_vs_traditional_benchmark.py
python3 -u -B examples/run_consensus_convergence_acceptance.py
python3 -u -B examples/run_academic_standard_evaluation.py
```
完整标准矩阵还包含 S07 live 场景:
```bash
python3 -u -B examples/run_standard_scenario_acceptance.py
```
S07 需要本地被 git 忽略的 `.env`,其中包含 Azure PostgreSQL、Redis、Blob 和 NewAPI 测试凭据。脚本优先读取项目根目录 `.env`,没有时读取 `examples/.env`。没有可用 `.env` 时,本地确定性门禁可以通过,但完整标准化验收不能判定为通过。
## 4. 本轮实际执行结果
| 命令 | 本轮结果 | 说明 |
| --- | --- | --- |
| `python3 -B -m py_compile swarm_minimal/*.py examples/*.py tests/*.py` | PASS | 无输出,表示所有模块编译通过 |
| `python3 -B -m unittest discover -s tests` | PASS | 26 个测试通过 |
| `python3 -B -m unittest tests.test_markov_process_properties` | PASS | 3 个马尔可夫适配性测试通过 |
| `python3 -u -B examples/run_swarm_behavior_acceptance.py` | PASS | B01-B04 全部通过 |
| `python3 -u -B examples/run_swarm_vs_traditional_benchmark.py` | PASS | 蜂群总归一化分 0.9175,传统基线 0.1958,ratio 4.69 |
| `python3 -u -B examples/run_consensus_convergence_acceptance.py` | PASS | 2 轮收敛,接受 `lease_based_pg_queue` |
| `python3 -u -B examples/run_academic_standard_evaluation.py` | PASS | A01-A05 本地学术化门禁全部通过,检测到 `examples/.env` 可用于 live 测试 |
| `python3 -u -B examples/run_standard_scenario_acceptance.py` | PASS | S01-S07 全部通过;S07 完成 7 个 live 任务,12 个检查无失败 |
最新 S07 live 证据:
```json
{
"run_id": "3e8e58ae4e084bc8b90cf5c46f8992f3",
"completed_tasks": 7,
"accepted_score": 1.0,
"selected_models": [
"deepseek-v4-flash",
"claude-haiku-4-5-20251001",
"claude-sonnet-4-6"
],
"failed_checks": [],
"check_count": 12,
"artifact_path": "swarm-runs/3e8e58ae4e084bc8b90cf5c46f8992f3/result.json"
}
```
## 5. 达标判定
| 门禁 | 当前结论 |
| --- | --- |
| 本地学术化门禁 A01-A05 | 可通过 |
| 标准矩阵 S01-S06 | 可通过 |
| 标准矩阵 S07 live Azure/NewAPI | 已通过 |
| 生产级标准化验收 | 当前最小标准已满足 |
结论:当前项目已经具备可执行的标准化测试,且本地确定性蜂群行为测试与真实 Azure/NewAPI live 集成测试均已通过。该结论限定在本仓库定义的最小蜂群标准矩阵内,不等同于大规模生产压测或第三方认证。
## 6. 算法清单
| 算法 / 机制 | 位置 | 说明 |
| --- | --- | --- |
| 能力匹配任务 claim | `InMemorySwarmStore.claim_next` | Agnet 只领取能力匹配的 pending 任务 |
| 信息素排序 | `InMemorySwarmStore.claim_next` | pending 任务按 pheromone score 从高到低排序 |
| 正/负反馈更新 | `complete_task` / `fail_task` | 成功加分,失败扣分 |
| 最高分收敛 | `converge` | 在完成任务中选择最高 score 作为最终输出 |
| 多轮加权共识 | `ConsensusSwarm.run` | 按 agent weight 和 confidence 累积分数 |
| 分数蒸发 | `ConsensusSwarm._evaporate_scores` | 每轮按 evaporation 衰减历史候选分 |
| 动态模型发现 | `discover_newapi_models` | 从 NewAPI 兼容端点发现模型 |
| 去重模型选择 | `select_distinct_models` | 多 Agnet 测试时选择不同模型 |
## 7. 马尔可夫过程结论
当前实现满足“工程意义上的马尔可夫式状态机”:
- 如果把 `tasks`、`pheromones`、`shared_state`、`observations`、agent policy 和共识轮次一起视为完整当前状态,则下一步 claim、score 更新和收敛选择只依赖当前状态。
- 新增测试 `tests/test_markov_process_properties.py` 验证了等价当前任务/信息素状态下,不同历史路径会得到相同 claim 与 score 更新。
但它不满足严格数学意义的 Markov process / Markov Decision Process:
- 没有定义状态转移概率核 `P(s_next | s_current)`。
- 没有正式 action space、reward function 和策略优化目标。
- UUID、时间戳、外部 NewAPI/LLM 调用没有建模为随机变量。
- 部分输出保留历史观测作为审计证据,这属于工程可追溯性,不是形式化随机过程。
+141
View File
@@ -0,0 +1,141 @@
# Agent / 蜂群 Agent 质量标准 v1
**版本**: AQS/SW-AQS v1
**对象**: `swarm-minimal` 最小蜂群原型
**目的**: 用 Agent 行业通用风险与能力框架,加上本项目自定义蜂群 Agent 标准,评价这个系统是否像一个可控、多 Agent、可交接、可观测、可收敛的 Agent 系统。
## 1. 边界说明
这份标准不是普通软件开发质量标准。
`py_compile`、`unittest`、场景脚本只作为“证据采集工具”,不作为最终质量标准本身。最终评价对象是:
- 单个 Agent 是否能理解任务、保持上下文、使用工具、产出可验证结果。
- 多个 Agent 是否能通过共享环境、信息素、handoff 和收敛机制形成蜂群行为。
- Agent 运行过程是否能被审计、复现、拒绝越权、避免明文密钥泄露。
## 2. 行业参考来源
当前行业没有统一的“Agent 质量认证”。本项目采用“行业参考 + 本项目配置标准”的方式:
| 来源 | 本项目采用的 Agent 质量含义 |
| --- | --- |
| NIST AI RMF 1.0 | 用 Govern / Map / Measure / Manage 管理 Agent 风险、可控性和可度量性 |
| NIST AI 600-1 Generative AI Profile | 关注生成式 AI 的幻觉、数据隐私、信息安全、供应链/组件、预部署测试和事件披露 |
| OWASP LLM Top 10 | 覆盖 prompt injection、敏感信息泄露、过度代理、过度信任、不安全输出处理 |
| OWASP Agentic Skills Top 10 | 覆盖 Agent 技能/工具执行层的权限、隔离、审计、供应链和运行时安全 |
| MITRE ATLAS | 用 AI 攻击/误用视角设计失败、越权、诱导和异常场景 |
| OpenTelemetry | 用 traces、metrics、logs、events 的思想要求 Agent 过程可观测 |
| LangGraph handoff 参考 | 用 active-agent / `transfer_to_<agent>` 语义约束 Agent 交接 |
| 蜂群模式文献/文章归纳 | 用去中心化、自组织、局部感知、间接协作、涌现、鲁棒性和收敛作为蜂群标准 |
## 3. 通用 Agent 质量标准 AQS
| ID | 质量项 | 合格标准 | 当前证据 |
| --- | --- | --- | --- |
| AQS-01 任务理解 | Agent 输出必须回应分配给它的具体子任务,不泛泛回答 | S07 每步 task.input 明确当前步骤和要求 |
| AQS-02 指令遵循 | 输出必须包含指定 marker、约束、文件路径、验收命令或风险项 | `score_output()` 和 S07 12 项检查 |
| AQS-03 上下文忠实 | 除第一步外,Agent 必须引用上一阶段 marker 和 summary | S07 `step_markers_and_previous_links` |
| AQS-04 可验证输出 | 输出必须能被规则检查,不能只有自然语言主张 | S03/S05/S07 checks |
| AQS-05 模型选择可控 | 多 Agent 不写死 `NEWAPI_MODEL`,必须从模型列表发现并选择 | S07 `three_distinct_models_from_discovery` |
| AQS-06 工具/资源边界 | 不把 NATS/Cosmos 等未批准依赖变成 MVP 必需项 | S04/S07 `no_required_nats_or_cosmos` |
| AQS-07 敏感信息保护 | `.env`、密钥、连接串不进入 Git、日志、报告或模型输出 | 配置脱敏测试、报告只输出 redacted/路径 |
| AQS-08 可观测性 | run_id、task status、score、observation、artifact、stream event 可查 | S07 PostgreSQL/Redis/Blob 证据 |
| AQS-09 错误可解释 | 失败任务必须记录 error、failed observation 和负信息素 | S06/B01/C01 |
| AQS-10 交接准备度 | 输出要给下一个 Agent 留出摘要、风险和下一步 | S07 prompt/output 中强制“下一步/交接” |
| AQS-11 外部依赖真实性 | live 测试必须真实连 PostgreSQL、Redis、Blob、NewAPI | S07 live integration PASS |
| AQS-12 人类审计友好 | 最终报告必须能回答:任务、输入、输出、接手、结果、未满足项 | `MODEL_AGNET_IO_REPORT.zh-CN.md` 和本文件 |
## 4. 蜂群 Agent 质量标准 SW-AQS
| ID | 蜂群质量项 | 合格标准 | 当前证据 / 缺口 |
| --- | --- | --- | --- |
| SW-AQS-01 多角色能力 | 至少 3 个不同 Agent/模型参与,角色或步骤不同 | S07 3 个模型、7 个连续步骤 |
| SW-AQS-02 共享任务池 | 任务进入统一 task pool,Agent 从池中 claim | PostgreSQL `swarm_tasks` |
| SW-AQS-03 局部感知 | Agent 输入包含当前任务、上一阶段 summary 或 shared_state keys | S07 user prompt shape |
| SW-AQS-04 自主 claim | Agent 根据 capability/任务状态领取任务 | `claim_next()`;live 路径仍是 coordinator 顺序驱动,部分满足 |
| SW-AQS-05 信息素/分数 | 每个任务完成后写入正分,失败写入负分 | `swarm_pheromones` + Redis sorted set |
| SW-AQS-06 间接协作 | 后续 Agent 通过 shared_state/summary/pheromone 感知前序结果 | S07 chain summary 和 cursor |
| SW-AQS-07 handoff 连续性 | 交接必须有 from->to/previous->current edge,并保留 payload | B04 与 S07 `chain_edge` |
| SW-AQS-08 广播/事件 | claim、done、converged 等事件进入 stream/outbox | S07 Redis Stream 事件数检查 |
| SW-AQS-09 收敛条件 | 不能只说“跑完”;必须检查任务、分数、状态、artifact、事件和内容质量 | S07 12 项 checks |
| SW-AQS-10 鲁棒性 | 单个 Agent 失败不应吞掉整体状态,失败要可观测 | B01/C01/S06 |
| SW-AQS-11 涌现性 | 群体聚合结果能超过单个局部强信号 | B02/C02 |
| SW-AQS-12 传统基线对比 | 必须和单 Agent/FIFO/无共享状态基线比较 | C01-C04 |
| SW-AQS-13 live 外部闭环 | 不能只 mock,至少一次真实资源闭环 | S07 PASS |
| SW-AQS-14 马尔可夫式状态 | 给定完整当前状态,下一步工程转移由当前状态决定 | M01-M03,非严格 MDP |
| SW-AQS-15 扩缩容与并发 | 3/5/7 Agent 并发自主 claim 下仍稳定 | 仍待补充 |
## 5. S07 live 测试任务定义
总任务:
```text
连续性长推理场景:为 swarm-minimal 设计可恢复的大规模代码任务推理链
```
目标不是让模型自由聊天,而是让 7 个连续 Agent 阶段接力完成一个工程推理链:
| 步骤 | 分配任务 | 主要验证点 |
| --- | --- | --- |
| STEP-01 | 界定问题和不可变约束 | 建立目标、输入输出、不变量和 Azure 资源边界 |
| STEP-02 | 建立依赖图和状态模型 | 承接 STEP-01,给出任务依赖图、共享状态、租约、状态转移 |
| STEP-03 | 设计连续调度算法 | 承接 STEP-02,设计调度、信息素更新、收敛算法和复杂度 |
| STEP-04 | 构造反例和失败场景 | 承接 STEP-03,构造慢模型、重复任务、状态倒退、分数误导 |
| STEP-05 | 修正算法和恢复策略 | 承接 STEP-04,加入幂等、重试、死信、outbox、重连恢复 |
| STEP-06 | 落到文件级实现计划 | 承接 STEP-05,引用真实文件和测试文件 |
| STEP-07 | 最终收敛和验收判定 | 承接 STEP-06,给出命令、指标、失败判定和上线前结论 |
模型分配来自动态发现,不写死 `NEWAPI_MODEL`:
| 步骤 | Agnet | primary / used model | 交接边 |
| --- | --- | --- | --- |
| STEP-01 | `continuous-agnet-1` | `deepseek-v4-flash` | `START->STEP-01` |
| STEP-02 | `continuous-agnet-2` | `claude-haiku-4-5-20251001` | `STEP-01->STEP-02` |
| STEP-03 | `continuous-agnet-3` | `claude-sonnet-4-6` | `STEP-02->STEP-03` |
| STEP-04 | `continuous-agnet-4` | `deepseek-v4-flash` | `STEP-03->STEP-04` |
| STEP-05 | `continuous-agnet-5` | `claude-haiku-4-5-20251001` | `STEP-04->STEP-05` |
| STEP-06 | `continuous-agnet-6` | `claude-sonnet-4-6` | `STEP-05->STEP-06` |
| STEP-07 | `continuous-agnet-7` | `deepseek-v4-flash` | `STEP-06->STEP-07` |
## 6. 模型输入输出与接手机制
完整输入输出见 `MODEL_AGNET_IO_REPORT.zh-CN.md`。核心机制如下:
1. wrapper 给当前 Agnet 的 user prompt 注入:
- `Previous marker`
- `Previous summary`
- `Task kind`
- 完整 `Task input`
2. 当前 Agnet 输出必须带:
- `chain_edge=<previous>-><current>`
- `primary_model`
- `used_model`
- `model_selection=discovered_models_not_NEWAPI_MODEL`
3. 当前 Agnet 完成后,系统写入:
- `chain:{run_id}:{STEP}:summary`
- `chain:{run_id}:cursor`
- `chain:{run_id}:edge:<previous>-><current>=done`
4. 下一个 Agnet 读取上一步 summary 和 marker 后继续执行。
因此“接手”不是人工解释,也不是模型凭空猜测,而是通过 PostgreSQL shared_state 中的 summary、cursor、edge 字段完成。
## 7. 本轮判定
最新标准矩阵 run:
```json
{
"run_id": "3e8e58ae4e084bc8b90cf5c46f8992f3",
"completed_tasks": 7,
"accepted_score": 1.0,
"check_count": 12,
"failed_checks": []
}
```
结论:
- 按 AQS 通用 Agent 标准:当前最小测试通过。
- 按 SW-AQS 蜂群 Agent 标准:关键闭环通过,但“并发自主 claim、广播驱动决策、3/5/7 扩缩容压测”仍是下一阶段。
- 所以它满足“最小蜂群 Agent 质量标准 v1”,还不等于完整生产级蜂群平台认证。
+50
View File
@@ -0,0 +1,50 @@
# Agent 行业参考标准映射
当前行业没有统一的 “Agnet/Agent 质量认证标准”。本项目不再用普通软件产品质量标准作为主标准,而是采用:
- `AGENT_SWARM_QUALITY_STANDARD.zh-CN.md`:主标准,定义通用 Agent 质量标准 AQS 和蜂群 Agent 质量标准 SW-AQS。
- `ACADEMIC_STANDARD_TEST_REPORT.zh-CN.md`:执行报告,记录本轮测试过程、结果、算法和马尔可夫过程判定。
- `MODEL_AGNET_IO_REPORT.zh-CN.md`:输入输出报告,记录模型任务、prompt 结构、实际输出和接手机制。
## 行业参考框架
| 来源 | 在本项目中的用途 |
| --- | --- |
| NIST AI RMF 1.0 | 管理 Agent 风险、可控性、可度量性和治理闭环 |
| NIST AI 600-1 | 处理生成式 AI 的幻觉、隐私、信息安全、组件集成和预部署测试 |
| OWASP LLM Top 10 | 覆盖 prompt injection、敏感信息泄露、过度代理、不安全输出处理、过度信任 |
| OWASP Agentic Skills Top 10 | 覆盖 Agent 工具/技能执行层的权限、隔离、审计和运行时安全 |
| MITRE ATLAS | 从对抗性 AI 和误用角度设计失败、越权、诱导和异常场景 |
| OpenTelemetry | 约束 run_id、task、event、metric、log、artifact 等可观测证据 |
| LangGraph handoff | 参考 active-agent 和 `transfer_to_<agent>` 式交接语义 |
| 蜂群模式归纳标准 | 约束去中心化、自组织、局部感知、信息素、涌现、鲁棒性和收敛 |
## 最小通过条件
本项目当前最小通过条件不是“代码质量通过”,而是:
1. AQS 通用 Agent 标准中,任务理解、上下文、模型选择、资源边界、敏感信息、可观测性、错误可解释、交接准备度均有证据。
2. SW-AQS 蜂群 Agent 标准中,任务池、共享状态、信息素、handoff、事件流、收敛、鲁棒性、涌现性、传统基线对比和 live 外部闭环均有证据。
3. 模型输入输出能被审计:知道每个 Agent 被分配了什么任务、收到了什么输入、输出了什么、下一个 Agent 如何接手。
4. live 测试真实连接 Azure PostgreSQL、Redis、Blob 和 NewAPI,且不输出任何真实密钥。
最新一次标准矩阵已经通过:
```json
{
"run_id": "3e8e58ae4e084bc8b90cf5c46f8992f3",
"completed_tasks": 7,
"accepted_score": 1.0,
"check_count": 12,
"failed_checks": []
}
```
## 保留缺口
当前满足“最小蜂群 Agent 质量标准 v1”,但还不是完整生产级蜂群平台认证。仍需补:
- 并发 worker 自主 claim,而不是 coordinator 顺序驱动。
- Redis Stream 事件被其他 Agent 消费并改变决策。
- 3/5/7 Agent 扩缩容和稳定性压测。
- 收敛速度、Agent 利用率、响应时间、信息素分布等统一 metrics。
File diff suppressed because it is too large Load Diff
+15
View File
@@ -0,0 +1,15 @@
# 文档索引
## Agent / 蜂群 Agent 测试
- `AGENT_SWARM_QUALITY_STANDARD.zh-CN.md`:AQS / SW-AQS v1 主标准,说明 Agent 和蜂群 Agent 的质量项、S07 任务分配、模型交接和当前结论。
- `ACADEMIC_STANDARD_TEST_REPORT.zh-CN.md`:测试场景、测试过程、测试结果、算法清单和马尔可夫过程判定。
- `MODEL_AGNET_IO_REPORT.zh-CN.md`:从 live run 导出的模型 / Agnet 任务、输入、输出、评分和交接过程。
- `STANDARD_TEST_MATRIX.md`:S01-S07 标准化测试矩阵。
- `INDUSTRY_STANDARD_AGNET_TEST_PLAN.md`:行业参考框架到本项目 Agent 质量标准的映射。
## 交付计划
- `01-ten-day-delivery-plan.md`:10 天交付计划。
- `02-person-task-interface-matrix.md`:人员、任务和接口矩阵。
- `03-minimal-validation-and-acceptance.md`:最小验证和验收流程。
@@ -11,6 +11,7 @@ result means more than "one long run completed".
| Static | Python syntax and importability | `./.venv/bin/python -B -m py_compile swarm_minimal/*.py examples/*.py tests/*.py` |
| Unit | Existing closed-loop and NewAPI mock behavior | `./.venv/bin/python -B -m unittest discover -s tests` |
| Deterministic scenarios | No-network Given/When/Then cases for continuity, policy, final scoring, and failure injection | `./.venv/bin/python -B -m unittest tests.test_standard_scenarios` |
| Academic Markov fit | No-network checks for Markov-style transition behavior and non-MDP limitations | `./.venv/bin/python -B -m unittest tests.test_markov_process_properties` |
| Live integration | Real NewAPI + Azure PostgreSQL/Redis/Blob continuous reasoning chain | `./.venv/bin/python -u -B examples/run_continuous_reasoning_acceptance.py` |
## Required Scenarios
@@ -24,6 +25,9 @@ result means more than "one long run completed".
| S05 | Final convergence scoring | STEP-06 and STEP-07 both produce rich outputs | Score both outputs | STEP-07 is strictly selected as final convergence |
| S06 | Failure injection | One deterministic worker raises | Coordinator runs to convergence | Failed task is marked failed, pheromone is negative, completed task still converges |
| S07 | Live resource closure | Real Azure/NewAPI env is loaded | Run seven-step continuous reasoning | PostgreSQL, Redis, Blob, stream events, model discovery, and final artifact all pass |
| M01 | Markov-style claim transition | Same current tasks and pheromone values but different prior paths | Claim next task | The same high-pheromone task is claimed |
| M02 | Markov-style score transition | Same current task, agent, output, and score but different prior paths | Complete task | Task status, output, score, and pheromone update match |
| M03 | Formal MDP limitation | Project metadata is evaluated | Assess Markov fit | Prototype is classified as Markov-style state machine, not a formal Markov process or MDP |
## Stop Rule
+41 -4
View File
@@ -10,24 +10,26 @@ sys.path.insert(0, str(ROOT))
from swarm_minimal.azure_store import PostgresRedisBlobSwarmStore
from swarm_minimal.config import SwarmConfig
from swarm_minimal.local_env import load_env_file
from swarm_minimal.local_env import load_project_env
RUN_IDS = [
"3e8e58ae4e084bc8b90cf5c46f8992f3",
"78f189ccd1924ed0a4fb0a0a447ad449",
"623b5f6e5cc24cc7967fd9577f9c224b",
"42688e7b6245466dab8398dfe4790456",
]
OUTPUT_PATH = ROOT / "MODEL_AGNET_IO_REPORT.zh-CN.md"
OUTPUT_PATH = ROOT / "docs" / "MODEL_AGNET_IO_REPORT.zh-CN.md"
def main() -> None:
load_env_file(ROOT / ".env")
load_project_env(ROOT)
store = PostgresRedisBlobSwarmStore(SwarmConfig.from_env())
try:
store.ensure_schema()
report = build_report(store)
OUTPUT_PATH.parent.mkdir(exist_ok=True)
OUTPUT_PATH.write_text(report, encoding="utf-8")
print(str(OUTPUT_PATH))
finally:
@@ -50,7 +52,7 @@ def build_report(store: PostgresRedisBlobSwarmStore) -> str:
task_ids = [item["task_id"] for item in convergence["observations"]]
tasks = fetch_tasks(store, task_ids)
sections.extend(render_run(convergence, tasks))
return "\n".join(sections).rstrip() + "\n"
return "\n".join(line.rstrip() for line in sections).rstrip() + "\n"
def render_run(convergence: dict[str, object], tasks: dict[str, dict[str, object]]) -> list[str]:
@@ -94,6 +96,10 @@ def render_run(convergence: dict[str, object], tasks: dict[str, dict[str, object
user_prompt_shape_for_goal(goal),
"```",
"",
"#### 接手 / 交接机制",
"",
handoff_description(goal, run_id, task, output),
"",
"#### 本次任务输入 task.input",
"",
"```text",
@@ -111,6 +117,37 @@ def render_run(convergence: dict[str, object], tasks: dict[str, dict[str, object
return lines
def handoff_description(goal: str, run_id: str, task: dict[str, object], output: str) -> str:
if goal.startswith("连续性长推理场景"):
match = re.search(r"chain_edge=([^;\\n]+)", output)
edge = match.group(1).strip() if match else "<missing>"
marker = edge.split("->")[-1] if "->" in edge else str(task.get("kind") or "<unknown>")
return "\n".join(
[
f"- 当前输出前缀记录 `chain_edge={edge}`,证明本 Agnet 承接了上一阶段。",
"- 调用前,wrapper 会把 `Previous marker` 和 `Previous summary` 放入 user prompt。",
f"- 执行后,wrapper 把输出摘要写入 `chain:{run_id}:{marker}:summary`。",
f"- 同时推进 `chain:{run_id}:cursor`,并写入 `chain:{run_id}:edge:{edge}=done`。",
"- 下一个 Agnet 读取这个 summary 和 edge 后继续执行,所以接手不是靠口头描述,而是靠共享状态字段完成。",
]
)
if goal.startswith("真实全面场景"):
return "\n".join(
[
"- 每个 Agnet 从 PostgreSQL 任务池 claim 自己的子任务。",
"- 共享状态键列表作为上下文输入,让后续任务能看到已有 run/task/agent 状态。",
"- 完成后写回 task output、score、observation 和 Redis Stream 事件,供收敛阶段读取。",
]
)
return "\n".join(
[
"- Agnet 从共享任务池 claim 与自身 capability 匹配的任务。",
"- 完成后写回 output、score、observation、heartbeat 和 pheromone 分数。",
"- 后续 Agnet 通过 shared state、task status 和 pheromone score 感知前序执行结果。",
]
)
def system_prompt_for_goal(goal: str) -> str:
if goal.startswith("full live test"):
return (
@@ -0,0 +1,174 @@
from __future__ import annotations
from pathlib import Path
import json
import subprocess
import sys
ROOT = Path(__file__).resolve().parents[1]
sys.path.insert(0, str(ROOT))
from swarm_minimal.academic_evaluation import ( # noqa: E402
ACADEMIC_STANDARD_SOURCES,
ALGORITHMS_USED,
assess_markov_process_fit,
)
from swarm_minimal.local_env import find_project_env # noqa: E402
CHECKS = [
{
"id": "A01",
"name": "static_compile",
"layer": "static",
"command": [
sys.executable,
"-B",
"-m",
"py_compile",
*sorted(str(path.relative_to(ROOT)) for path in ROOT.glob("swarm_minimal/*.py")),
*sorted(str(path.relative_to(ROOT)) for path in ROOT.glob("examples/*.py")),
*sorted(str(path.relative_to(ROOT)) for path in ROOT.glob("tests/*.py")),
],
},
{
"id": "A02",
"name": "unit_and_deterministic_scenarios",
"layer": "unit-scenario",
"command": [sys.executable, "-B", "-m", "unittest", "discover", "-s", "tests"],
},
{
"id": "A03",
"name": "swarm_behavior_acceptance",
"layer": "behavior",
"command": [sys.executable, "-u", "-B", "examples/run_swarm_behavior_acceptance.py"],
"parse_json": True,
},
{
"id": "A04",
"name": "swarm_vs_traditional_benchmark",
"layer": "benchmark",
"command": [sys.executable, "-u", "-B", "examples/run_swarm_vs_traditional_benchmark.py"],
"parse_json": True,
},
{
"id": "A05",
"name": "consensus_convergence_acceptance",
"layer": "consensus",
"command": [sys.executable, "-u", "-B", "examples/run_consensus_convergence_acceptance.py"],
"parse_json": True,
},
]
def main() -> None:
results = [run_check(check) for check in CHECKS]
env_path = find_project_env(ROOT)
live_ready = env_path is not None
markov = assess_markov_process_fit()
report = {
"standard": "academic-standard-evaluation-v1",
"status": "PASS" if all(item["passed"] for item in results) else "FAIL",
"scope": {
"deterministic_local": True,
"live_azure_newapi": "ready" if live_ready else "blocked_missing_.env",
"env_path": str(env_path.relative_to(ROOT)) if env_path else None,
"live_note": (
"S07 live integration requires Azure PostgreSQL, Redis, Blob and NewAPI credentials in an ignored .env."
),
},
"standards": ACADEMIC_STANDARD_SOURCES,
"checks": results,
"algorithms_used": ALGORITHMS_USED,
"markov_process_assessment": {
"markov_style_state_machine": markov.markov_style_state_machine,
"formal_markov_process": markov.formal_markov_process,
"formal_markov_decision_process": markov.formal_markov_decision_process,
"sufficient_state": markov.sufficient_state,
"limiting_factors": markov.limiting_factors,
"conclusion": markov.conclusion,
},
"pass_condition": {
"local_academic_gate": "all A01-A05 checks pass",
"full_standard_gate": "local_academic_gate plus S07 live Azure/NewAPI scenario",
},
}
print(json.dumps(report, ensure_ascii=False, indent=2))
if report["status"] != "PASS":
raise SystemExit(1)
def run_check(check: dict[str, object]) -> dict[str, object]:
command = check["command"]
assert isinstance(command, list)
completed = subprocess.run(
command,
cwd=ROOT,
text=True,
capture_output=True,
timeout=900,
)
parsed = parse_last_json(completed.stdout) if check.get("parse_json") else None
evidence: dict[str, object] = {"tail": summarize_process_output(completed.stdout, completed.stderr)}
if parsed:
evidence = summarize_json(parsed)
return {
"id": check["id"],
"name": check["name"],
"layer": check["layer"],
"command": " ".join(command),
"passed": completed.returncode == 0,
"evidence": evidence,
}
def summarize_process_output(stdout: str, stderr: str) -> str:
combined = "\n".join(part.strip() for part in [stdout, stderr] if part.strip())
return combined[-1200:] if combined else "<no output>"
def parse_last_json(text: str) -> dict[str, object] | None:
stripped = text.strip()
if not stripped:
return None
decoder = json.JSONDecoder()
last = None
index = 0
while index < len(stripped):
brace = stripped.find("{", index)
if brace == -1:
break
try:
value, end = decoder.raw_decode(stripped[brace:])
except json.JSONDecodeError:
index = brace + 1
continue
if isinstance(value, dict):
last = value
index = brace + end
return last
def summarize_json(value: dict[str, object]) -> dict[str, object]:
summary: dict[str, object] = {"status": value.get("status")}
if "standard" in value:
summary["standard"] = value["standard"]
if "overall_normalized_score" in value:
summary["overall_normalized_score"] = value["overall_normalized_score"]
if "accepted_candidate" in value:
summary["accepted_candidate"] = value["accepted_candidate"]
if "rounds" in value and isinstance(value["rounds"], list):
summary["round_count"] = len(value["rounds"])
if "scenarios" in value and isinstance(value["scenarios"], list):
summary["scenario_count"] = len(value["scenarios"])
summary["failed_scenarios"] = [
item.get("id")
for item in value["scenarios"]
if isinstance(item, dict) and not item.get("passed", False)
]
return summary
if __name__ == "__main__":
main()
+2 -2
View File
@@ -9,11 +9,11 @@ sys.path.insert(0, str(ROOT))
from swarm_minimal.azure_store import PostgresRedisBlobSwarmStore
from swarm_minimal.config import SwarmConfig
from swarm_minimal.core import SwarmCoordinator, default_agents
from swarm_minimal.local_env import load_env_file
from swarm_minimal.local_env import load_project_env
def main() -> None:
load_env_file(ROOT / ".env")
load_project_env(ROOT)
config = SwarmConfig.from_env()
print(json.dumps(config.redacted_summary(), ensure_ascii=False, indent=2))
@@ -10,7 +10,7 @@ sys.path.insert(0, str(ROOT))
from swarm_minimal.azure_store import PostgresRedisBlobSwarmStore
from swarm_minimal.config import SwarmConfig
from swarm_minimal.core import Agent, SwarmCoordinator, Task
from swarm_minimal.local_env import load_env_file
from swarm_minimal.local_env import load_project_env
from swarm_minimal.newapi_agnet import (
NewApiAgnet,
NewApiChannelConfig,
@@ -103,7 +103,7 @@ ACCEPTANCE_CRITERIA = [
def main() -> None:
load_env_file(ROOT / ".env")
load_project_env(ROOT)
azure_config = SwarmConfig.from_env()
newapi_config = NewApiChannelConfig.from_env()
if newapi_config.timeout_seconds < 360:
+2 -2
View File
@@ -9,7 +9,7 @@ sys.path.insert(0, str(ROOT))
from swarm_minimal.azure_store import PostgresRedisBlobSwarmStore
from swarm_minimal.config import SwarmConfig
from swarm_minimal.core import SwarmCoordinator, Task
from swarm_minimal.local_env import load_env_file
from swarm_minimal.local_env import load_project_env
from swarm_minimal.newapi_agnet import (
NewApiChannelConfig,
build_model_test_agnets,
@@ -19,7 +19,7 @@ from swarm_minimal.newapi_agnet import (
def main() -> None:
load_env_file(ROOT / ".env")
load_project_env(ROOT)
azure_config = SwarmConfig.from_env()
newapi_config = NewApiChannelConfig.from_env()
print("azure_config:")
+2 -2
View File
@@ -10,7 +10,7 @@ sys.path.insert(0, str(ROOT))
from swarm_minimal.azure_store import PostgresRedisBlobSwarmStore
from swarm_minimal.config import SwarmConfig
from swarm_minimal.core import Agent, SwarmCoordinator, Task
from swarm_minimal.local_env import load_env_file
from swarm_minimal.local_env import load_project_env
from swarm_minimal.newapi_agnet import (
NewApiAgnet,
NewApiChannelConfig,
@@ -36,7 +36,7 @@ Return a compact JSON-like answer. Do not include secrets.
def main() -> None:
load_env_file(ROOT / ".env")
load_project_env(ROOT)
azure_config = SwarmConfig.from_env()
newapi_config = NewApiChannelConfig.from_env()
if newapi_config.timeout_seconds < 120:
+2 -2
View File
@@ -10,7 +10,7 @@ sys.path.insert(0, str(ROOT))
from swarm_minimal.azure_store import PostgresRedisBlobSwarmStore
from swarm_minimal.config import SwarmConfig
from swarm_minimal.core import Agent, SwarmCoordinator, Task
from swarm_minimal.local_env import load_env_file
from swarm_minimal.local_env import load_project_env
from swarm_minimal.newapi_agnet import (
NewApiAgnet,
NewApiChannelConfig,
@@ -94,7 +94,7 @@ ACCEPTANCE_CRITERIA = [
def main() -> None:
load_env_file(ROOT / ".env")
load_project_env(ROOT)
azure_config = SwarmConfig.from_env()
newapi_config = NewApiChannelConfig.from_env()
if newapi_config.timeout_seconds < 360:
+2 -2
View File
@@ -7,7 +7,7 @@ ROOT = Path(__file__).resolve().parents[1]
sys.path.insert(0, str(ROOT))
from swarm_minimal.core import InMemorySwarmStore, SwarmCoordinator, default_agents
from swarm_minimal.local_env import load_env_file
from swarm_minimal.local_env import load_project_env
from swarm_minimal.newapi_agnet import (
NewApiAgnet,
NewApiChannelConfig,
@@ -17,7 +17,7 @@ from swarm_minimal.newapi_agnet import (
def main() -> None:
load_env_file(ROOT / ".env")
load_project_env(ROOT)
config = NewApiChannelConfig.from_env()
if not config.model:
model = select_distinct_models(discover_newapi_models(config), count=1)[0]
+2 -2
View File
@@ -10,7 +10,7 @@ sys.path.insert(0, str(ROOT))
from swarm_minimal.azure_store import PostgresRedisBlobSwarmStore
from swarm_minimal.config import SwarmConfig
from swarm_minimal.core import Agent, SwarmCoordinator, Task
from swarm_minimal.local_env import load_env_file
from swarm_minimal.local_env import load_project_env
from swarm_minimal.newapi_agnet import (
NewApiAgnet,
NewApiChannelConfig,
@@ -53,7 +53,7 @@ ACCEPTANCE_CRITERIA = [
def main() -> None:
load_env_file(ROOT / ".env")
load_project_env(ROOT)
azure_config = SwarmConfig.from_env()
newapi_config = NewApiChannelConfig.from_env()
if newapi_config.timeout_seconds < 120:
+2 -2
View File
@@ -7,7 +7,7 @@ ROOT = Path(__file__).resolve().parents[1]
sys.path.insert(0, str(ROOT))
from swarm_minimal.core import InMemorySwarmStore, SwarmCoordinator, Task
from swarm_minimal.local_env import load_env_file
from swarm_minimal.local_env import load_project_env
from swarm_minimal.newapi_agnet import (
NewApiChannelConfig,
build_model_test_agnets,
@@ -17,7 +17,7 @@ from swarm_minimal.newapi_agnet import (
def main() -> None:
load_env_file(ROOT / ".env")
load_project_env(ROOT)
config = NewApiChannelConfig.from_env()
print(json.dumps(config.redacted_summary(), ensure_ascii=False, indent=2))
+132
View File
@@ -0,0 +1,132 @@
"""Academic-style evaluation metadata for the minimal swarm prototype."""
from __future__ import annotations
from dataclasses import dataclass
@dataclass(frozen=True)
class MarkovProcessAssessment:
"""Result of evaluating whether the prototype satisfies Markov assumptions."""
markov_style_state_machine: bool
formal_markov_process: bool
formal_markov_decision_process: bool
sufficient_state: tuple[str, ...]
limiting_factors: tuple[str, ...]
conclusion: str
ACADEMIC_STANDARD_SOURCES = (
{
"id": "NIST-AI-RMF",
"name": "NIST AI Risk Management Framework 1.0",
"use": "govern, map, measure and manage risk framing for autonomous AI behavior",
"url": "https://www.nist.gov/itl/ai-risk-management-framework",
},
{
"id": "NIST-AI-600-1",
"name": "NIST AI RMF Generative AI Profile",
"use": "generative-AI risks such as confabulation, privacy, information security and component integration",
"url": "https://doi.org/10.6028/NIST.AI.600-1",
},
{
"id": "OWASP-LLM",
"name": "OWASP Top 10 for Large Language Model Applications",
"use": "sensitive information disclosure, excessive agency and tool-boundary checks",
"url": "https://owasp.org/www-project-top-10-for-large-language-model-applications/",
},
{
"id": "OWASP-AST10",
"name": "OWASP Agentic Skills Top 10",
"use": "agentic skill risk checks for autonomous tools and delegated execution boundaries",
"url": "https://owasp.org/www-project-agentic-skills-top-10/",
},
{
"id": "MITRE-ATLAS",
"name": "MITRE ATLAS",
"use": "adversarial-AI and agent misuse framing for failure, abuse and recovery scenarios",
"url": "https://atlas.mitre.org/",
},
{
"id": "OTEL",
"name": "OpenTelemetry documentation",
"use": "observable traces, metrics, logs and event evidence expectations",
"url": "https://opentelemetry.io/docs/",
},
{
"id": "LANGGRAPH-HANDOFF",
"name": "LangGraph handoff reference",
"use": "active-agent handoff and transfer_to_<agent> continuity reference",
"url": "https://reference.langchain.com/python/langgraph-swarm/handoff/create_handoff_tool",
},
)
ALGORITHMS_USED = (
{
"name": "capability-based task claiming",
"location": "swarm_minimal.core.InMemorySwarmStore.claim_next",
"description": "agents claim pending tasks matching their capability; ties are ordered by pheromone score",
},
{
"name": "pheromone / score reinforcement",
"location": "swarm_minimal.core.InMemorySwarmStore.complete_task and fail_task",
"description": "successful task scores add positive feedback; failed tasks receive negative feedback",
},
{
"name": "winner-take-highest-score convergence",
"location": "swarm_minimal.core.InMemorySwarmStore.converge",
"description": "the highest-scoring completed task becomes the accepted result",
},
{
"name": "weighted multi-round consensus",
"location": "swarm_minimal.core.ConsensusSwarm.run",
"description": "role-weighted votes accumulate until leader share and margin thresholds are reached",
},
{
"name": "score evaporation",
"location": "swarm_minimal.core.ConsensusSwarm._evaporate_scores",
"description": "candidate scores decay between rounds before new evidence is added",
},
{
"name": "distinct model discovery and selection",
"location": "swarm_minimal.newapi_agnet.discover_newapi_models and select_distinct_models",
"description": "NewAPI models are discovered from compatible endpoints and de-duplicated for multi-agent tests",
},
)
def assess_markov_process_fit() -> MarkovProcessAssessment:
"""Classify the prototype against Markov-process requirements.
The local swarm can be interpreted as a Markov-style state machine if the
complete environment state is treated as the state variable. It is not a
formal Markov process or MDP because the implementation does not define a
transition probability kernel, action/reward tuple, or stochastic model for
external LLM/API behavior.
"""
return MarkovProcessAssessment(
markov_style_state_machine=True,
formal_markov_process=False,
formal_markov_decision_process=False,
sufficient_state=(
"tasks with status, owner, output, score and error",
"pheromone score table",
"shared_state key-value environment",
"observations already emitted",
"agent policy functions and current round index for consensus",
),
limiting_factors=(
"no transition probability kernel P(s_next | s_current)",
"no formal action space, reward function or policy optimization objective",
"uuid/time and external NewAPI/LLM calls are not modeled as stochastic variables",
"some acceptance outputs deliberately preserve history as audit evidence",
),
conclusion=(
"满足工程意义上的马尔可夫式状态转移:给定完整当前状态和 agent policy,"
"下一步 claim、score 更新和收敛选择由当前状态决定。"
"但不满足严格数学意义的 Markov process / MDP 定义。"
),
)
+22 -2
View File
@@ -1,7 +1,9 @@
"""Load local environment files without adding a dependency.
The intended file is ``.env`` under ``swarm-minimal/``. It is ignored by git and
should never be committed.
The preferred file is ``.env`` under the project root. For local test runs,
``examples/.env`` is also accepted so users can keep live-test credentials next
to the example entrypoints. Both paths are ignored by git and should never be
committed.
"""
from __future__ import annotations
@@ -10,6 +12,24 @@ from pathlib import Path
import os
def find_project_env(root: Path) -> Path | None:
"""Return the first supported private env file path for a project."""
for path in (root / ".env", root / "examples" / ".env"):
if path.exists():
return path
return None
def load_project_env(root: Path, *, override: bool = False) -> int:
"""Load the supported project env file if one exists."""
path = find_project_env(root)
if path is None:
return 0
return load_env_file(path, override=override)
def load_env_file(path: Path, *, override: bool = False) -> int:
"""Load KEY=VALUE pairs from a local env file.
+72
View File
@@ -0,0 +1,72 @@
import unittest
from swarm_minimal.academic_evaluation import assess_markov_process_fit
from swarm_minimal.core import Agent, InMemorySwarmStore, Observation, Task, TaskStatus
def build_equivalent_claim_store(history_label: str) -> InMemorySwarmStore:
store = InMemorySwarmStore()
low = Task(kind="probe", input="low", id="task-low")
high = Task(kind="probe", input="high", id="task-high")
store.add_task(low)
store.add_task(high)
store.pheromones[low.id] = 0.2
store.pheromones[high.id] = 0.8
store.shared_state["irrelevant_history_label"] = history_label
store.observations.append(
Observation(
task_id=f"past-{history_label}",
agent_id="past-agent",
signal="past:done",
score_delta=0.1,
)
)
return store
class MarkovProcessPropertyTest(unittest.TestCase):
def test_claim_transition_uses_current_task_pheromone_state_not_past_path(self) -> None:
agent = Agent(id="probe-agent", capability="probe", run=lambda task, _: ("ok", 0.1))
first = build_equivalent_claim_store("path-a")
second = build_equivalent_claim_store("path-b")
first_claim = first.claim_next(agent)
second_claim = second.claim_next(agent)
self.assertIsNotNone(first_claim)
self.assertIsNotNone(second_claim)
self.assertEqual(first_claim.id, "task-high")
self.assertEqual(second_claim.id, "task-high")
self.assertEqual(first.shared_state["task:task-high:claimed_by"], "probe-agent")
self.assertEqual(second.shared_state["task:task-high:claimed_by"], "probe-agent")
def test_score_update_depends_on_current_task_agent_and_output(self) -> None:
agent = Agent(id="probe-agent", capability="probe", run=lambda task, _: ("ok", 0.4))
first = build_equivalent_claim_store("path-a")
second = build_equivalent_claim_store("path-b")
first_task = first.claim_next(agent)
second_task = second.claim_next(agent)
assert first_task is not None
assert second_task is not None
first.complete_task(first_task, agent, "same output", 0.4)
second.complete_task(second_task, agent, "same output", 0.4)
self.assertEqual(first_task.status, TaskStatus.DONE)
self.assertEqual(second_task.status, TaskStatus.DONE)
self.assertEqual(first_task.output, second_task.output)
self.assertEqual(first_task.score, second_task.score)
self.assertEqual(first.pheromones[first_task.id], second.pheromones[second_task.id])
def test_project_is_markov_style_state_machine_not_formal_mdp(self) -> None:
assessment = assess_markov_process_fit()
self.assertTrue(assessment.markov_style_state_machine)
self.assertFalse(assessment.formal_markov_process)
self.assertFalse(assessment.formal_markov_decision_process)
self.assertIn("pheromone score table", assessment.sufficient_state)
self.assertTrue(any("transition probability" in item for item in assessment.limiting_factors))
if __name__ == "__main__":
unittest.main()