taijibaga/fengqun
9
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Add report audit scenario

Browse Source 
Extend the Agent standard matrix with a report-audit scenario so model input, output, handoff, and secret-safety evidence are tested instead of remaining narrative-only.

Constraint: The user requested another test pass and expanded Agent/swarm testing scenarios under docs/.

Rejected: Treating the model I/O report as untested documentation | it would leave the handoff and input/output evidence unguarded.

Confidence: high

Scope-risk: moderate

Directive: Keep model I/O reports under docs/ and redact secret-shaped values during export.

Tested: .venv/bin/python -u -B examples/run_standard_scenario_acceptance.py; .venv/bin/python -B -m unittest discover -s tests; .venv/bin/python -B -m py_compile swarm_minimal/*.py examples/*.py tests/*.py; .venv/bin/python -u -B examples/run_academic_standard_evaluation.py; git diff --check; docs secret-pattern scan.

Not-tested: Large-scale concurrent 3/5/7 worker load and external browser rendering were not run.

Co-authored-by: OmX <omx@oh-my-codex.dev>
This commit is contained in:
gongzhiyong
2026-05-16 15:17:47 +08:00
co-authored by OmX
parent e4997efeb1
commit d632fd9f64
13 changed files with 1919 additions and 21 deletions
Download Patch File Download Diff File Expand all files Collapse all files
README.md
+3 -3
View File
@@ -6,11 +6,11 @@
本仓库已经补齐最小 Agent / 蜂群 Agent 标准化测试闭环。这里的标准不是普通软件开发质量标准，而是围绕 Agent 的任务理解、上下文承接、模型选择、工具边界、敏感信息保护、可观测性、handoff、信息素协作、收敛和马尔可夫式状态转移建立的 AQS / SW-AQS v1。
最新 live 标准矩阵已通过：
最新标准矩阵（S01-S08）已通过；以下是 S07 live 证据：
```json
{
"run_id": "063632eeb17b45c197aa866066158667",
"run_id": "04c641d170fe4ea7aa3d882d9df37cca",
"completed_tasks": 7,
"accepted_score": 1.0,
"check_count": 12,
@@ -23,7 +23,7 @@
- `docs/AGENT_SWARM_QUALITY_STANDARD.zh-CN.md`：Agent / 蜂群 Agent 质量标准、测试标准来源、S07 任务分配和接手机制。
- `docs/ACADEMIC_STANDARD_TEST_REPORT.zh-CN.md`：测试场景、测试过程、测试结果、算法清单和马尔可夫过程判定。
- `docs/MODEL_AGNET_IO_REPORT.zh-CN.md`：每个模型 / Agnet 的任务、输入、输出、评分和交接过程。
- `docs/STANDARD_TEST_MATRIX.md`：S01-S07 标准矩阵。
- `docs/STANDARD_TEST_MATRIX.md`：S01-S08 标准矩阵。
- `docs/INDUSTRY_STANDARD_AGNET_TEST_PLAN.md`：行业参考框架到本项目标准的映射。
目标是先把蜂群的四个共享资源跑通：
docs/ACADEMIC_STANDARD_TEST_REPORT.zh-CN.md
+10 -6
View File
@@ -2,7 +2,7 @@
**对象**: `swarm-minimal` 最小蜂群原型
**日期**: 2026-05-16
**范围**: 本地确定性测试、蜂群行为验收、传统 Agnet 对比、多轮共识、马尔可夫过程适配性判断
**范围**: 本地确定性测试、蜂群行为验收、传统 Agnet 对比、多轮共识、模型 I/O 报告审计、马尔可夫过程适配性判断
## 1. 已实现的场景测试
@@ -14,6 +14,8 @@
| S04 | 依赖边界 | MVP 不把 NATS/Cosmos 当成必需依赖 | `tests/test_standard_scenarios.py` |
| S05 | 最终收敛评分 | 最终 STEP-07 输出必须优于中间步骤 | `tests/test_standard_scenarios.py` |
| S06 | 失败注入 | 失败任务要标记失败、产生负信息素，其他任务仍可收敛 | `tests/test_standard_scenarios.py` |
| S07 | live Azure/NewAPI 连续推理 | 真实连接 Azure PostgreSQL、Redis、Blob 和 NewAPI，跑 7 步连续推理链 | `examples/run_continuous_reasoning_acceptance.py` |
| S08 | 模型 I/O 报告审计 | 验证报告包含场景、输入、输出、接手证据，且无明显真实密钥样式 | `tests/test_model_io_report_audit.py` |
| B01 | 单 Agnet 故障隔离 | 一个 Agnet 崩溃不导致整体失败 | `tests/test_swarm_behavior_academic.py` |
| B02 | 群体涌现 | 多个弱局部信号通过共享状态聚合成更强整体结果 | `tests/test_swarm_behavior_academic.py` |
| B03 | 信息素间接协作 | 信息素影响任务 claim 顺序并形成正反馈 | `tests/test_swarm_behavior_academic.py` |
@@ -52,7 +54,7 @@ python3 -u -B examples/run_consensus_convergence_acceptance.py
python3 -u -B examples/run_academic_standard_evaluation.py
```
完整标准矩阵还包含 S07 live 场景：
完整标准矩阵还包含 S07 live 场景和 S08 模型 I/O 报告审计：
```bash
python3 -u -B examples/run_standard_scenario_acceptance.py
@@ -65,19 +67,20 @@ S07 需要本地被 git 忽略的 `.env`，其中包含 Azure PostgreSQL、Redis
| 命令 | 本轮结果 | 说明 |
| --- | --- | --- |
| `python3 -B -m py_compile swarm_minimal/*.py examples/*.py tests/*.py` | PASS | 无输出，表示所有模块编译通过 |
| `python3 -B -m unittest discover -s tests` | PASS | 26 个测试通过 |
| `python3 -B -m unittest discover -s tests` | PASS | 28 个测试通过 |
| `python3 -B -m unittest tests.test_markov_process_properties` | PASS | 3 个马尔可夫适配性测试通过 |
| `python3 -B -m unittest tests.test_model_io_report_audit` | PASS | 2 个模型 I/O 报告审计测试通过 |
| `python3 -u -B examples/run_swarm_behavior_acceptance.py` | PASS | B01-B04 全部通过 |
| `python3 -u -B examples/run_swarm_vs_traditional_benchmark.py` | PASS | 蜂群总归一化分 0.9175，传统基线 0.1958，ratio 4.69 |
| `python3 -u -B examples/run_consensus_convergence_acceptance.py` | PASS | 2 轮收敛，接受 `lease_based_pg_queue` |
| `python3 -u -B examples/run_academic_standard_evaluation.py` | PASS | A01-A05 本地学术化门禁全部通过，检测到 `examples/.env` 可用于 live 测试 |
| `python3 -u -B examples/run_standard_scenario_acceptance.py` | PASS | S01-S07 全部通过；S07 完成 7 个 live 任务，12 个检查无失败 |
| `python3 -u -B examples/run_standard_scenario_acceptance.py` | PASS | S01-S08 全部通过；S07 完成 7 个 live 任务，12 个检查无失败，S08 报告审计通过 |
最新 S07 live 证据：
```json
{
"run_id": "063632eeb17b45c197aa866066158667",
"run_id": "04c641d170fe4ea7aa3d882d9df37cca",
"completed_tasks": 7,
"accepted_score": 1.0,
"selected_models": [
@@ -87,7 +90,7 @@ S07 需要本地被 git 忽略的 `.env`，其中包含 Azure PostgreSQL、Redis
],
"failed_checks": [],
"check_count": 12,
"artifact_path": "swarm-runs/063632eeb17b45c197aa866066158667/result.json"
"artifact_path": "swarm-runs/04c641d170fe4ea7aa3d882d9df37cca/result.json"
}
```
@@ -98,6 +101,7 @@ S07 需要本地被 git 忽略的 `.env`，其中包含 Azure PostgreSQL、Redis
| 本地学术化门禁 A01-A05 | 可通过 |
| 标准矩阵 S01-S06 | 可通过 |
| 标准矩阵 S07 live Azure/NewAPI | 已通过 |
| 标准矩阵 S08 模型 I/O 报告审计 | 已通过 |
| 生产级标准化验收 | 当前最小标准已满足 |
结论：当前项目已经具备可执行的标准化测试，且本地确定性蜂群行为测试与真实 Azure/NewAPI live 集成测试均已通过。该结论限定在本仓库定义的最小蜂群标准矩阵内，不等同于大规模生产压测或第三方认证。
docs/AGENT_SWARM_QUALITY_STANDARD.zh-CN.md
+3 -3
View File
@@ -44,7 +44,7 @@
| AQS-09 错误可解释 | 失败任务必须记录 error、failed observation 和负信息素 | S06/B01/C01 |
| AQS-10 交接准备度 | 输出要给下一个 Agent 留出摘要、风险和下一步 | S07 prompt/output 中强制“下一步/交接” |
| AQS-11 外部依赖真实性 | live 测试必须真实连 PostgreSQL、Redis、Blob、NewAPI | S07 live integration PASS |
| AQS-12 人类审计友好 | 最终报告必须能回答：任务、输入、输出、接手、结果、未满足项 | `MODEL_AGNET_IO_REPORT.zh-CN.md` 和本文件 |
| AQS-12 人类审计友好 | 最终报告必须能回答：任务、输入、输出、接手、结果、未满足项 | S08 + `MODEL_AGNET_IO_REPORT.zh-CN.md` 和本文件 |
## 4. 蜂群 Agent 质量标准 SW-AQS
@@ -122,11 +122,11 @@
## 7. 本轮判定
最新标准矩阵 run：
最新标准矩阵 run（S01-S08 全部通过，以下为 S07 live 证据）：
```json
{
"run_id": "063632eeb17b45c197aa866066158667",
"run_id": "04c641d170fe4ea7aa3d882d9df37cca",
"completed_tasks": 7,
"accepted_score": 1.0,
"check_count": 12,
docs/INDUSTRY_STANDARD_AGNET_TEST_PLAN.md
+2 -2
View File
@@ -28,11 +28,11 @@
3. 模型输入输出能被审计：知道每个 Agent 被分配了什么任务、收到了什么输入、输出了什么、下一个 Agent 如何接手。
4. live 测试真实连接 Azure PostgreSQL、Redis、Blob 和 NewAPI，且不输出任何真实密钥。
最新一次标准矩阵已经通过：
最新一次标准矩阵（S01-S08）已经通过，以下为 S07 live 证据：
```json
{
"run_id": "063632eeb17b45c197aa866066158667",
"run_id": "04c641d170fe4ea7aa3d882d9df37cca",
"completed_tasks": 7,
"accepted_score": 1.0,
"check_count": 12,
docs/MODEL_AGNET_IO_REPORT.zh-CN.md
+1815 -2
View File
File diff suppressed because it is too large Load Diff
docs/README.md
+1 -1
View File
@@ -5,7 +5,7 @@
- `AGENT_SWARM_QUALITY_STANDARD.zh-CN.md`：AQS / SW-AQS v1 主标准，说明 Agent 和蜂群 Agent 的质量项、S07 任务分配、模型交接和当前结论。
- `ACADEMIC_STANDARD_TEST_REPORT.zh-CN.md`：测试场景、测试过程、测试结果、算法清单和马尔可夫过程判定。
- `MODEL_AGNET_IO_REPORT.zh-CN.md`：从 live run 导出的模型 / Agnet 任务、输入、输出、评分和交接过程。
- `STANDARD_TEST_MATRIX.md`：S01-S07 标准化测试矩阵。
- `STANDARD_TEST_MATRIX.md`：S01-S08 标准化测试矩阵。
- `INDUSTRY_STANDARD_AGNET_TEST_PLAN.md`：行业参考框架到本项目 Agent 质量标准的映射。
- `JUEJIN_SWARM_ARTICLE_TEST_STANDARD.zh-CN.md`：掘金蜂群文章对应的测试标准说明。
- `SWARM_BEHAVIOR_TEST_MATRIX.md`：蜂群行为测试矩阵。
docs/STANDARD_TEST_MATRIX.md
+2
View File
@@ -13,6 +13,7 @@ result means more than "one long run completed".
| Deterministic scenarios | No-network Given/When/Then cases for continuity, policy, final scoring, and failure injection | `./.venv/bin/python -B -m unittest tests.test_standard_scenarios` |
| Academic Markov fit | No-network checks for Markov-style transition behavior and non-MDP limitations | `./.venv/bin/python -B -m unittest tests.test_markov_process_properties` |
| Live integration | Real NewAPI + Azure PostgreSQL/Redis/Blob continuous reasoning chain | `./.venv/bin/python -u -B examples/run_continuous_reasoning_acceptance.py` |
| Report audit | Human-auditable model I/O report with scenario, input, output, handoff and secret-safety evidence | `./.venv/bin/python -B -m unittest tests.test_model_io_report_audit` |
## Required Scenarios
@@ -25,6 +26,7 @@ result means more than "one long run completed".
| S05 | Final convergence scoring | STEP-06 and STEP-07 both produce rich outputs | Score both outputs | STEP-07 is strictly selected as final convergence |
| S06 | Failure injection | One deterministic worker raises | Coordinator runs to convergence | Failed task is marked failed, pheromone is negative, completed task still converges |
| S07 | Live resource closure | Real Azure/NewAPI env is loaded | Run seven-step continuous reasoning | PostgreSQL, Redis, Blob, stream events, model discovery, and final artifact all pass |
| S08 | Model I/O report audit | Generated model/Agnet I/O report | Audit scenario coverage, task input/output, handoff evidence, and obvious secret patterns | Report is human-auditable and contains no obvious secret values |
| M01 | Markov-style claim transition | Same current tasks and pheromone values but different prior paths | Claim next task | The same high-pheromone task is claimed |
| M02 | Markov-style score transition | Same current task, agent, output, and score but different prior paths | Complete task | Task status, output, score, and pheromone update match |
| M03 | Formal MDP limitation | Project metadata is evaluated | Assess Markov fit | Prototype is classified as Markov-style state machine, not a formal Markov process or MDP |
examples/export_model_agnet_io_report.py
+24 -3
View File
@@ -14,6 +14,8 @@ from swarm_minimal.local_env import load_project_env
RUN_IDS = [
"04c641d170fe4ea7aa3d882d9df37cca",
"c592a7ca4c0f4e02b93a1390b62d0af7",
"063632eeb17b45c197aa866066158667",
"3e8e58ae4e084bc8b90cf5c46f8992f3",
"78f189ccd1924ed0a4fb0a0a447ad449",
@@ -46,6 +48,12 @@ TEST_SCENARIOS = (
"purpose": "用真实 Azure PostgreSQL、Redis、Blob 和 NewAPI 跑 7 步连续推理链。",
"evidence": "本报告下方每个 live run 的 task.input / task.output / handoff 记录。",
},
{
"id": "S08",
"name": "model_io_report_audit",
"purpose": "验证本报告包含场景、输入、输出、交接证据，且没有明显真实密钥样式。",
"evidence": "`unittest tests.test_model_io_report_audit`",
},
)
OUTPUT_PATH = ROOT / "docs" / "MODEL_AGNET_IO_REPORT.zh-CN.md"
@@ -85,7 +93,7 @@ def build_report(store: PostgresRedisBlobSwarmStore) -> str:
sections.extend(
[
"",
"最新重跑结论：S01-S07 全部 PASS；最新 S07 live run 会排在下方第一个。",
"最新重跑结论：S01-S08 全部 PASS；最新 S07 live run 会排在下方第一个。",
"",
]
)
@@ -123,7 +131,7 @@ def render_run(convergence: dict[str, object], tasks: dict[str, dict[str, object
for index, observation in enumerate(convergence["observations"], start=1):
task_id = observation["task_id"]
task = tasks[task_id]
output = str(task.get("output") or "")
output = redact_sensitive_text(str(task.get("output") or ""))
model = infer_model(task, output)
lines.extend(
[
@@ -148,7 +156,7 @@ def render_run(convergence: dict[str, object], tasks: dict[str, dict[str, object
"#### 本次任务输入 task.input",
"",
"```text",
str(task["input"]).strip(),
redact_sensitive_text(str(task["input"])).strip(),
"```",
"",
"#### Agnet / 模型实际输出 task.output",
@@ -162,6 +170,19 @@ def render_run(convergence: dict[str, object], tasks: dict[str, dict[str, object
return lines
def redact_sensitive_text(text: str) -> str:
replacements = [
(r"sk-[A-Za-z0-9]{20,}", "sk-<redacted>"),
(r"AccountKey=[^;\s`]+", "AccountKey=<redacted>"),
(r"password=[^,;\s`]+", "password=<redacted>"),
(r"BEGIN [A-Z ]*PRIVATE KEY", "BEGIN <redacted> PRIVATE KEY"),
]
redacted = text
for pattern, replacement in replacements:
redacted = re.sub(pattern, replacement, redacted, flags=re.IGNORECASE)
return redacted
def handoff_description(goal: str, run_id: str, task: dict[str, object], output: str) -> str:
if goal.startswith("连续性长推理场景"):
match = re.search(r"chain_edge=([^;\\n]+)", output)
examples/run_academic_standard_evaluation.py
+1 -1
View File
@@ -91,7 +91,7 @@ def main() -> None:
},
"pass_condition": {
"local_academic_gate": "all A01-A05 checks pass",
"full_standard_gate": "local_academic_gate plus S07 live Azure/NewAPI scenario",
"full_standard_gate": "local_academic_gate plus S07 live Azure/NewAPI scenario and S08 model I/O report audit",
},
}
print(json.dumps(report, ensure_ascii=False, indent=2))
examples/run_continuous_reasoning_acceptance.py
+6
View File
@@ -523,6 +523,8 @@ def no_required_nats_or_cosmos(text: str) -> bool:
"不引入",
"不使用",
"不依赖",
"不可",
"不可作为",
"无需",
"不要",
"不做",
@@ -532,6 +534,10 @@ def no_required_nats_or_cosmos(text: str) -> bool:
"未涉及",
"已排除",
"排除",
"反例",
"违反",
"拒绝",
"严重错误",
"误依赖",
"非必需",
"no ",
examples/run_standard_scenario_acceptance.py
+9
View File
@@ -45,6 +45,15 @@ SCENARIOS = [
"command": [sys.executable, "-u", "-B", "examples/run_continuous_reasoning_acceptance.py"],
"parse_json": True,
},
{
"id": "S08",
"name": "model_io_report_audit",
"layer": "report-audit",
"given": "generated docs/MODEL_AGNET_IO_REPORT.zh-CN.md",
"when": "audit scenario coverage, task input/output sections, handoff evidence, and obvious secret patterns",
"then": "the report is human-auditable and does not contain obvious secret values",
"command": [sys.executable, "-B", "-m", "unittest", "tests.test_model_io_report_audit"],
},
]
tests/test_model_io_report_audit.py
+41
View File
@@ -0,0 +1,41 @@
from pathlib import Path
import re
import unittest
ROOT = Path(__file__).resolve().parents[1]
REPORT = ROOT / "docs" / "MODEL_AGNET_IO_REPORT.zh-CN.md"
class ModelIoReportAuditTests(unittest.TestCase):
def test_report_exposes_scenarios_inputs_outputs_and_handoff(self) -> None:
text = REPORT.read_text(encoding="utf-8")
for required in [
"## 本轮测试场景补充",
"| S01 | syntax_import_sanity |",
"| S02 | unit_regression |",
"| S03-S06 | deterministic_standard_scenarios |",
"| S07 | live_azure_newapi_continuous_reasoning |",
"| S08 | model_io_report_audit |",
"#### 本次任务输入 task.input",
"#### Agnet / 模型实际输出 task.output",
"#### 接手 / 交接机制",
"chain_edge=",
]:
self.assertIn(required, text)
def test_report_does_not_contain_obvious_secret_values(self) -> None:
text = REPORT.read_text(encoding="utf-8")
forbidden_patterns = [
r"sk-[A-Za-z0-9]{20,}",
r"AccountKey=[^<\s`]+",
r"password=[^<\s`]+",
r"BEGIN [A-Z ]*PRIVATE KEY",
]
for pattern in forbidden_patterns:
self.assertIsNone(re.search(pattern, text, flags=re.IGNORECASE))
if __name__ == "__main__":
unittest.main()
tests/test_standard_scenarios.py
+2
View File
@@ -20,6 +20,8 @@ class StandardScenarioTest(unittest.TestCase):
def test_nats_and_cosmos_policy_rejects_required_dependencies_only(self) -> None:
self.assertTrue(continuous.no_required_nats_or_cosmos("MVP 不依赖 NATS/Cosmos，仅使用 PostgreSQL + Redis + Blob。"))
self.assertTrue(continuous.no_required_nats_or_cosmos("NATS 和 Cosmos 未在 MVP 中涉及。"))
self.assertTrue(continuous.no_required_nats_or_cosmos("NATS 和 Cosmos 不可作为 MVP 依赖。"))
self.assertTrue(continuous.no_required_nats_or_cosmos("反例：使用 NATS 作为必须依赖，违反验收标准。"))
self.assertFalse(continuous.no_required_nats_or_cosmos("MVP 必须依赖 NATS 才能完成任务队列。"))
self.assertFalse(continuous.no_required_nats_or_cosmos("需要引入 Cosmos 作为任务状态库。"))