gongzhiyongandOmX
fc877dad24
Keep the repository root focused on README while moving the remaining planning and swarm test documents under docs/ and indexing them there. Constraint: The user explicitly requested that documents should live under docs/. Rejected: Leaving older root-level Markdown files in place | it keeps the homepage cluttered and splits the documentation surface. Confidence: high Scope-risk: narrow Directive: Keep future ordinary documentation under docs/ unless it is a root entrypoint such as README.md. Tested: git diff --cached --check; find . -maxdepth 1 -type f -name '*.md' shows only README.md. Not-tested: No runtime tests rerun because this is a documentation path-only move. Co-authored-by: OmX <omx@oh-my-codex.dev>
364 lines
8.0 KiB
Markdown
364 lines
8.0 KiB
Markdown
# Swarm Test Input / Output Report
|
|
|
|
This report shows the actual deterministic test content for the minimal swarm
|
|
prototype. It separates the swarm-behavior tests from the traditional-vs-swarm
|
|
comparison benchmark.
|
|
|
|
## Commands Executed
|
|
|
|
```bash
|
|
./.venv/bin/python -u -B examples/run_swarm_behavior_acceptance.py
|
|
./.venv/bin/python -u -B examples/run_swarm_vs_traditional_benchmark.py
|
|
```
|
|
|
|
Both commands returned `status: PASS`.
|
|
|
|
## Behavior Test Content
|
|
|
|
### B01: Single Agnet Failure Isolation
|
|
|
|
Purpose: prove one failed Agnet does not collapse the swarm result.
|
|
|
|
Input:
|
|
|
|
```json
|
|
{
|
|
"run_id": "behavior-fault-isolation",
|
|
"goal": "fault isolation with redundant routes",
|
|
"tasks": [
|
|
{"kind": "route", "input": "fragile-route"},
|
|
{"kind": "route", "input": "robust-route-a"},
|
|
{"kind": "route", "input": "robust-route-b"}
|
|
],
|
|
"agents": [
|
|
{"id": "crashing-agnet", "capability": "route", "behavior": "raise RuntimeError('single agnet crashed')"},
|
|
{"id": "backup-agnet-a", "capability": "route", "behavior": "return healthy result, score 0.91"},
|
|
{"id": "backup-agnet-b", "capability": "route", "behavior": "return alternative result, score 0.86"}
|
|
]
|
|
}
|
|
```
|
|
|
|
Expected output:
|
|
|
|
```json
|
|
{
|
|
"failed_tasks": 1,
|
|
"done_tasks": 2,
|
|
"run_status": "converged",
|
|
"accepted_output_contains": "healthy result",
|
|
"observations_include": ["route:failed", "route:done"]
|
|
}
|
|
```
|
|
|
|
Actual output:
|
|
|
|
```json
|
|
{
|
|
"passed": true,
|
|
"evidence": "Ran 1 test ... OK"
|
|
}
|
|
```
|
|
|
|
Concrete assertions are in
|
|
`tests/test_swarm_behavior_academic.py::test_single_agnet_failure_isolated_by_redundant_convergence`.
|
|
|
|
### B02: Emergent Consensus From Local Evidence
|
|
|
|
Purpose: prove group-level aggregation can beat the best single local signal.
|
|
|
|
Input:
|
|
|
|
```json
|
|
{
|
|
"run_id": "behavior-emergent-consensus",
|
|
"goal": "local evidence should create group consensus",
|
|
"tasks": [
|
|
{"kind": "evidence", "input": "alpha:0.31"},
|
|
{"kind": "evidence", "input": "beta:0.33"},
|
|
{"kind": "evidence", "input": "beta:0.34"},
|
|
{"kind": "evidence", "input": "gamma:0.45"}
|
|
],
|
|
"rule": "each agent adds its local value into shared_state['candidate:<name>:score']"
|
|
}
|
|
```
|
|
|
|
Traditional baseline output:
|
|
|
|
```json
|
|
{
|
|
"best_single_candidate": "gamma",
|
|
"best_single_score": 0.45
|
|
}
|
|
```
|
|
|
|
Expected swarm output:
|
|
|
|
```json
|
|
{
|
|
"accepted_candidate": "beta",
|
|
"aggregated_beta_score": 0.67,
|
|
"accepted_score_greater_than_best_single_weak_signal": true,
|
|
"run_status": "converged"
|
|
}
|
|
```
|
|
|
|
Actual output:
|
|
|
|
```json
|
|
{
|
|
"passed": true,
|
|
"evidence": "Ran 1 test ... OK"
|
|
}
|
|
```
|
|
|
|
Concrete assertions are in
|
|
`tests/test_swarm_behavior_academic.py::test_emergent_consensus_accumulates_local_evidence`.
|
|
|
|
### B03: Pheromone / Stigmergy Selection
|
|
|
|
Purpose: prove the shared pheromone environment changes task selection order.
|
|
|
|
Input:
|
|
|
|
```json
|
|
{
|
|
"run_id": "behavior-pheromone",
|
|
"goal": "pheromone should bias task selection",
|
|
"tasks": [
|
|
{"kind": "probe", "input": "low-signal", "initial_pheromone": 0.1, "completion_score": 0.31},
|
|
{"kind": "probe", "input": "high-signal", "initial_pheromone": 0.9, "completion_score": 0.82},
|
|
{"kind": "probe", "input": "medium-signal", "initial_pheromone": 0.4, "completion_score": 0.62}
|
|
],
|
|
"claim_rule": "claim_next sorts pending tasks by pheromone descending"
|
|
}
|
|
```
|
|
|
|
Expected output:
|
|
|
|
```json
|
|
{
|
|
"first_claimed_task": "high-signal",
|
|
"accepted_output_contains": "high-signal",
|
|
"final_pheromone_order": ["high-signal", "medium-signal", "low-signal"]
|
|
}
|
|
```
|
|
|
|
Actual output:
|
|
|
|
```json
|
|
{
|
|
"passed": true,
|
|
"evidence": "Ran 1 test ... OK"
|
|
}
|
|
```
|
|
|
|
Concrete assertions are in
|
|
`tests/test_swarm_behavior_academic.py::test_pheromone_biases_claim_order_and_records_positive_feedback`.
|
|
|
|
### B04: LangGraph-Style Handoff Continuity
|
|
|
|
Purpose: prove handoff preserves active agent and payload continuity.
|
|
|
|
Input:
|
|
|
|
```json
|
|
{
|
|
"run_id": "behavior-handoff",
|
|
"goal": "handoff continuity",
|
|
"initial_active_agent": "collector",
|
|
"tasks": [
|
|
{"kind": "collector", "input": "collect code facts"},
|
|
{"kind": "analyst", "input": "analyze code facts"},
|
|
{"kind": "reporter", "input": "write final report"}
|
|
],
|
|
"handoffs": [
|
|
{"from": "collector", "to": "analyst", "tool_name": "transfer_to_analyst"},
|
|
{"from": "analyst", "to": "reporter", "tool_name": "transfer_to_reporter"}
|
|
],
|
|
"payload": "facts: task_pool pheromone shared_state convergence; analysis: ready"
|
|
}
|
|
```
|
|
|
|
Expected output:
|
|
|
|
```json
|
|
{
|
|
"handoff_collector_to_analyst": "transfer_to_analyst",
|
|
"handoff_analyst_to_reporter": "transfer_to_reporter",
|
|
"final_output_contains": "reporter final result",
|
|
"completed_tasks": 3
|
|
}
|
|
```
|
|
|
|
Actual output:
|
|
|
|
```json
|
|
{
|
|
"passed": true,
|
|
"evidence": "Ran 1 test ... OK"
|
|
}
|
|
```
|
|
|
|
Concrete assertions are in
|
|
`tests/test_swarm_behavior_academic.py::test_langgraph_style_handoff_preserves_active_agent_and_payload`.
|
|
|
|
## Traditional vs Swarm Benchmark Content
|
|
|
|
### Baseline Definitions
|
|
|
|
Traditional Agnet:
|
|
|
|
```json
|
|
[
|
|
"single agent fails closed when its one route fails",
|
|
"best-of local answers without shared-state aggregation",
|
|
"FIFO task selection without pheromone feedback",
|
|
"stateless handoff without active-agent/payload continuity"
|
|
]
|
|
```
|
|
|
|
Swarm Agnet:
|
|
|
|
```json
|
|
[
|
|
"redundant agents share task pool and converge despite a failed individual",
|
|
"local observations accumulate through shared_state",
|
|
"pheromone scores bias claim order and final selection",
|
|
"handoff records active agent, transfer target, and payload continuity"
|
|
]
|
|
```
|
|
|
|
### C01: Fault Isolation Comparison
|
|
|
|
Input:
|
|
|
|
```json
|
|
{
|
|
"traditional": {
|
|
"tasks": [{"kind": "route", "input": "fragile-route"}],
|
|
"agents": [{"id": "single-agnet", "behavior": "raise RuntimeError('single agnet crashed')"}]
|
|
},
|
|
"swarm": {
|
|
"tasks": ["fragile-route", "robust-route-a", "robust-route-b"],
|
|
"agents": ["crashing-agnet", "backup-agnet-a", "backup-agnet-b"]
|
|
}
|
|
}
|
|
```
|
|
|
|
Actual output:
|
|
|
|
```json
|
|
{
|
|
"traditional": 0.0,
|
|
"swarm": 1.0,
|
|
"traditional_result": "failed before convergence",
|
|
"swarm_completed_tasks": 2,
|
|
"swarm_failed_tasks": 1
|
|
}
|
|
```
|
|
|
|
### C02: Emergent Consensus Comparison
|
|
|
|
Input:
|
|
|
|
```json
|
|
{
|
|
"local_values": {
|
|
"alpha": [0.31],
|
|
"beta": [0.33, 0.34],
|
|
"gamma": [0.45]
|
|
}
|
|
}
|
|
```
|
|
|
|
Actual output:
|
|
|
|
```json
|
|
{
|
|
"traditional_best_single": {"candidate": "gamma", "score": 0.45},
|
|
"swarm_aggregated_best": {"candidate": "beta", "score": 0.67},
|
|
"relative_gain_percent": 48.9
|
|
}
|
|
```
|
|
|
|
### C03: Pheromone Efficiency Comparison
|
|
|
|
Input:
|
|
|
|
```json
|
|
{
|
|
"traditional_order": ["low-signal", "medium-signal", "high-signal"],
|
|
"swarm_pheromone_order": ["high-signal", "medium-signal", "low-signal"],
|
|
"quality": {
|
|
"low-signal": 0.31,
|
|
"medium-signal": 0.62,
|
|
"high-signal": 0.82
|
|
}
|
|
}
|
|
```
|
|
|
|
Actual output:
|
|
|
|
```json
|
|
{
|
|
"traditional_steps_to_best": 3,
|
|
"swarm_steps_to_best": 1,
|
|
"best_route_efficiency_traditional": 0.3333,
|
|
"best_route_efficiency_swarm": 1.0,
|
|
"relative_gain_percent": 200.0,
|
|
"first_claim_quality_gain_percent": 164.5,
|
|
"steps_to_best_reduction_percent": 66.7
|
|
}
|
|
```
|
|
|
|
### C04: Handoff Context Retention Comparison
|
|
|
|
Input:
|
|
|
|
```json
|
|
{
|
|
"required_context": [
|
|
"task_pool",
|
|
"pheromone",
|
|
"shared_state",
|
|
"convergence",
|
|
"analysis: ready"
|
|
],
|
|
"traditional_payload": "final report",
|
|
"swarm_payload": "facts: task_pool pheromone shared_state convergence; analysis: ready"
|
|
}
|
|
```
|
|
|
|
Actual output:
|
|
|
|
```json
|
|
{
|
|
"traditional_retained_ratio": 0.0,
|
|
"swarm_retained_ratio": 1.0
|
|
}
|
|
```
|
|
|
|
## Full Benchmark Output
|
|
|
|
```json
|
|
{
|
|
"standard": "swarm-vs-traditional-deterministic-benchmark-v1",
|
|
"status": "PASS",
|
|
"overall_normalized_score": {
|
|
"traditional": 0.1958,
|
|
"swarm": 0.9175,
|
|
"relative_gain_percent": 368.5,
|
|
"ratio": 4.69,
|
|
"note": "This aggregate is a deterministic academic benchmark over four selected swarm properties, not a universal production claim."
|
|
}
|
|
}
|
|
```
|
|
|
|
## Verification Boundary
|
|
|
|
This report proves deterministic behavior for the four selected swarm properties:
|
|
fault isolation, emergence, pheromone coordination, and handoff continuity. It
|
|
does not prove all production-scale properties, large-cluster performance,
|
|
malicious-agent resistance, or certified compatibility with the external
|
|
LangGraph package.
|