fengqun/docs/SWARM_TEST_INPUT_OUTPUT_REPORT.md

Swarm Test Input / Output Report

This report shows the actual deterministic test content for the minimal swarm prototype. It separates the swarm-behavior tests from the traditional-vs-swarm comparison benchmark.

Commands Executed

./.venv/bin/python -u -B examples/run_swarm_behavior_acceptance.py
./.venv/bin/python -u -B examples/run_swarm_vs_traditional_benchmark.py

Both commands returned status: PASS.

Behavior Test Content

B01: Single Agnet Failure Isolation

Purpose: prove one failed Agnet does not collapse the swarm result.

Input:

{
  "run_id": "behavior-fault-isolation",
  "goal": "fault isolation with redundant routes",
  "tasks": [
    {"kind": "route", "input": "fragile-route"},
    {"kind": "route", "input": "robust-route-a"},
    {"kind": "route", "input": "robust-route-b"}
  ],
  "agents": [
    {"id": "crashing-agnet", "capability": "route", "behavior": "raise RuntimeError('single agnet crashed')"},
    {"id": "backup-agnet-a", "capability": "route", "behavior": "return healthy result, score 0.91"},
    {"id": "backup-agnet-b", "capability": "route", "behavior": "return alternative result, score 0.86"}
  ]
}

Expected output:

{
  "failed_tasks": 1,
  "done_tasks": 2,
  "run_status": "converged",
  "accepted_output_contains": "healthy result",
  "observations_include": ["route:failed", "route:done"]
}

Actual output:

{
  "passed": true,
  "evidence": "Ran 1 test ... OK"
}

Concrete assertions are in tests/test_swarm_behavior_academic.py::test_single_agnet_failure_isolated_by_redundant_convergence.

B02: Emergent Consensus From Local Evidence

Purpose: prove group-level aggregation can beat the best single local signal.

Input:

{
  "run_id": "behavior-emergent-consensus",
  "goal": "local evidence should create group consensus",
  "tasks": [
    {"kind": "evidence", "input": "alpha:0.31"},
    {"kind": "evidence", "input": "beta:0.33"},
    {"kind": "evidence", "input": "beta:0.34"},
    {"kind": "evidence", "input": "gamma:0.45"}
  ],
  "rule": "each agent adds its local value into shared_state['candidate:<name>:score']"
}

Traditional baseline output:

{
  "best_single_candidate": "gamma",
  "best_single_score": 0.45
}

Expected swarm output:

{
  "accepted_candidate": "beta",
  "aggregated_beta_score": 0.67,
  "accepted_score_greater_than_best_single_weak_signal": true,
  "run_status": "converged"
}

Actual output:

{
  "passed": true,
  "evidence": "Ran 1 test ... OK"
}

Concrete assertions are in tests/test_swarm_behavior_academic.py::test_emergent_consensus_accumulates_local_evidence.

B03: Pheromone / Stigmergy Selection

Purpose: prove the shared pheromone environment changes task selection order.

Input:

{
  "run_id": "behavior-pheromone",
  "goal": "pheromone should bias task selection",
  "tasks": [
    {"kind": "probe", "input": "low-signal", "initial_pheromone": 0.1, "completion_score": 0.31},
    {"kind": "probe", "input": "high-signal", "initial_pheromone": 0.9, "completion_score": 0.82},
    {"kind": "probe", "input": "medium-signal", "initial_pheromone": 0.4, "completion_score": 0.62}
  ],
  "claim_rule": "claim_next sorts pending tasks by pheromone descending"
}

Expected output:

{
  "first_claimed_task": "high-signal",
  "accepted_output_contains": "high-signal",
  "final_pheromone_order": ["high-signal", "medium-signal", "low-signal"]
}

Actual output:

{
  "passed": true,
  "evidence": "Ran 1 test ... OK"
}

Concrete assertions are in tests/test_swarm_behavior_academic.py::test_pheromone_biases_claim_order_and_records_positive_feedback.

B04: LangGraph-Style Handoff Continuity

Purpose: prove handoff preserves active agent and payload continuity.

Input:

{
  "run_id": "behavior-handoff",
  "goal": "handoff continuity",
  "initial_active_agent": "collector",
  "tasks": [
    {"kind": "collector", "input": "collect code facts"},
    {"kind": "analyst", "input": "analyze code facts"},
    {"kind": "reporter", "input": "write final report"}
  ],
  "handoffs": [
    {"from": "collector", "to": "analyst", "tool_name": "transfer_to_analyst"},
    {"from": "analyst", "to": "reporter", "tool_name": "transfer_to_reporter"}
  ],
  "payload": "facts: task_pool pheromone shared_state convergence; analysis: ready"
}

Expected output:

{
  "handoff_collector_to_analyst": "transfer_to_analyst",
  "handoff_analyst_to_reporter": "transfer_to_reporter",
  "final_output_contains": "reporter final result",
  "completed_tasks": 3
}

Actual output:

{
  "passed": true,
  "evidence": "Ran 1 test ... OK"
}

Concrete assertions are in tests/test_swarm_behavior_academic.py::test_langgraph_style_handoff_preserves_active_agent_and_payload.

Traditional vs Swarm Benchmark Content

Baseline Definitions

Traditional Agnet:

[
  "single agent fails closed when its one route fails",
  "best-of local answers without shared-state aggregation",
  "FIFO task selection without pheromone feedback",
  "stateless handoff without active-agent/payload continuity"
]

Swarm Agnet:

[
  "redundant agents share task pool and converge despite a failed individual",
  "local observations accumulate through shared_state",
  "pheromone scores bias claim order and final selection",
  "handoff records active agent, transfer target, and payload continuity"
]

C01: Fault Isolation Comparison

Input:

{
  "traditional": {
    "tasks": [{"kind": "route", "input": "fragile-route"}],
    "agents": [{"id": "single-agnet", "behavior": "raise RuntimeError('single agnet crashed')"}]
  },
  "swarm": {
    "tasks": ["fragile-route", "robust-route-a", "robust-route-b"],
    "agents": ["crashing-agnet", "backup-agnet-a", "backup-agnet-b"]
  }
}

Actual output:

{
  "traditional": 0.0,
  "swarm": 1.0,
  "traditional_result": "failed before convergence",
  "swarm_completed_tasks": 2,
  "swarm_failed_tasks": 1
}

C02: Emergent Consensus Comparison

Input:

{
  "local_values": {
    "alpha": [0.31],
    "beta": [0.33, 0.34],
    "gamma": [0.45]
  }
}

Actual output:

{
  "traditional_best_single": {"candidate": "gamma", "score": 0.45},
  "swarm_aggregated_best": {"candidate": "beta", "score": 0.67},
  "relative_gain_percent": 48.9
}

C03: Pheromone Efficiency Comparison

Input:

{
  "traditional_order": ["low-signal", "medium-signal", "high-signal"],
  "swarm_pheromone_order": ["high-signal", "medium-signal", "low-signal"],
  "quality": {
    "low-signal": 0.31,
    "medium-signal": 0.62,
    "high-signal": 0.82
  }
}

Actual output:

{
  "traditional_steps_to_best": 3,
  "swarm_steps_to_best": 1,
  "best_route_efficiency_traditional": 0.3333,
  "best_route_efficiency_swarm": 1.0,
  "relative_gain_percent": 200.0,
  "first_claim_quality_gain_percent": 164.5,
  "steps_to_best_reduction_percent": 66.7
}

C04: Handoff Context Retention Comparison

Input:

{
  "required_context": [
    "task_pool",
    "pheromone",
    "shared_state",
    "convergence",
    "analysis: ready"
  ],
  "traditional_payload": "final report",
  "swarm_payload": "facts: task_pool pheromone shared_state convergence; analysis: ready"
}

Actual output:

{
  "traditional_retained_ratio": 0.0,
  "swarm_retained_ratio": 1.0
}

Full Benchmark Output

{
  "standard": "swarm-vs-traditional-deterministic-benchmark-v1",
  "status": "PASS",
  "overall_normalized_score": {
    "traditional": 0.1958,
    "swarm": 0.9175,
    "relative_gain_percent": 368.5,
    "ratio": 4.69,
    "note": "This aggregate is a deterministic academic benchmark over four selected swarm properties, not a universal production claim."
  }
}

Verification Boundary

This report proves deterministic behavior for the four selected swarm properties: fault isolation, emergence, pheromone coordination, and handoff continuity. It does not prove all production-scale properties, large-cluster performance, malicious-agent resistance, or certified compatibility with the external LangGraph package.