gongzhiyongandOmX
d632fd9f64
Extend the Agent standard matrix with a report-audit scenario so model input, output, handoff, and secret-safety evidence are tested instead of remaining narrative-only. Constraint: The user requested another test pass and expanded Agent/swarm testing scenarios under docs/. Rejected: Treating the model I/O report as untested documentation | it would leave the handoff and input/output evidence unguarded. Confidence: high Scope-risk: moderate Directive: Keep model I/O reports under docs/ and redact secret-shaped values during export. Tested: .venv/bin/python -u -B examples/run_standard_scenario_acceptance.py; .venv/bin/python -B -m unittest discover -s tests; .venv/bin/python -B -m py_compile swarm_minimal/*.py examples/*.py tests/*.py; .venv/bin/python -u -B examples/run_academic_standard_evaluation.py; git diff --check; docs secret-pattern scan. Not-tested: Large-scale concurrent 3/5/7 worker load and external browser rendering were not run. Co-authored-by: OmX <omx@oh-my-codex.dev>
3.3 KiB
3.3 KiB
Standard Test Matrix
This file defines the stricter acceptance shape for the standalone minimal swarm. It separates deterministic checks from live Azure/NewAPI checks so a passing result means more than "one long run completed".
Layers
| Layer | Purpose | Command |
|---|---|---|
| Static | Python syntax and importability | ./.venv/bin/python -B -m py_compile swarm_minimal/*.py examples/*.py tests/*.py |
| Unit | Existing closed-loop and NewAPI mock behavior | ./.venv/bin/python -B -m unittest discover -s tests |
| Deterministic scenarios | No-network Given/When/Then cases for continuity, policy, final scoring, and failure injection | ./.venv/bin/python -B -m unittest tests.test_standard_scenarios |
| Academic Markov fit | No-network checks for Markov-style transition behavior and non-MDP limitations | ./.venv/bin/python -B -m unittest tests.test_markov_process_properties |
| Live integration | Real NewAPI + Azure PostgreSQL/Redis/Blob continuous reasoning chain | ./.venv/bin/python -u -B examples/run_continuous_reasoning_acceptance.py |
| Report audit | Human-auditable model I/O report with scenario, input, output, handoff and secret-safety evidence | ./.venv/bin/python -B -m unittest tests.test_model_io_report_audit |
Required Scenarios
| ID | Scenario | Given | When | Then |
|---|---|---|---|---|
| S01 | Syntax/import sanity | All prototype Python files | Compile every module | No syntax/import compile errors |
| S02 | Unit regression | Mock NewAPI and in-memory store | Run all unit tests | All tests pass |
| S03 | Chain continuity | Seven deterministic step outputs | Remove one previous-step link | Continuity checker fails the broken chain |
| S04 | Dependency boundary | Text mentions NATS/Cosmos | Use negative and positive dependency wording | Negative wording passes; required dependency wording fails |
| S05 | Final convergence scoring | STEP-06 and STEP-07 both produce rich outputs | Score both outputs | STEP-07 is strictly selected as final convergence |
| S06 | Failure injection | One deterministic worker raises | Coordinator runs to convergence | Failed task is marked failed, pheromone is negative, completed task still converges |
| S07 | Live resource closure | Real Azure/NewAPI env is loaded | Run seven-step continuous reasoning | PostgreSQL, Redis, Blob, stream events, model discovery, and final artifact all pass |
| S08 | Model I/O report audit | Generated model/Agnet I/O report | Audit scenario coverage, task input/output, handoff evidence, and obvious secret patterns | Report is human-auditable and contains no obvious secret values |
| M01 | Markov-style claim transition | Same current tasks and pheromone values but different prior paths | Claim next task | The same high-pheromone task is claimed |
| M02 | Markov-style score transition | Same current task, agent, output, and score but different prior paths | Complete task | Task status, output, score, and pheromone update match |
| M03 | Formal MDP limitation | Project metadata is evaluated | Assess Markov fit | Prototype is classified as Markov-style state machine, not a formal Markov process or MDP |
Stop Rule
The standard acceptance result is PASS only if every scenario passes. Any FAIL must include the scenario ID, the failed assertion, and enough evidence to reproduce it without printing secrets.