diff --git a/docs/integration/runtime-contract.md b/docs/integration/runtime-contract.md index e2bfd3e..ff4148c 100644 --- a/docs/integration/runtime-contract.md +++ b/docs/integration/runtime-contract.md @@ -66,8 +66,9 @@ create 响应 `data`：`deployment_id`、`runtime_deployment_id`、`manager_depl **拉起后端**（`AGENT_LAUNCH_BACKEND`，fail-soft——拉起失败不影响 create）： - `none`（默认）：不自动拉起，agent 由外部供给（保留 CI/e2e 与「外部拉起」部署）； -- `subprocess`：本地起 `python -m agent.main`（dev）； -- `command`：执行部署注入的模板 `AGENT_LAUNCH_CMD`（env 经进程环境传入，模板包装真实拉起如 kubectl/pod-create；**密钥不上 argv**）。 +- `subprocess`：本地起 `python -m agent.main`（**仅 dev**——进程在编排器 pod 内，无隔离/限额）； +- `command`：执行部署注入的模板 `AGENT_LAUNCH_CMD`（env 经进程环境传入，**密钥不上 argv**）； +- **`kubernetes`（生产）**：**每 agent 一个 Pod**，带资源 requests/limits + 标签（`heicode-swarm-id`/`heicode-user-id`，供 GC/teardown）。模型 key 经**每-swarm k8s Secret**（`build_secret_manifest`，via stdin 应用）由 Pod `secretKeyRef` 引用——**绝不**内联进 PodSpec env（否则进 etcd/`kubectl get -o yaml`）。停止时按标签 `kubectl delete pod,secret`。**前置**：编排器镜像含 `kubectl` + 一个对 `AGENT_POD_NAMESPACE` 有 pod/secret RBAC 的 ServiceAccount；`ORCHESTRATOR_URL` = **集群内 Service DNS**（如 `ws://swarm-orchestrator..svc.cluster.local:8000`）；NetworkPolicy 放行出网到 HM `/v1` + git。**硬化替代**：用 azkv CSI SecretProviderClass 让 Pod 直接挂载密钥（编排器全程不碰明文），见 security-boundary §6。config：`AGENT_POD_IMAGE`/`AGENT_POD_NAMESPACE`/`AGENT_POD_SERVICE_ACCOUNT`/`AGENT_POD_CPU|MEM_REQUEST|LIMIT`。 **限额（Swarm 执行）**：池大小 `AGENT_LAUNCH_POOL_SIZE`（默认 3）；**实际拉起数 = min(池大小, `MAX_AGENTS_PER_USER` − 该用户已连接数)**——Swarm 在**拉起时**限额，并在 **agent 注册时**兜底硬拒（security-boundary §6 / PR#32）。 diff --git a/docs/integration/security-boundary.md b/docs/integration/security-boundary.md index 429dd5a..122fd61 100644 --- a/docs/integration/security-boundary.md +++ b/docs/integration/security-boundary.md @@ -59,7 +59,8 @@ - **Swarm 模型**：Agent 主动出站连编排器 WebSocket（`/ws/{agent_id}`），**不**对公网暴露每 Agent 子域名。AM 单 Agent 模型里的「客户端↔agent 直连 + `AGENT_ACCESS_TOKEN` 本地校验」**不适用于** swarm（无直连回路）。 - 服务间鉴权：HM→Swarm 用 `AGENT_RUNTIME_SERVICE_TOKEN`（Bearer）；回调 HMAC 签名。 - **每用户并发 Agent 配额**：一个 `user_id` 同时连接的 Agent 数上限为 `MAX_AGENTS_PER_USER`（env，默认 10）。注册（WS `register` 消息携带 `user_id`）超额即被拒绝（回 `registration_rejected` 并关闭，code 1008），断开后释放名额。归因主轴仍为 `user.id`/`channelId`。未带 `user_id` 的 Agent 为 unbound，不计入该配额。实现：`ConnectionManager.can_bind_user/bind_user/unbind` + 注册处强制；测试 `scripts/test-max-agents-per-user.py`。 -- **Swarm 拉起 agent + 服务端解析 key（team 决议，runtime-contract §3.3）**：由 **Swarm 运行时**（`orchestrator/agent_launcher.py`）拉起专家 agent 池（拉起数 `min(池大小, MAX_AGENTS_PER_USER − 已连)`，与上面的注册兜底一致）。模型 key 由 **Swarm 从 `billing_context.secret_ref`（`azkv://`）服务端解析**后注入被拉起 agent 的 env——**不入** create 请求体 / 回调 / 日志 / argv（`command` 后端的密钥经进程 env 传入，不上命令行）。azkv 真实解析为部署侧 SecretResolver；dev/CI 用 `HEICODE_SECRET_`。解析不到即 keyless 启动并明确报错（不伪造）。 +- **Swarm 拉起 agent + 服务端解析 key（team 决议，runtime-contract §3.3）**：由 **Swarm 运行时**（`orchestrator/agent_launcher.py`）拉起专家 agent 池（拉起数 `min(池大小, MAX_AGENTS_PER_USER − 已连)`，与上面的注册兜底一致）。模型 key 由 **Swarm 从 `billing_context.secret_ref`（`azkv://`）服务端解析**后注入被拉起 agent——**不入** create 请求体 / 回调 / 日志 / argv。azkv 真实解析为部署侧 SecretResolver；dev/CI 用 `HEICODE_SECRET_`。解析不到即 keyless 启动并明确报错（不伪造）。 +- **K8s pod 边界（`AGENT_LAUNCH_BACKEND=kubernetes`，生产）**：**每 agent 一个 Pod**，带 CPU/内存 requests+limits、标签（`heicode-swarm-id`/`heicode-user-id`）。模型 key 经**每-swarm k8s Secret**（manifest via stdin 应用）由 Pod `secretKeyRef` 引用，**绝不内联进 PodSpec env**（否则暴露于 etcd / `kubectl get pod -o yaml`）。编排器需 `kubectl` + 一个仅对 `AGENT_POD_NAMESPACE` 有 pod/secret 权限的 **ServiceAccount（最小 RBAC）**；Pod 出网由 NetworkPolicy 限定到 HM `/v1` + git；停止/删除按标签 `kubectl delete pod,secret`。**更硬化（推荐 Infra 评估）**：azkv **CSI SecretProviderClass** 让 Pod 直接从 Key Vault 挂载密钥，编排器**全程不接触明文**。代码执行沙箱仍按 §8.1 在 Pod 层强制。 - 🟡 待接入：多租户运行时隔离（命名空间/网络/配额）由 Agent 平台（AKS Workload Identity）承载，非本仓编排器；归因主轴为 `user.id`/`channelId`（见 `usage-billing-schema.md`），不引入 tenant 概念。 ## 7. 外部 API 与传输 diff --git a/orchestrator/agent_launcher.py b/orchestrator/agent_launcher.py index e9521a1..a684474 100644 --- a/orchestrator/agent_launcher.py +++ b/orchestrator/agent_launcher.py @@ -13,6 +13,12 @@ Pluggable backend (`AGENT_LAUNCH_BACKEND`): - ``subprocess`` : spawn ``python -m agent.main`` per agent (local/dev). - ``command`` : run a deployment-provided template ``AGENT_LAUNCH_CMD`` per agent (the env is passed through; the template wraps the real spawn, e.g. a kubectl/pod-create). + - ``kubernetes`` : **one Pod per agent** (production). Pods carry resource limits + labels + (`heicode-swarm-id`/`heicode-user-id`) for GC; the model key is delivered via a + per-swarm k8s **Secret** referenced by the pods (never inline PodSpec env, so it + is not exposed in etcd/`kubectl get -o yaml`/argv). Teardown deletes pods+secret + by label. Needs kubectl + a ServiceAccount with pod/secret RBAC in + AGENT_POD_NAMESPACE; ORCHESTRATOR_URL = in-cluster Service DNS. Secret handling: the model key (per-user ``sk-``) is **resolved here from ``billing_context.secret_ref`` (azkv://)** and injected into the launched agent's env *server-side* @@ -25,6 +31,7 @@ it. Pure planning (`plan_launch_specs`) is separated from side-effecting launch from __future__ import annotations import asyncio +import json import logging import os import shlex @@ -169,6 +176,13 @@ async def launch(specs: List[AgentLaunchSpec], *, swarm_id: str) -> List[str]: backend = launch_backend() if backend == "none" or not specs: return [] + if backend == "kubernetes": + # Pod-per-agent (production). Key goes into a per-swarm k8s Secret (referenced by the pods), + # NOT inline PodSpec env — so it never lands in etcd/`kubectl get pod -o yaml`/argv. + launched = await _launch_kubernetes_pool(specs, swarm_id) + if launched: + logger.info("launched %d agent pod(s) for swarm %s (backend=kubernetes)", len(launched), swarm_id) + return launched launched: List[str] = [] for spec in specs: try: @@ -220,15 +234,141 @@ async def _launch_command(spec: AgentLaunchSpec) -> None: await proc.wait() # the template is a quick spawn-wrapper (e.g. kubectl apply), not the agent itself +# ── Kubernetes backend ──────────────────────────────────────────────────────── +# Production runs agents as **one Pod per agent** in the swarm cluster. The model key is delivered +# via a per-swarm k8s Secret referenced by the pods (NOT inline PodSpec env → not exposed in +# etcd/`kubectl get -o yaml`/argv). Requires: kubectl in the orchestrator image + a ServiceAccount +# with RBAC to create/delete pods+secrets in AGENT_POD_NAMESPACE; ORCHESTRATOR_URL must be the +# in-cluster Service DNS (e.g. ws://swarm-orchestrator..svc.cluster.local:8000). NetworkPolicy +# egress to HM /v1 + git. Hardened alternative: mount the azkv secret via a CSI SecretProviderClass +# (orchestrator never touches plaintext) — deployment/Infra option, see security-boundary §6. +_k8s_swarms: set = set() + + +def pod_image() -> str: + return os.getenv("AGENT_POD_IMAGE", "heicode/swarm-agent:latest") + + +def pod_namespace() -> str: + return os.getenv("AGENT_POD_NAMESPACE", "heicode-swarm") + + +def pod_resources() -> Dict[str, Any]: + return { + "requests": {"cpu": os.getenv("AGENT_POD_CPU_REQUEST", "250m"), + "memory": os.getenv("AGENT_POD_MEM_REQUEST", "256Mi")}, + "limits": {"cpu": os.getenv("AGENT_POD_CPU_LIMIT", "1"), + "memory": os.getenv("AGENT_POD_MEM_LIMIT", "1Gi")}, + } + + +def _k8s_labels(spec: AgentLaunchSpec, swarm_id: str) -> Dict[str, str]: + labels = {"app": "heicode-swarm-agent", "heicode-swarm-id": swarm_id} + uid = spec.env.get("HEICODE_USER_ID") + if uid: + labels["heicode-user-id"] = uid + return labels + + +def build_secret_manifest(*, key_value: str, namespace: str, name: str, labels: Dict[str, str]) -> Dict[str, Any]: + """k8s Secret carrying the per-user model key (applied via stdin → value never on argv).""" + return { + "apiVersion": "v1", "kind": "Secret", "type": "Opaque", + "metadata": {"name": name, "namespace": namespace, "labels": labels}, + "stringData": {"OPENAI_API_KEY": key_value}, + } + + +def build_pod_manifest(spec: AgentLaunchSpec, *, namespace: str, swarm_id: str, + image: Optional[str] = None, secret_name: Optional[str] = None, + service_account: Optional[str] = None, + resources: Optional[Dict[str, Any]] = None) -> Dict[str, Any]: + """One agent Pod. Non-sensitive env inline; `OPENAI_API_KEY` via `secretKeyRef` (never inline).""" + env_list: List[Dict[str, Any]] = [] + for k, v in spec.env.items(): + if k == "OPENAI_API_KEY": + if secret_name: # reference the Secret; never inline the key into the PodSpec + env_list.append({"name": k, "valueFrom": {"secretKeyRef": {"name": secret_name, "key": "OPENAI_API_KEY"}}}) + else: + env_list.append({"name": k, "value": v}) + pod_spec: Dict[str, Any] = { + "restartPolicy": "OnFailure", + "containers": [{ + "name": "agent", + "image": image or pod_image(), + "command": ["python", "-m", "agent.main"], + "env": env_list, + "resources": resources or pod_resources(), + }], + } + if service_account: + pod_spec["serviceAccountName"] = service_account + return { + "apiVersion": "v1", "kind": "Pod", + "metadata": {"name": spec.agent_id, "namespace": namespace, "labels": _k8s_labels(spec, swarm_id)}, + "spec": pod_spec, + } + + +async def _kubectl_apply(manifest: Dict[str, Any]) -> None: + proc = await asyncio.create_subprocess_exec( + "kubectl", "apply", "-f", "-", + stdin=asyncio.subprocess.PIPE, stdout=asyncio.subprocess.PIPE, stderr=asyncio.subprocess.PIPE, + ) + _, err = await proc.communicate(json.dumps(manifest).encode()) + if proc.returncode != 0: + raise RuntimeError(f"kubectl apply failed: {(err or b'').decode()[:200]}") + + +async def _launch_kubernetes_pool(specs: List[AgentLaunchSpec], swarm_id: str) -> List[str]: + ns, sa, res, img = pod_namespace(), pod_service_account(), pod_resources(), pod_image() + labels = _k8s_labels(specs[0], swarm_id) + secret_name = None + key_value = next((s.env.get("OPENAI_API_KEY") for s in specs if s.env.get("OPENAI_API_KEY")), None) + if key_value: + secret_name = f"swarm-agent-key-{swarm_id}" + try: + await _kubectl_apply(build_secret_manifest(key_value=key_value, namespace=ns, name=secret_name, labels=labels)) + except Exception as exc: # without the secret the pods can't get the key; still try keyless (clear error) + logger.warning("k8s secret apply failed for %s: %s", swarm_id, exc) + secret_name = None + launched: List[str] = [] + for spec in specs: + try: + await _kubectl_apply(build_pod_manifest(spec, namespace=ns, swarm_id=swarm_id, image=img, + secret_name=secret_name, service_account=sa, resources=res)) + launched.append(spec.agent_id) + except Exception as exc: + logger.warning("k8s agent pod apply failed for %s: %s", spec.agent_id, exc) + if launched: + _k8s_swarms.add(swarm_id) + return launched + + +def pod_service_account() -> Optional[str]: + return os.getenv("AGENT_POD_SERVICE_ACCOUNT") or None + + async def stop_launched(swarm_id: str) -> int: - """Best-effort stop of subprocess-launched agents for a run (other backends: deployment-managed).""" - procs = _subprocess_agents.pop(swarm_id, []) + """Best-effort teardown of agents launched for a run. subprocess → terminate; kubernetes → + delete pods+secret by label. `command` backend pods are deployment-managed.""" stopped = 0 - for proc in procs: + for proc in _subprocess_agents.pop(swarm_id, []): try: if proc.returncode is None: proc.terminate() stopped += 1 except Exception as exc: logger.warning("failed to stop launched agent proc for %s: %s", swarm_id, exc) + if swarm_id in _k8s_swarms: + _k8s_swarms.discard(swarm_id) + try: + proc = await asyncio.create_subprocess_exec( + "kubectl", "delete", "pod,secret", "-l", f"heicode-swarm-id={swarm_id}", + "-n", pod_namespace(), "--ignore-not-found=true", + stdout=asyncio.subprocess.DEVNULL, stderr=asyncio.subprocess.DEVNULL, + ) + await proc.wait() + except Exception as exc: + logger.warning("k8s teardown failed for %s: %s", swarm_id, exc) return stopped diff --git a/scripts/test-agent-launcher.py b/scripts/test-agent-launcher.py index 6e23aee..6eb1257 100644 --- a/scripts/test-agent-launcher.py +++ b/scripts/test-agent-launcher.py @@ -100,12 +100,51 @@ async def test_backend_none_noop(): os.environ.pop("AGENT_LAUNCH_BACKEND") +def test_k8s_manifests(): + import json + spec = al.AgentLaunchSpec( + agent_id="swarm-abc-agent-1", capabilities="python,general", + env={"ORCHESTRATOR_URL": "ws://orch.svc:8000", "AGENT_ID": "swarm-abc-agent-1", + "AGENT_CAPABILITIES": "python,general", "OPENAI_API_KEY": "sk-secret-xyz", + "OPENAI_API_BASE": "https://hm/v1", "HEICODE_USER_ID": "u-1"}, + ) + # Secret manifest carries the key (applied via stdin, not argv). + sec = al.build_secret_manifest(key_value="sk-secret-xyz", namespace="heicode-swarm", + name="swarm-agent-key-swarm-abc", labels={"app": "heicode-swarm-agent"}) + check("secret kind/type", sec["kind"] == "Secret" and sec["type"] == "Opaque") + check("secret stringData has the key", sec["stringData"]["OPENAI_API_KEY"] == "sk-secret-xyz") + + # Pod manifest references the key via secretKeyRef — NEVER inline. + pod = al.build_pod_manifest(spec, namespace="heicode-swarm", swarm_id="swarm-abc", + image="img:1", secret_name="swarm-agent-key-swarm-abc", + service_account="swarm-agent-sa") + blob = json.dumps(pod) + check("pod kind/name", pod["kind"] == "Pod" and pod["metadata"]["name"] == "swarm-abc-agent-1") + check("pod labels include swarm-id + user-id", + pod["metadata"]["labels"].get("heicode-swarm-id") == "swarm-abc" + and pod["metadata"]["labels"].get("heicode-user-id") == "u-1") + env = {e["name"]: e for e in pod["spec"]["containers"][0]["env"]} + check("OPENAI_API_KEY via secretKeyRef (not inline)", + "value" not in env["OPENAI_API_KEY"] and env["OPENAI_API_KEY"]["valueFrom"]["secretKeyRef"]["name"] == "swarm-agent-key-swarm-abc") + check("raw key value NOT in pod manifest (only in Secret)", "sk-secret-xyz" not in blob) + check("non-secret env inline", env["ORCHESTRATOR_URL"]["value"] == "ws://orch.svc:8000") + check("pod has resource limits + restartPolicy + serviceAccount", + pod["spec"]["containers"][0]["resources"]["limits"] + and pod["spec"]["restartPolicy"] == "OnFailure" + and pod["spec"]["serviceAccountName"] == "swarm-agent-sa") + # No secret_name -> key omitted entirely (never inlined). + pod2 = al.build_pod_manifest(spec, namespace="heicode-swarm", swarm_id="swarm-abc", secret_name=None) + env2 = {e["name"] for e in pod2["spec"]["containers"][0]["env"]} + check("no secret -> OPENAI_API_KEY omitted from pod env", "OPENAI_API_KEY" not in env2) + + def main(): import asyncio test_launch_count() test_plan_specs() test_resolve_model_key() test_command_backend_build() + test_k8s_manifests() asyncio.run(test_backend_none_noop()) print() if failures: