Merge branch 'main' into fix/orphan-agent-lifecycle
This commit is contained in:
@@ -8,7 +8,7 @@
|
||||
> | 能力 | 状态 |
|
||||
> |---|---|
|
||||
> | 任务分解(规划回退)、能力路由派发、专家执行(OpenAI 兼容)、评审/重做循环、结果汇总、peer 协作消息路由、WebSocket Agent 协议、Redis 持久化、Prometheus 指标 | ✅ 已实现(仓内可运行) |
|
||||
> | Manager↔Runtime 生命周期契约(对齐 `heicode-am-contract`)、HMAC 签名回调事件 envelope、稳定的 `deployment_id`/`workflow_id`/`trace_id` | 🟡 待接入(见 `docs/integration/runtime-contract.md`;事件 envelope 与签名待 Manager 对齐) |
|
||||
> | Manager↔Runtime 生命周期契约(对齐 `heicode-am-contract`)、HMAC 签名回调事件 envelope、稳定的 `deployment_id`/`workflow_id`/`trace_id` | 🟡 机制已实现,待主链路联调验收(生命周期接口见 `docs/integration/runtime-contract.md §3`;envelope/HMAC 签名机制已实现且规范与 HM 对齐,见 `docs/integration/event-schema.md §1`,待 Manager 端验签端到端联调验收) |
|
||||
> | 统一 usage/计费聚合(归属 NewAPI)、审计/lineage trace、前端事件 API、统一 Agent registry/scheduling、统一 secret/workspace/tool/MCP/tenant 安全边界 | 🟡 待接入(需与 Billing / Audit / Frontend / Infra / Security Team 对齐) |
|
||||
> | Benchmark 自证(`Benchmark_Agent`、`S_swarm`、`G_E`、`G_E,c`、治理/协作/通信/鲁棒性指标、baseline 对比、telemetry 架构) | 🟡 采集器已落地(`benchmark/`:指标公式 `metrics.py`、活体 run 采集 `collectors/`、4 基线 runner `runners/`、`baselines.compare` 算 `G_E`/`G_E,c`、自证合流 `selfcert_collector`、telemetry 导出 Cosmos/Blob `export/`);**评分标准待定**——`O`(可观测性)公式、`BASE_COEFFICIENTS`、`S_gain` 阈值需累积真实用户使用数据后经验标定(长期工单)。诚实原则:缺真实输入的量返回 `NaN`、不伪造分值(标准见 `docs/benchmark/`) |
|
||||
>
|
||||
|
||||
@@ -14,7 +14,7 @@
|
||||
|
||||
- Swarm → HM:`POST {callback.url}`(create 时下发)。
|
||||
- 鉴权二选一:服务令牌 `X-Agent-Service-Token`(或 `Authorization: Bearer`),或 HMAC 签名。
|
||||
- **HMAC 签名(已实现,与 HM 一致)**:
|
||||
- **HMAC 签名(机制已实现,规范与 HM 对齐;待主链路端到端联调验收)**:
|
||||
- 头:`X-Agent-Timestamp`(Unix 毫秒)、`X-Agent-Signature`、`X-Agent-Event-Id`、`X-Correlation-ID`(均含 `X-Agnet-` 兼容别名)。
|
||||
- 规范串:`canonical = f"{timestamp}.{event_id}.{raw_body}"`。
|
||||
- 签名:`X-Agent-Signature = "sha256=" + hex(HMAC_SHA256(secret, canonical))`。
|
||||
|
||||
@@ -85,10 +85,14 @@ Swarm 拉起每个专家 agent 时注入以下进程环境变量(`agent/main.p
|
||||
| `AGENT_ID` | agent 实例 id(唯一) | **Swarm launcher** |
|
||||
| `AGENT_CAPABILITIES` | 能力集合(逗号分隔) | **Swarm launcher**(`AGENT_LAUNCH_CAPABILITIES` 池策略) |
|
||||
| `HEICODE_USER_ID` | 发起用户;用于每用户并发上限 `MAX_AGENTS_PER_USER`(注册时强制) | **Swarm 从 create `x-user-id` 透传** |
|
||||
| `WORKSPACE_DIR` / `GIT_REPO_URL` | 可选:工作区 / 代码仓 | 资源授权 |
|
||||
| `WORKSPACE_DIR` | 可选:工作区根 | 部署(`WORKSPACE_DIR`) |
|
||||
| `GIT_REPO_URL` | 代码仓克隆地址;agent 有此值才 clone | **Swarm 从 git `resource_grant.metadata.repo_url` 解析** |
|
||||
| `GIT_USERNAME` / `GIT_PASSWORD` | git HTTPS 凭据(注入 clone URL) | **Swarm 从 git `resource_grant.secret_ref`(`azkv://`)解析** |
|
||||
| `GIT_BASE_BRANCH` | 可选:基线分支(默认 `main`) | git grant `metadata.base_branch` |
|
||||
|
||||
**约束**:
|
||||
- 模型 key 由 **Swarm 服务端从 `billing_context.secret_ref`(`azkv://`)解析**(`resolve_model_key`:override → azkv(部署 SecretResolver / dev `HEICODE_SECRET_<name>`)→ 编排器 `OPENAI_API_KEY` 兜底)后注入被拉起 agent 的 env。key **不入** create 请求体 / 回调 / 日志 / argv(满足 §3.1);解析不到则 agent keyless 启动并明确报错(不伪造)。`billing_context` 的 `newapi_user_ref`/`quota_ref` 仍仅为归因元数据。见 [usage-billing-schema.md §2](./usage-billing-schema.md)。
|
||||
- **git 仓库绑定**(agent_swarm#63 / HM #92)由 **Swarm 服务端从 `resource_grants` 里 `resource_type=git` 的 grant 解析**(`resolve_git_grant`):仓库地址 = `grant.metadata.repo_url`(非密文,inline 注入 `GIT_REPO_URL`);git 凭据 = `grant.secret_ref`(`azkv://`,同模型 key 路径解析 → 注入 `GIT_USERNAME`/`GIT_PASSWORD`)。git 凭据**不入** create 请求体 / 回调 / 日志 / argv;k8s 后端 `GIT_PASSWORD` 经 per-swarm Secret 的 `secretKeyRef` 注入(绝不内联 PodSpec)。grant 有 repo 无凭据时仍注入 `GIT_REPO_URL`(公有仓可 clone;私有仓 clone 报错,不伪造)。git KV secret 值约定为 JSON `{"git_username","git_password"}`(接受 `git_token`/`token` 形式 + 裸 token),**待 HM #92 对齐**。同一用户 agent 池注入同一 git grant。
|
||||
- 同一用户的专家 agent 注入**同一把** `sk-` + `OPENAI_API_BASE=HM/v1`,计费归一到发起用户;`task_executor` 随模型请求带 `X-Agent-*` 归因头供 HM/NewAPI 关联。token 吊销走 §3.3.1 A.5 的 `swarm.pool_terminated` 握手(per-user,全部 run stop 后吊销)。
|
||||
- **运行时事件前置**:只有 (i) HM 真把 create 派发到编排器(`SWARM_RUNTIME_ENABLED=true`,非 manager-local 适配器)且 (ii) Swarm 已按上表拉起 agent 并连入,编排器才回推 `task.*`/`swarm.*` 运行时事件(否则 events feed 仅有 HM 控制面 `deployment.status_changed`,见 #39)。
|
||||
|
||||
|
||||
@@ -60,7 +60,8 @@
|
||||
- 服务间鉴权:HM→Swarm 用 `AGENT_RUNTIME_SERVICE_TOKEN`(Bearer);回调 HMAC 签名。
|
||||
- **每用户并发 Agent 配额**:一个 `user_id` 同时连接的 Agent 数上限为 `MAX_AGENTS_PER_USER`(env,默认 10)。注册(WS `register` 消息携带 `user_id`)超额即被拒绝(回 `registration_rejected` 并关闭,code 1008),断开后释放名额。归因主轴仍为 `user.id`/`channelId`。未带 `user_id` 的 Agent 为 unbound,不计入该配额。实现:`ConnectionManager.can_bind_user/bind_user/unbind` + 注册处强制;测试 `scripts/test-max-agents-per-user.py`。
|
||||
- **Swarm 拉起 agent + 服务端解析 key(team 决议,runtime-contract §3.3)**:由 **Swarm 运行时**(`orchestrator/agent_launcher.py`)拉起专家 agent 池(拉起数 `min(池大小, MAX_AGENTS_PER_USER − 已连)`,与上面的注册兜底一致)。模型 key 由 **Swarm 从 `billing_context.secret_ref`(`azkv://`)服务端解析**后注入被拉起 agent——**不入** create 请求体 / 回调 / 日志 / argv。azkv 真实解析为部署侧 SecretResolver;dev/CI 用 `HEICODE_SECRET_<name>`。解析不到即 keyless 启动并明确报错(不伪造)。
|
||||
- **K8s pod 边界(`AGENT_LAUNCH_BACKEND=kubernetes`,生产)**:**每 agent 一个 Pod**,带 CPU/内存 requests+limits、标签(`heicode-swarm-id`/`heicode-user-id`)。模型 key 经**每-swarm k8s Secret**(manifest via stdin 应用)由 Pod `secretKeyRef` 引用,**绝不内联进 PodSpec env**(否则暴露于 etcd / `kubectl get pod -o yaml`)。编排器需 `kubectl` + 一个仅对 `AGENT_POD_NAMESPACE` 有 pod/secret 权限的 **ServiceAccount(最小 RBAC)**;Pod 出网由 NetworkPolicy 限定到 HM `/v1` + git;停止/删除按标签 `kubectl delete pod,secret`。**更硬化(推荐 Infra 评估)**:azkv **CSI SecretProviderClass** 让 Pod 直接从 Key Vault 挂载密钥,编排器**全程不接触明文**。代码执行沙箱仍按 §8.1 在 Pod 层强制。
|
||||
- **Git 仓库绑定凭据服务端解析(agent_swarm#63 / HM #92)**:git 绑定经 `resource_grants` 里 `resource_type=git` 的 grant 下发。Swarm 同模型 key 方式**服务端解析**(`resolve_git_grant`):`metadata.repo_url`(非密文)→ `GIT_REPO_URL`;`secret_ref`(`azkv://` git 凭据引用)→ `GIT_USERNAME`/`GIT_PASSWORD`。git 凭据**不入** create 请求体 / 回调 / 日志 / argv;明文 git token/密码不得进 create 体(`_reject_plaintext_secrets` 已覆盖 `resource_grants`)。git KV secret 值约定 JSON `{"git_username","git_password"}`(接受 token 形式),**待 HM #92 对齐**。
|
||||
- **K8s pod 边界(`AGENT_LAUNCH_BACKEND=kubernetes`,生产)**:**每 agent 一个 Pod**,带 CPU/内存 requests+limits、标签(`heicode-swarm-id`/`heicode-user-id`)。模型 key 与 **git 凭据(`GIT_PASSWORD`)** 均经**每-swarm k8s Secret**(manifest via stdin 应用)由 Pod `secretKeyRef` 引用,**绝不内联进 PodSpec env**(否则暴露于 etcd / `kubectl get pod -o yaml`);`GIT_REPO_URL`/`GIT_USERNAME` 为非密文,inline 注入。编排器需 `kubectl` + 一个仅对 `AGENT_POD_NAMESPACE` 有 pod/secret 权限的 **ServiceAccount(最小 RBAC)**;Pod 出网由 NetworkPolicy 限定到 HM `/v1` + git;停止/删除按标签 `kubectl delete pod,secret`。**更硬化(推荐 Infra 评估)**:azkv **CSI SecretProviderClass** 让 Pod 直接从 Key Vault 挂载密钥,编排器**全程不接触明文**。代码执行沙箱仍按 §8.1 在 Pod 层强制。
|
||||
- 🟡 待接入:多租户运行时隔离(命名空间/网络/配额)由 Agent 平台(AKS Workload Identity)承载,非本仓编排器;归因主轴为 `user.id`/`channelId`(见 `usage-billing-schema.md`),不引入 tenant 概念。
|
||||
|
||||
## 7. 外部 API 与传输
|
||||
|
||||
+150
-20
@@ -25,6 +25,14 @@ Secret handling: the model key (per-user ``sk-``) is **resolved here from
|
||||
— it never travels in the create request body (runtime-contract §3.1) or in events/logs. Real
|
||||
Azure Key Vault resolution is a deployment adapter (`SECRET_RESOLVER`); dev/CI use an env map.
|
||||
|
||||
Git binding (agent_swarm#63 / HM #92): a swarm-run needs a repo to operate on. HM ships the git
|
||||
binding as a ``resource_type=git`` entry in ``resource_grants`` — ``metadata.repo_url`` (clone URL,
|
||||
non-secret) + ``secret_ref`` (``azkv://`` ref to the git credential, never plaintext). The launcher
|
||||
resolves it here the same server-side way as the model key (``resolve_git_grant``) and injects
|
||||
``GIT_REPO_URL`` (+ ``GIT_USERNAME``/``GIT_PASSWORD`` when a credential resolves) into the agent env;
|
||||
the agent (`agent/main.py` + `agent/git_operations.py`) clones from those vars. Without this the
|
||||
agent has no repo URL → cannot clone → the "需绑定 git 仓库才能使用" symptom in HM #92.
|
||||
|
||||
Everything is **fail-soft**: a launch error is logged and swallowed so run creation never fails on
|
||||
it. Pure planning (`plan_launch_specs`) is separated from side-effecting launch for testability.
|
||||
"""
|
||||
@@ -192,13 +200,14 @@ def _parse_azkv_ref(secret_ref: str):
|
||||
return vault_url, name, version
|
||||
|
||||
|
||||
def _resolve_from_keyvault(secret_ref: str) -> Optional[str]:
|
||||
"""Read the model key from Azure Key Vault using the pod's workload identity.
|
||||
def _read_kv_secret_value(secret_ref: str) -> Optional[str]:
|
||||
"""Read the **raw** secret value string from Azure Key Vault using the pod's workload identity.
|
||||
|
||||
`DefaultAzureCredential` picks up the federated token the AKS workload-identity webhook injects
|
||||
(see `_azkv_enabled`). Lazy-imports the azure SDK so dev/CI without it are unaffected. Returns
|
||||
None on ANY failure (not enabled / unparseable ref / SDK missing / no credential / network /
|
||||
secret absent) — never fabricates, never raises.
|
||||
Shared by the model-key and git-credential resolvers (each applies its own extractor to the raw
|
||||
value). `DefaultAzureCredential` picks up the federated token the AKS workload-identity webhook
|
||||
injects (see `_azkv_enabled`). Lazy-imports the azure SDK so dev/CI without it are unaffected.
|
||||
Returns None on ANY failure (not enabled / unparseable ref / SDK missing / no credential /
|
||||
network / secret absent) — never fabricates, never raises.
|
||||
"""
|
||||
if not _azkv_enabled():
|
||||
return None
|
||||
@@ -216,12 +225,114 @@ def _resolve_from_keyvault(secret_ref: str) -> Optional[str]:
|
||||
try:
|
||||
client = SecretClient(vault_url=vault_url, credential=DefaultAzureCredential())
|
||||
secret = client.get_secret(name, version) if version else client.get_secret(name)
|
||||
return _extract_model_key(secret.value)
|
||||
return secret.value
|
||||
except Exception as exc: # no credential / RBAC / network / missing secret
|
||||
logger.warning("azkv resolver: failed to read secret '%s' from %s: %s", name, vault_url, exc)
|
||||
return None
|
||||
|
||||
|
||||
def _resolve_from_keyvault(secret_ref: str) -> Optional[str]:
|
||||
"""Read the model key from Azure Key Vault (raw value → `_extract_model_key`). None on failure."""
|
||||
raw = _read_kv_secret_value(secret_ref)
|
||||
return _extract_model_key(raw) if raw is not None else None
|
||||
|
||||
|
||||
# ── Git binding resolution (agent_swarm#63 / HM #92) ───────────────────────────
|
||||
#: Default git username when the KV secret carries only a token/PAT (GitHub HTTPS convention: any
|
||||
#: non-empty username + PAT-as-password works; ``x-access-token`` is the documented placeholder).
|
||||
DEFAULT_GIT_USERNAME = "x-access-token"
|
||||
|
||||
|
||||
def _extract_git_credentials(secret_value: str) -> Optional[Dict[str, str]]:
|
||||
"""Pull git credentials out of a resolved KV secret value.
|
||||
|
||||
Contract (mirrors the model-key JSON convention; pending HM #92 alignment): the git KV secret is
|
||||
JSON ``{"git_username": "...", "git_password": "..."}``. Accepted aliases:
|
||||
``username``/``password``, and a token form ``git_token``/``token`` (username defaults to
|
||||
``x-access-token``). A bare string value is treated as a token. Returns
|
||||
``{"username","password"}`` or None (never fabricates — missing password → None).
|
||||
"""
|
||||
value = (secret_value or "").strip()
|
||||
if not value:
|
||||
return None
|
||||
if value.startswith("{"):
|
||||
try:
|
||||
obj = json.loads(value)
|
||||
except ValueError:
|
||||
return None
|
||||
if not isinstance(obj, dict):
|
||||
return None
|
||||
username = (obj.get("git_username") or obj.get("username") or "").strip() or DEFAULT_GIT_USERNAME
|
||||
password = (obj.get("git_password") or obj.get("git_token")
|
||||
or obj.get("password") or obj.get("token") or "").strip()
|
||||
return {"username": username, "password": password} if password else None
|
||||
return {"username": DEFAULT_GIT_USERNAME, "password": value} # bare token string
|
||||
|
||||
|
||||
def _resolve_git_secret_ref(secret_ref: str) -> Optional[Dict[str, str]]:
|
||||
"""Resolve an azkv:// git ``secret_ref`` to ``{"username","password"}``.
|
||||
|
||||
Same source order as the model key: dev/CI env map ``HEICODE_SECRET_<name>`` (hermetic) →
|
||||
production Azure Key Vault via workload identity. None when unavailable — never fabricated.
|
||||
"""
|
||||
name = secret_ref.rstrip("/").rsplit("/", 1)[-1]
|
||||
raw = os.getenv(f"HEICODE_SECRET_{name}")
|
||||
if raw is None:
|
||||
raw = _read_kv_secret_value(secret_ref)
|
||||
return _extract_git_credentials(raw) if raw is not None else None
|
||||
|
||||
|
||||
def _iter_resource_grants(body: Dict[str, Any]):
|
||||
"""Yield every resource grant in the create body (top-level + per-agent), in declaration order."""
|
||||
yield from (body.get("resource_grants") or [])
|
||||
plan = body.get("orchestration_plan") or {}
|
||||
for agent in (plan.get("agents") or []):
|
||||
yield from (agent.get("resource_grants") or [])
|
||||
for agent in (body.get("agents") or []):
|
||||
yield from (agent.get("resource_grants") or [])
|
||||
|
||||
|
||||
def _first_git_grant(body: Dict[str, Any]) -> Optional[Dict[str, Any]]:
|
||||
"""First grant with ``resource_type`` (or ``type``) == ``git``, or None."""
|
||||
for grant in _iter_resource_grants(body):
|
||||
if not isinstance(grant, dict):
|
||||
continue
|
||||
if (grant.get("resource_type") or grant.get("type")) == "git":
|
||||
return grant
|
||||
return None
|
||||
|
||||
|
||||
def resolve_git_grant(body: Dict[str, Any]) -> Optional[Dict[str, str]]:
|
||||
"""Resolve the git binding for launched agents — server-side, mirroring `resolve_model_key`.
|
||||
|
||||
Returns the git env to inject (``GIT_REPO_URL`` + optional ``GIT_USERNAME``/``GIT_PASSWORD`` +
|
||||
optional ``GIT_BASE_BRANCH``) or None when there is no git grant / no repo URL. The credential
|
||||
(``secret_ref`` → azkv) is resolved here and never travels in the create body / events / logs.
|
||||
A grant with a repo URL but unresolved credential still yields ``GIT_REPO_URL`` (public-repo
|
||||
clone works; a private repo then fails to clone with a clear git error — never fabricated).
|
||||
"""
|
||||
grant = _first_git_grant(body)
|
||||
if not grant:
|
||||
return None
|
||||
meta = grant.get("metadata") or {}
|
||||
repo_url = (meta.get("repo_url") or grant.get("repo_url") or "").strip()
|
||||
if not repo_url:
|
||||
return None
|
||||
env: Dict[str, str] = {"GIT_REPO_URL": repo_url}
|
||||
branch = (meta.get("base_branch") or meta.get("branch") or "").strip()
|
||||
if branch:
|
||||
env["GIT_BASE_BRANCH"] = branch
|
||||
secret_ref = (grant.get("secret_ref") or grant.get("ref") or "").strip()
|
||||
if secret_ref.startswith("azkv://"):
|
||||
creds = _resolve_git_secret_ref(secret_ref)
|
||||
if creds:
|
||||
env["GIT_USERNAME"] = creds["username"]
|
||||
env["GIT_PASSWORD"] = creds["password"]
|
||||
else:
|
||||
logger.warning("git grant: secret_ref present but unresolved; agent will clone keyless (private repo will fail)")
|
||||
return env
|
||||
|
||||
|
||||
def model_api_base() -> str:
|
||||
return os.getenv("AGENT_OPENAI_API_BASE") or os.getenv("OPENAI_API_BASE") or "https://api.openai.com/v1"
|
||||
|
||||
@@ -421,6 +532,11 @@ async def _launch_command(spec: AgentLaunchSpec) -> None:
|
||||
# (orchestrator never touches plaintext) — deployment/Infra option, see security-boundary §6.
|
||||
_k8s_swarms: set = set()
|
||||
|
||||
#: Env keys that carry secrets — delivered via the per-swarm k8s Secret (`secretKeyRef`), NEVER
|
||||
#: inlined into the PodSpec (would leak into etcd / `kubectl get pod -o yaml`). `GIT_USERNAME` and
|
||||
#: `GIT_REPO_URL` are non-secret and stay inline.
|
||||
SENSITIVE_ENV_KEYS = ("OPENAI_API_KEY", "GIT_PASSWORD")
|
||||
|
||||
|
||||
def pod_image() -> str:
|
||||
return os.getenv("AGENT_POD_IMAGE", "heicode/swarm-agent:latest")
|
||||
@@ -461,25 +577,30 @@ def _k8s_labels(spec: AgentLaunchSpec, swarm_id: str) -> Dict[str, str]:
|
||||
return labels
|
||||
|
||||
|
||||
def build_secret_manifest(*, key_value: str, namespace: str, name: str, labels: Dict[str, str]) -> Dict[str, Any]:
|
||||
"""k8s Secret carrying the per-user model key (applied via stdin → value never on argv)."""
|
||||
def build_secret_manifest(*, namespace: str, name: str, labels: Dict[str, str],
|
||||
secret_data: Dict[str, str]) -> Dict[str, Any]:
|
||||
"""k8s Secret carrying the per-user sensitive env (model key + git password) — applied via stdin
|
||||
so values never hit argv. `secret_data` maps env key → value (only SENSITIVE_ENV_KEYS)."""
|
||||
return {
|
||||
"apiVersion": "v1", "kind": "Secret", "type": "Opaque",
|
||||
"metadata": {"name": name, "namespace": namespace, "labels": labels},
|
||||
"stringData": {"OPENAI_API_KEY": key_value},
|
||||
"stringData": dict(secret_data),
|
||||
}
|
||||
|
||||
|
||||
def build_pod_manifest(spec: AgentLaunchSpec, *, namespace: str, swarm_id: str,
|
||||
image: Optional[str] = None, secret_name: Optional[str] = None,
|
||||
service_account: Optional[str] = None,
|
||||
secret_keys=None, service_account: Optional[str] = None,
|
||||
resources: Optional[Dict[str, Any]] = None) -> Dict[str, Any]:
|
||||
"""One agent Pod. Non-sensitive env inline; `OPENAI_API_KEY` via `secretKeyRef` (never inline)."""
|
||||
"""One agent Pod. Non-sensitive env inline; secret env (`OPENAI_API_KEY`/`GIT_PASSWORD`) via
|
||||
`secretKeyRef` (never inline). `secret_keys` = the keys actually present in the Secret."""
|
||||
keys_in_secret = set(secret_keys or ())
|
||||
env_list: List[Dict[str, Any]] = []
|
||||
for k, v in spec.env.items():
|
||||
if k == "OPENAI_API_KEY":
|
||||
if secret_name: # reference the Secret; never inline the key into the PodSpec
|
||||
env_list.append({"name": k, "valueFrom": {"secretKeyRef": {"name": secret_name, "key": "OPENAI_API_KEY"}}})
|
||||
if k in SENSITIVE_ENV_KEYS:
|
||||
if secret_name and k in keys_in_secret: # reference the Secret; never inline into PodSpec
|
||||
env_list.append({"name": k, "valueFrom": {"secretKeyRef": {"name": secret_name, "key": k}}})
|
||||
# else: omit entirely — a secret value is never inlined
|
||||
else:
|
||||
env_list.append({"name": k, "value": v})
|
||||
pod_spec: Dict[str, Any] = {
|
||||
@@ -521,19 +642,28 @@ async def _launch_kubernetes_pool(specs: List[AgentLaunchSpec], swarm_id: str) -
|
||||
ns, sa, res, img = pod_namespace(), pod_service_account(), pod_resources(), pod_image()
|
||||
labels = _k8s_labels(specs[0], swarm_id)
|
||||
secret_name = None
|
||||
key_value = next((s.env.get("OPENAI_API_KEY") for s in specs if s.env.get("OPENAI_API_KEY")), None)
|
||||
if key_value:
|
||||
# Sensitive env (model key + git password) is per-user/per-run — same across the pool; collect
|
||||
# the first non-empty value of each into one per-swarm Secret.
|
||||
secret_data: Dict[str, str] = {}
|
||||
for skey in SENSITIVE_ENV_KEYS:
|
||||
val = next((s.env.get(skey) for s in specs if s.env.get(skey)), None)
|
||||
if val:
|
||||
secret_data[skey] = val
|
||||
if secret_data:
|
||||
secret_name = f"swarm-agent-key-{swarm_id}"
|
||||
try:
|
||||
await _kubectl_apply(build_secret_manifest(key_value=key_value, namespace=ns, name=secret_name, labels=labels))
|
||||
except Exception as exc: # without the secret the pods can't get the key; still try keyless (clear error)
|
||||
await _kubectl_apply(build_secret_manifest(namespace=ns, name=secret_name, labels=labels, secret_data=secret_data))
|
||||
except Exception as exc: # without the secret the pods can't get the values; still try without (clear error)
|
||||
logger.warning("k8s secret apply failed for %s: %s", swarm_id, exc)
|
||||
secret_name = None
|
||||
secret_data = {}
|
||||
secret_keys = set(secret_data)
|
||||
launched: List[str] = []
|
||||
for spec in specs:
|
||||
try:
|
||||
await _kubectl_apply(build_pod_manifest(spec, namespace=ns, swarm_id=swarm_id, image=img,
|
||||
secret_name=secret_name, service_account=sa, resources=res))
|
||||
secret_name=secret_name, secret_keys=secret_keys,
|
||||
service_account=sa, resources=res))
|
||||
launched.append(spec.agent_id)
|
||||
except Exception as exc:
|
||||
logger.warning("k8s agent pod apply failed for %s: %s", spec.agent_id, exc)
|
||||
|
||||
@@ -1767,10 +1767,12 @@ async def launch_swarm_agents(run, body: Dict[str, Any]) -> None:
|
||||
connected = manager.user_agent_count(user_id) if user_id else 0
|
||||
backend = agent_launcher.launch_backend()
|
||||
model_key = agent_launcher.resolve_model_key(body)
|
||||
git_env = agent_launcher.git_launch_env(body)
|
||||
git_env = agent_launcher.resolve_git_grant(body)
|
||||
info: Dict[str, Any] = {
|
||||
"backend": backend, "planned": 0, "launched": 0, "launched_ids": [],
|
||||
"model_key_resolved": bool(model_key), "repo_bound": bool(git_env.get("GIT_REPO_URL")),
|
||||
"model_key_resolved": bool(model_key),
|
||||
"git_repo_resolved": bool(git_env and git_env.get("GIT_REPO_URL")),
|
||||
"git_creds_resolved": bool(git_env and git_env.get("GIT_PASSWORD")),
|
||||
"note": "",
|
||||
}
|
||||
try:
|
||||
|
||||
@@ -91,55 +91,60 @@ def test_pool_window():
|
||||
def test_plan_specs():
|
||||
run = FakeRun()
|
||||
body = {"orchestration_plan": {"objective": "x"}, "billing_context": {"default_model_id": "gpt-x"}}
|
||||
# 8 connected, cap 10 -> headroom 2, but MIN floor 3 wins (intentional [3,16] hard-floor-first).
|
||||
git_env = {"GIT_REPO_URL": "https://git.example/r.git", "GIT_USERNAME": "x-access-token",
|
||||
"GIT_PASSWORD": "ghp_xyz"}
|
||||
specs = al.plan_launch_specs(run, body, connected_user_agents=8, limit=10, pool_size=3,
|
||||
model_key="sk-test", orchestrator_url="ws://orch:8000", user_id="u-1")
|
||||
check("plan applies MIN floor over cap headroom (8 connected, cap 10 -> 3 not 2)", len(specs) == 3)
|
||||
model_key="sk-test", orchestrator_url="ws://orch:8000", user_id="u-1",
|
||||
git_env=git_env)
|
||||
check("plan caps at limit (8 connected, cap 10 -> launch 2)", len(specs) == 2)
|
||||
s = specs[0]
|
||||
check("spec env has ORCHESTRATOR_URL", s.env.get("ORCHESTRATOR_URL") == "ws://orch:8000")
|
||||
check("spec env has model key (server-side injected)", s.env.get("OPENAI_API_KEY") == "sk-test")
|
||||
check("spec env has model id from billing_context", s.env.get("OPENAI_MODEL") == "gpt-x")
|
||||
check("spec env has HEICODE_USER_ID for per-user cap", s.env.get("HEICODE_USER_ID") == "u-1")
|
||||
check("spec env has GIT_REPO_URL (server-side injected)", s.env.get("GIT_REPO_URL") == "https://git.example/r.git")
|
||||
check("spec env has GIT_USERNAME + GIT_PASSWORD", s.env.get("GIT_USERNAME") == "x-access-token" and s.env.get("GIT_PASSWORD") == "ghp_xyz")
|
||||
check("spec has AGENT_ID + capabilities", bool(s.agent_id) and bool(s.env.get("AGENT_CAPABILITIES")))
|
||||
check("agent ids unique", len({sp.agent_id for sp in specs}) == len(specs))
|
||||
# No key -> OPENAI_API_KEY omitted (not fabricated), no user -> HEICODE_USER_ID omitted.
|
||||
# No key -> OPENAI_API_KEY omitted (not fabricated), no user -> HEICODE_USER_ID omitted, no git -> GIT_* omitted.
|
||||
specs2 = al.plan_launch_specs(run, body, connected_user_agents=0, limit=10, pool_size=1,
|
||||
model_key=None, orchestrator_url="ws://orch", user_id=None)
|
||||
model_key=None, orchestrator_url="ws://orch", user_id=None, git_env=None)
|
||||
check("no model key -> OPENAI_API_KEY omitted", "OPENAI_API_KEY" not in specs2[0].env)
|
||||
check("no user -> HEICODE_USER_ID omitted", "HEICODE_USER_ID" not in specs2[0].env)
|
||||
# git_env (GIT_REPO_URL) merged into every agent's env so the agent clones the bound repo.
|
||||
specs3 = al.plan_launch_specs(run, body, connected_user_agents=0, limit=10, pool_size=3,
|
||||
model_key="sk-test", orchestrator_url="ws://orch", user_id="u-1",
|
||||
git_env={"GIT_REPO_URL": "https://github.com/Fasthei/swe", "GIT_PROVIDER": "github"})
|
||||
check("git_env merged into every spec (GIT_REPO_URL)",
|
||||
all(sp.env.get("GIT_REPO_URL") == "https://github.com/Fasthei/swe" for sp in specs3))
|
||||
check("git_env GIT_PROVIDER propagated", specs3[0].env.get("GIT_PROVIDER") == "github")
|
||||
check("no git_env -> GIT_REPO_URL omitted", "GIT_REPO_URL" not in specs2[0].env)
|
||||
check("no git grant -> GIT_REPO_URL omitted", "GIT_REPO_URL" not in specs2[0].env)
|
||||
|
||||
|
||||
def test_git_launch_env():
|
||||
# Public repo bound via resource grant (metadata.repo_url) -> GIT_REPO_URL injected, no creds.
|
||||
body = {"resource_grants": [{
|
||||
"resource_type": "git", "binding_scope": "https://github.com/Fasthei/swe",
|
||||
"metadata": {"repo_url": "https://github.com/Fasthei/swe", "provider": "github", "default_branch": "main"},
|
||||
"ref": "azkv://heicode-vault.vault.azure.net/secrets/users-3-bindings-x", # present but NOT resolved (public)
|
||||
}]}
|
||||
env = al.git_launch_env(body)
|
||||
check("git_launch_env: GIT_REPO_URL from grant metadata.repo_url", env.get("GIT_REPO_URL") == "https://github.com/Fasthei/swe")
|
||||
check("git_launch_env: GIT_PROVIDER from metadata", env.get("GIT_PROVIDER") == "github")
|
||||
check("git_launch_env: GIT_DEFAULT_BRANCH + GIT_BASE_BRANCH from metadata",
|
||||
env.get("GIT_DEFAULT_BRANCH") == "main" and env.get("GIT_BASE_BRANCH") == "main")
|
||||
check("git_launch_env: public repo injects NO credentials (GIT_TOKEN absent)", "GIT_TOKEN" not in env)
|
||||
# grant found via orchestration_plan.resource_grants too
|
||||
body_plan = {"orchestration_plan": {"resource_grants": [{"resource_type": "git",
|
||||
"metadata": {"repo_url": "https://github.com/a/b"}}]}}
|
||||
check("git_launch_env: reads plan.resource_grants", al.git_launch_env(body_plan).get("GIT_REPO_URL") == "https://github.com/a/b")
|
||||
# git resource via binding_scope (no metadata.repo_url) when resource_type == git
|
||||
body_scope = {"resource_grants": [{"resource_type": "git", "binding_scope": "https://gitea.x/o/r"}]}
|
||||
check("git_launch_env: falls back to binding_scope for git type", al.git_launch_env(body_scope).get("GIT_REPO_URL") == "https://gitea.x/o/r")
|
||||
# no git grant -> empty (a non-git grant is ignored)
|
||||
check("git_launch_env: no git grant -> {}", al.git_launch_env({"resource_grants": [{"resource_type": "database", "metadata": {"host": "h"}}]}) == {})
|
||||
check("git_launch_env: empty body -> {}", al.git_launch_env({}) == {})
|
||||
def test_resolve_git_grant():
|
||||
# ── credential extraction (pure) ──
|
||||
check("git creds: JSON git_username/git_password",
|
||||
al._extract_git_credentials('{"git_username":"u","git_password":"p"}') == {"username": "u", "password": "p"})
|
||||
check("git creds: token form -> default username",
|
||||
al._extract_git_credentials('{"git_token":"ghp_x"}') == {"username": al.DEFAULT_GIT_USERNAME, "password": "ghp_x"})
|
||||
check("git creds: bare string -> token",
|
||||
al._extract_git_credentials("ghp_bare") == {"username": al.DEFAULT_GIT_USERNAME, "password": "ghp_bare"})
|
||||
check("git creds: missing password -> None (never fabricated)", al._extract_git_credentials('{"git_username":"u"}') is None)
|
||||
check("git creds: empty -> None", al._extract_git_credentials("") is None)
|
||||
|
||||
# ── grant lookup + resolution (dev env map, hermetic) ──
|
||||
os.environ.pop("HEICODE_SECRET_res_git_1", None)
|
||||
grant_inline = {"resource_type": "git", "secret_ref": "azkv://heicode-vault/secrets/res_git_1",
|
||||
"metadata": {"repo_url": "https://git.example/acme/app.git", "base_branch": "develop"}}
|
||||
body = {"resource_grants": [grant_inline]}
|
||||
check("no secret in env -> repo url still resolved (public-repo path)",
|
||||
al.resolve_git_grant(body) == {"GIT_REPO_URL": "https://git.example/acme/app.git", "GIT_BASE_BRANCH": "develop"})
|
||||
os.environ["HEICODE_SECRET_res_git_1"] = '{"git_username":"bot","git_password":"ghp_kv"}'
|
||||
out = al.resolve_git_grant(body)
|
||||
check("secret_ref resolved via dev map -> creds injected",
|
||||
out.get("GIT_USERNAME") == "bot" and out.get("GIT_PASSWORD") == "ghp_kv"
|
||||
and out.get("GIT_REPO_URL") == "https://git.example/acme/app.git")
|
||||
os.environ.pop("HEICODE_SECRET_res_git_1")
|
||||
check("no git grant -> None", al.resolve_git_grant({"resource_grants": [{"resource_type": "newapi"}]}) is None)
|
||||
check("git grant without repo_url -> None",
|
||||
al.resolve_git_grant({"resource_grants": [{"resource_type": "git", "secret_ref": "azkv://v/secrets/x"}]}) is None)
|
||||
# per-agent grant (orchestration_plan.agents[].resource_grants) also discovered
|
||||
body_nested = {"orchestration_plan": {"agents": [{"resource_grants": [grant_inline]}]}}
|
||||
check("per-agent git grant discovered",
|
||||
(al.resolve_git_grant(body_nested) or {}).get("GIT_REPO_URL") == "https://git.example/acme/app.git")
|
||||
|
||||
|
||||
def test_resolve_model_key():
|
||||
@@ -216,17 +221,22 @@ def test_k8s_manifests():
|
||||
agent_id="swarm-abc-agent-1", capabilities="python,general",
|
||||
env={"ORCHESTRATOR_URL": "ws://orch.svc:8000", "AGENT_ID": "swarm-abc-agent-1",
|
||||
"AGENT_CAPABILITIES": "python,general", "OPENAI_API_KEY": "sk-secret-xyz",
|
||||
"OPENAI_API_BASE": "https://hm/v1", "HEICODE_USER_ID": "u-1"},
|
||||
"OPENAI_API_BASE": "https://hm/v1", "HEICODE_USER_ID": "u-1",
|
||||
"GIT_REPO_URL": "https://git.example/acme/app.git", "GIT_USERNAME": "x-access-token",
|
||||
"GIT_PASSWORD": "ghp_secret_pw"},
|
||||
)
|
||||
# Secret manifest carries the key (applied via stdin, not argv).
|
||||
sec = al.build_secret_manifest(key_value="sk-secret-xyz", namespace="heicode-swarm",
|
||||
name="swarm-agent-key-swarm-abc", labels={"app": "heicode-swarm-agent"})
|
||||
# Secret manifest carries BOTH sensitive values (model key + git password), applied via stdin.
|
||||
sec = al.build_secret_manifest(namespace="heicode-swarm", name="swarm-agent-key-swarm-abc",
|
||||
labels={"app": "heicode-swarm-agent"},
|
||||
secret_data={"OPENAI_API_KEY": "sk-secret-xyz", "GIT_PASSWORD": "ghp_secret_pw"})
|
||||
check("secret kind/type", sec["kind"] == "Secret" and sec["type"] == "Opaque")
|
||||
check("secret stringData has the key", sec["stringData"]["OPENAI_API_KEY"] == "sk-secret-xyz")
|
||||
check("secret stringData has model key + git password",
|
||||
sec["stringData"]["OPENAI_API_KEY"] == "sk-secret-xyz" and sec["stringData"]["GIT_PASSWORD"] == "ghp_secret_pw")
|
||||
|
||||
# Pod manifest references the key via secretKeyRef — NEVER inline.
|
||||
# Pod manifest references secret env via secretKeyRef — NEVER inline.
|
||||
pod = al.build_pod_manifest(spec, namespace="heicode-swarm", swarm_id="swarm-abc",
|
||||
image="img:1", secret_name="swarm-agent-key-swarm-abc",
|
||||
secret_keys={"OPENAI_API_KEY", "GIT_PASSWORD"},
|
||||
service_account="swarm-agent-sa")
|
||||
blob = json.dumps(pod)
|
||||
check("pod kind/name", pod["kind"] == "Pod" and pod["metadata"]["name"] == "swarm-abc-agent-1")
|
||||
@@ -235,17 +245,25 @@ def test_k8s_manifests():
|
||||
and pod["metadata"]["labels"].get("heicode-user-id") == "u-1")
|
||||
env = {e["name"]: e for e in pod["spec"]["containers"][0]["env"]}
|
||||
check("OPENAI_API_KEY via secretKeyRef (not inline)",
|
||||
"value" not in env["OPENAI_API_KEY"] and env["OPENAI_API_KEY"]["valueFrom"]["secretKeyRef"]["name"] == "swarm-agent-key-swarm-abc")
|
||||
check("raw key value NOT in pod manifest (only in Secret)", "sk-secret-xyz" not in blob)
|
||||
check("non-secret env inline", env["ORCHESTRATOR_URL"]["value"] == "ws://orch.svc:8000")
|
||||
"value" not in env["OPENAI_API_KEY"] and env["OPENAI_API_KEY"]["valueFrom"]["secretKeyRef"]["key"] == "OPENAI_API_KEY")
|
||||
check("GIT_PASSWORD via secretKeyRef (not inline)",
|
||||
"value" not in env["GIT_PASSWORD"] and env["GIT_PASSWORD"]["valueFrom"]["secretKeyRef"]["key"] == "GIT_PASSWORD")
|
||||
check("raw secret values NOT in pod manifest (only in Secret)",
|
||||
"sk-secret-xyz" not in blob and "ghp_secret_pw" not in blob)
|
||||
check("non-secret env inline (orch url + git repo url + git username)",
|
||||
env["ORCHESTRATOR_URL"]["value"] == "ws://orch.svc:8000"
|
||||
and env["GIT_REPO_URL"]["value"] == "https://git.example/acme/app.git"
|
||||
and env["GIT_USERNAME"]["value"] == "x-access-token")
|
||||
check("pod has resource limits + restartPolicy + serviceAccount",
|
||||
pod["spec"]["containers"][0]["resources"]["limits"]
|
||||
and pod["spec"]["restartPolicy"] == "OnFailure"
|
||||
and pod["spec"]["serviceAccountName"] == "swarm-agent-sa")
|
||||
# No secret_name -> key omitted entirely (never inlined).
|
||||
# No secret_name -> secret env omitted entirely (never inlined); non-secret git env still inline.
|
||||
pod2 = al.build_pod_manifest(spec, namespace="heicode-swarm", swarm_id="swarm-abc", secret_name=None)
|
||||
env2 = {e["name"] for e in pod2["spec"]["containers"][0]["env"]}
|
||||
check("no secret -> OPENAI_API_KEY omitted from pod env", "OPENAI_API_KEY" not in env2)
|
||||
check("no secret -> OPENAI_API_KEY + GIT_PASSWORD omitted from pod env",
|
||||
"OPENAI_API_KEY" not in env2 and "GIT_PASSWORD" not in env2)
|
||||
check("no secret -> non-secret GIT_REPO_URL/GIT_USERNAME still present", "GIT_REPO_URL" in env2 and "GIT_USERNAME" in env2)
|
||||
|
||||
|
||||
def main():
|
||||
@@ -255,6 +273,7 @@ def main():
|
||||
test_plan_specs()
|
||||
test_git_launch_env()
|
||||
test_resolve_model_key()
|
||||
test_resolve_git_grant()
|
||||
test_azkv_resolver()
|
||||
test_command_backend_build()
|
||||
test_k8s_manifests()
|
||||
|
||||
Reference in New Issue
Block a user