主要更新: - 新增 external_tool_api.py: 外部工具管理 API - 新增 tool_storage.py: 工具存储管理器 - 新增回调功能用于计费 (agent_callback_utils) - 支持多工具创建 Agent - 新增 CI/CD 构建状态查询 API - 新增部署信息查询 API - 更新文档 (EXTERNAL_TOOL_API.md v2.0) - 更新 Dockerfile 添加新模块 - 更新 app.py 集成外部工具路由
224 lines
6.4 KiB
Markdown
224 lines
6.4 KiB
Markdown
# Agent Manager DNS问题修复报告
|
||
|
||
## 问题描述
|
||
|
||
在K8s上部署的Agent Manager服务,创建新agent后没有按照预期返回域名和外网地址。
|
||
|
||
## 问题诊断
|
||
|
||
### 1. 症状
|
||
- 创建agent时返回外网IP地址(`external_ip`)
|
||
- 但没有返回DNS域名信息(`domain`, `domain_url`)
|
||
- 推荐访问地址使用的是IP地址而非域名
|
||
|
||
### 2. 根本原因
|
||
通过查看agent-manager日志发现:
|
||
```
|
||
ERROR:k8s_manager:创建 DNS 记录失败: 403 Client Error: Forbidden
|
||
ERROR:k8s_manager:错误详情: {"error":{"code":"AuthorizationFailed",
|
||
"message":"The client 'c5ba26db-f180-425f-bac3-93708d853988' with object id
|
||
'9dd70c60-bca4-478f-8cac-9faf9f171c93' does not have authorization to perform
|
||
action 'Microsoft.Network/dnsZones/A/write' over scope
|
||
'/subscriptions/45d7a360-af09-40fc-9afc-56dc475245ec/resourceGroups/taiji-ai-v0/
|
||
providers/Microsoft.Network/dnsZones/taijiagnet.com/A/...' or the scope is invalid."}}
|
||
```
|
||
|
||
**核心问题:** Azure Service Principal缺少DNS Zone的写权限。
|
||
|
||
## 解决方案
|
||
|
||
### 步骤1:为Service Principal分配DNS Zone Contributor角色
|
||
|
||
```bash
|
||
# Azure配置
|
||
AZURE_CLIENT_ID="c5ba26db-f180-425f-bac3-93708d853988"
|
||
AZURE_SUBSCRIPTION_ID="45d7a360-af09-40fc-9afc-56dc475245ec"
|
||
AZURE_RESOURCE_GROUP="taiji-ai-v0"
|
||
AZURE_DNS_ZONE="taijiagnet.com"
|
||
|
||
# 构建DNS Zone资源ID
|
||
DNS_ZONE_ID="/subscriptions/$AZURE_SUBSCRIPTION_ID/resourceGroups/$AZURE_RESOURCE_GROUP/providers/Microsoft.Network/dnsZones/$AZURE_DNS_ZONE"
|
||
|
||
# 分配DNS Zone Contributor角色
|
||
az role assignment create \
|
||
--assignee $AZURE_CLIENT_ID \
|
||
--role "DNS Zone Contributor" \
|
||
--scope $DNS_ZONE_ID
|
||
```
|
||
|
||
### 步骤2:验证权限分配
|
||
|
||
```bash
|
||
# 验证角色分配
|
||
az role assignment list \
|
||
--assignee $AZURE_CLIENT_ID \
|
||
--scope $DNS_ZONE_ID \
|
||
--output table
|
||
```
|
||
|
||
输出应显示:
|
||
```
|
||
Principal Role Scope
|
||
------------------------------------ -------------------- --------
|
||
c5ba26db-f180-425f-bac3-93708d853988 DNS Zone Contributor /subscriptions/.../dnsZones/taijiagnet.com
|
||
```
|
||
|
||
## 修复验证
|
||
|
||
### 测试1:创建新agent
|
||
```bash
|
||
curl -X POST http://20.212.121.126/agents \
|
||
-H "Content-Type: application/json" \
|
||
-d '{
|
||
"name": "test-dns-fixed",
|
||
"template": "echo_agent",
|
||
"framework": "API",
|
||
"config": {"user_id": "test-user"}
|
||
}'
|
||
```
|
||
|
||
### 结果(修复前)
|
||
```json
|
||
{
|
||
"access_info": {
|
||
"external_ip": "20.212.128.51",
|
||
"ip_url": "http://20.212.128.51:80",
|
||
"recommended": "http://20.212.128.51:80"
|
||
}
|
||
}
|
||
```
|
||
|
||
### 结果(修复后)✅
|
||
```json
|
||
{
|
||
"access_info": {
|
||
"external_ip": "20.198.137.237",
|
||
"ip_url": "http://20.198.137.237:80",
|
||
"service_url": "http://10.0.48.107:80",
|
||
"pod_url": "http://10.224.0.67:8000",
|
||
"domain": "test-dns-fixed.taijiagnet.com",
|
||
"domain_url": "http://test-dns-fixed.taijiagnet.com",
|
||
"recommended": "http://test-dns-fixed.taijiagnet.com"
|
||
}
|
||
}
|
||
```
|
||
|
||
### 测试2:验证DNS记录
|
||
```bash
|
||
# 查看DNS记录
|
||
az network dns record-set a show \
|
||
--resource-group taiji-ai-v0 \
|
||
--zone-name taijiagnet.com \
|
||
--name test-dns-fixed
|
||
|
||
# DNS解析测试
|
||
nslookup test-dns-fixed.taijiagnet.com
|
||
# Output: Address: 20.198.137.237
|
||
|
||
# 域名访问测试
|
||
curl http://test-dns-fixed.taijiagnet.com/
|
||
# Output: {"status":"healthy",...}
|
||
```
|
||
|
||
### 测试3:查询agent状态
|
||
```bash
|
||
curl http://20.212.121.126/agents/test-dns-fixed/status | jq .access_info
|
||
```
|
||
|
||
结果:
|
||
```json
|
||
{
|
||
"external_ip": "20.198.137.237",
|
||
"ip_url": "http://20.198.137.237:80",
|
||
"domain": "test-dns-fixed.taijiagnet.com",
|
||
"domain_url": "http://test-dns-fixed.taijiagnet.com",
|
||
"recommended_url": "http://test-dns-fixed.taijiagnet.com",
|
||
"service_name": "test-dns-fixed-service"
|
||
}
|
||
```
|
||
|
||
## Agent Manager日志(修复后)
|
||
|
||
```
|
||
INFO:app:收到创建Agent请求: test-dns-fixed, 模板: echo_agent
|
||
INFO:k8s_manager:✅ 创建命名空间 agent-test-dns-fixed
|
||
INFO:k8s_manager:✅ LoadBalancer Service test-dns-fixed-service 创建成功
|
||
INFO:k8s_manager:✅ LoadBalancer IP 已分配: 20.198.137.237
|
||
INFO:app:✅ LoadBalancer 外网 IP: 20.198.137.237
|
||
INFO:app:创建 DNS 记录: test-dns-fixed.taijiagnet.com
|
||
INFO:k8s_manager:✅ DNS 记录创建成功: test-dns-fixed.taijiagnet.com -> 20.198.137.237
|
||
INFO:app:✅ DNS 记录: test-dns-fixed.taijiagnet.com -> 20.198.137.237
|
||
INFO:app: - 推荐访问: http://test-dns-fixed.taijiagnet.com
|
||
INFO:app:✅ Agent创建成功!
|
||
INFO:app:✅ Agent信息已保存到数据库
|
||
```
|
||
|
||
## 涉及的文件和组件
|
||
|
||
1. **Azure Service Principal** (c5ba26db-f180-425f-bac3-93708d853988)
|
||
- 需要DNS Zone Contributor权限
|
||
|
||
2. **k8s_manager.py** - DNS记录创建逻辑
|
||
- `create_dns_record()` - 使用Azure DNS API创建A记录
|
||
- `wait_for_loadbalancer_ip()` - 等待LoadBalancer IP分配
|
||
|
||
3. **app.py** - Agent创建流程
|
||
- 在创建LoadBalancer Service后自动创建DNS记录
|
||
- 将域名信息保存到数据库
|
||
|
||
4. **database.py** - Agent模型
|
||
- 存储`domain`, `domain_url`, `recommended_url`等字段
|
||
|
||
## 修复完成清单 ✅
|
||
|
||
- [x] 诊断DNS创建失败原因(Azure权限不足)
|
||
- [x] 为Service Principal分配DNS Zone Contributor角色
|
||
- [x] 测试DNS记录创建功能
|
||
- [x] 验证agent创建返回完整的域名信息
|
||
- [x] 验证agent状态查询返回完整的访问信息
|
||
- [x] 验证DNS解析和域名访问正常
|
||
- [x] 清理测试资源
|
||
|
||
## 后续建议
|
||
|
||
### 1. 文档更新
|
||
更新部署文档,明确说明Service Principal需要的Azure权限:
|
||
- AKS Cluster的Contributor角色
|
||
- DNS Zone的DNS Zone Contributor角色
|
||
|
||
### 2. 自动化部署脚本
|
||
在`k8s/deploy.sh`中添加权限检查和自动分配逻辑:
|
||
```bash
|
||
# 检查并分配DNS Zone权限
|
||
check_dns_permissions() {
|
||
echo "检查DNS Zone权限..."
|
||
# 实现权限检查逻辑
|
||
}
|
||
```
|
||
|
||
### 3. 监控和告警
|
||
添加DNS创建失败的监控和告警机制:
|
||
- 记录DNS创建失败次数
|
||
- 当失败超过阈值时发送告警
|
||
|
||
### 4. 重试机制
|
||
在`k8s_manager.py`中为DNS记录创建添加重试逻辑:
|
||
```python
|
||
def create_dns_record_with_retry(self, subdomain, ip_address, max_retries=3):
|
||
for attempt in range(max_retries):
|
||
try:
|
||
return self.create_dns_record(subdomain, ip_address)
|
||
except Exception as e:
|
||
if attempt < max_retries - 1:
|
||
logger.warning(f"DNS创建失败,重试 {attempt + 1}/{max_retries}")
|
||
time.sleep(5)
|
||
else:
|
||
raise
|
||
```
|
||
|
||
## 修复日期
|
||
2026-01-28
|
||
|
||
## 修复人员
|
||
AI Assistant (Claude Sonnet 4.5)
|