更新api接口
This commit is contained in:
+71
-73
@@ -1,11 +1,11 @@
|
||||
# taiji-AI-PAD API 接口文档
|
||||
|
||||
**版本**: v1.2.1
|
||||
**更新时间**: 2025年12月22日
|
||||
**最后更新**: 2025年12月22日
|
||||
**版本**: v1.2.2
|
||||
**更新时间**: 2025年12月24日
|
||||
**最后更新**: 2025年12月24日
|
||||
**基础URL**:
|
||||
- Data Ingestion 服务: `http://localhost:8001`
|
||||
- MCP Server 服务: `http://localhost:8000`
|
||||
- MCP Server 服务: `http://localhost:8002`
|
||||
- API Gateway: `http://localhost:80`
|
||||
|
||||
---
|
||||
@@ -228,7 +228,10 @@ curl -X POST "http://localhost:8001/openapi/parse?url=https://api.example.com/op
|
||||
**请求参数说明**:
|
||||
- `api_doc` (string | object, 必需): API 文档，可以是字符串或对象
|
||||
- `context` (object, 可选): 上下文信息
|
||||
- `output_format` (string, 可选): 输出格式，可选值: `json_schema`, `pydantic`, `openapi` (默认: `json_schema`)
|
||||
- `output_format` (string, 可选): 输出格式，可选值: `pydantic`, `json_schema`, `openapi` (默认: `pydantic`)
|
||||
- `include_examples` (bool, 可选, 默认: true): 是否生成示例
|
||||
- `enhance_descriptions` (bool, 可选, 默认: true): 是否增强描述
|
||||
- `validate_schema` (bool, 可选, 默认: true): 是否校验 Schema 完整性
|
||||
|
||||
**请求示例**:
|
||||
```bash
|
||||
@@ -243,7 +246,10 @@ curl -X POST "http://localhost:8001/apillama/process" \
|
||||
]
|
||||
},
|
||||
"context": {"service": "Weather service", "version": "1.0"},
|
||||
"output_format": "json_schema"
|
||||
"output_format": "json_schema",
|
||||
"include_examples": true,
|
||||
"enhance_descriptions": true,
|
||||
"validate_schema": true
|
||||
}'
|
||||
```
|
||||
|
||||
@@ -291,7 +297,7 @@ curl -X POST "http://localhost:8001/apillama/process" \
|
||||
|
||||
从 API 端点生成工具定义。
|
||||
|
||||
**请求体**:
|
||||
**请求体 (APIEndpoint Schema)**:
|
||||
```json
|
||||
{
|
||||
"url": "https://api.example.com/users",
|
||||
@@ -302,12 +308,33 @@ curl -X POST "http://localhost:8001/apillama/process" \
|
||||
{
|
||||
"name": "page",
|
||||
"type": "integer",
|
||||
"location": "query",
|
||||
"description": "Page number",
|
||||
"required": false
|
||||
}
|
||||
],
|
||||
"headers": {
|
||||
"Authorization": "Bearer token"
|
||||
}
|
||||
"request_body": null,
|
||||
"responses": {
|
||||
"200": {
|
||||
"description": "Success",
|
||||
"content": {
|
||||
"application/json": {
|
||||
"schema": {
|
||||
"type": "object",
|
||||
"properties": {
|
||||
"items": {
|
||||
"type": "array",
|
||||
"items": {"type": "object"}
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
},
|
||||
"security": [],
|
||||
"tags": ["users"],
|
||||
"deprecated": false
|
||||
}
|
||||
```
|
||||
|
||||
@@ -320,8 +347,13 @@ curl -X POST "http://localhost:8001/tools/generate" \
|
||||
"method": "GET",
|
||||
"name": "get_users",
|
||||
"description": "Get list of users",
|
||||
"parameters": [{"name": "page", "type": "integer", "required": false}],
|
||||
"headers": {"Authorization": "Bearer token"}
|
||||
"parameters": [
|
||||
{"name": "page", "type": "integer", "location": "query", "required": false}
|
||||
],
|
||||
"responses": {"200": {"description": "Success"}},
|
||||
"security": [],
|
||||
"tags": ["users"],
|
||||
"deprecated": false
|
||||
}'
|
||||
```
|
||||
|
||||
@@ -345,7 +377,7 @@ curl -X POST "http://localhost:8001/tools/generate" \
|
||||
**查询参数**:
|
||||
- `category` (string, 可选): 工具分类
|
||||
- `limit` (int, 可选, 默认: 100): 返回数量限制
|
||||
- `offset` (int, 可选, 默认: 0): 偏移量
|
||||
- `offset` (int, 可选, 默认: 0): 偏移量（通过 Redis 集合切片实现）
|
||||
|
||||
**请求示例**:
|
||||
```bash
|
||||
@@ -518,7 +550,7 @@ apillama_processing_duration_seconds_bucket{le="1.0"} 200
|
||||
|
||||
## MCP Server 服务 API
|
||||
|
||||
**基础URL**: `http://localhost:8000`
|
||||
**基础URL**: `http://localhost:8002`
|
||||
|
||||
### 1. 健康检查
|
||||
|
||||
@@ -528,7 +560,7 @@ apillama_processing_duration_seconds_bucket{le="1.0"} 200
|
||||
|
||||
**请求示例**:
|
||||
```bash
|
||||
curl -X GET "http://localhost:8000/health" \
|
||||
curl -X GET "http://localhost:8002/health" \
|
||||
-H "Accept: application/json"
|
||||
```
|
||||
|
||||
@@ -537,6 +569,7 @@ curl -X GET "http://localhost:8000/health" \
|
||||
{
|
||||
"status": "healthy",
|
||||
"timestamp": "2025-12-22T05:04:23.211960",
|
||||
"version": "1.0.0",
|
||||
"services": {
|
||||
"database": "healthy",
|
||||
"redis": "healthy",
|
||||
@@ -570,7 +603,7 @@ curl -X GET "http://localhost:8000/health" \
|
||||
|
||||
**请求示例**:
|
||||
```bash
|
||||
curl -X POST "http://localhost:8000/agents" \
|
||||
curl -X POST "http://localhost:8002/agents" \
|
||||
-H "Content-Type: application/json" \
|
||||
-d '{
|
||||
"name": "weather-agent",
|
||||
@@ -617,13 +650,12 @@ curl -X POST "http://localhost:8000/agents" \
|
||||
获取所有注册的 Agent 列表。
|
||||
|
||||
**查询参数**:
|
||||
- `status` (string, 可选): 过滤状态，如 `active`, `inactive`
|
||||
- `skip` (int, 可选, 默认: 0): 起始偏移
|
||||
- `limit` (int, 可选, 默认: 100): 返回数量限制
|
||||
- `offset` (int, 可选, 默认: 0): 偏移量
|
||||
|
||||
**请求示例**:
|
||||
```bash
|
||||
curl -X GET "http://localhost:8000/agents?limit=20&offset=0" \
|
||||
curl -X GET "http://localhost:8002/agents?skip=0&limit=20" \
|
||||
-H "Accept: application/json"
|
||||
```
|
||||
|
||||
@@ -667,7 +699,7 @@ curl -X GET "http://localhost:8000/agents?limit=20&offset=0" \
|
||||
|
||||
**请求示例**:
|
||||
```bash
|
||||
curl -X GET "http://localhost:8000/agents/d7b0a5c2-5f6a-4c27-9ef9-8d51b94f7a1b" \
|
||||
curl -X GET "http://localhost:8002/agents/d7b0a5c2-5f6a-4c27-9ef9-8d51b94f7a1b" \
|
||||
-H "Accept: application/json"
|
||||
```
|
||||
|
||||
@@ -734,7 +766,7 @@ curl -X GET "http://localhost:8000/agents/d7b0a5c2-5f6a-4c27-9ef9-8d51b94f7a1b"
|
||||
|
||||
**请求示例**:
|
||||
```bash
|
||||
curl -X POST "http://localhost:8000/agents/d7b0a5c2-5f6a-4c27-9ef9-8d51b94f7a1b/execute" \
|
||||
curl -X POST "http://localhost:8002/agents/d7b0a5c2-5f6a-4c27-9ef9-8d51b94f7a1b/execute" \
|
||||
-H "Content-Type: application/json" \
|
||||
-d '{
|
||||
"jsonrpc": "2.0",
|
||||
@@ -786,52 +818,17 @@ curl -X POST "http://localhost:8000/agents/d7b0a5c2-5f6a-4c27-9ef9-8d51b94f7a1b/
|
||||
|
||||
**GET** `/tools`
|
||||
|
||||
获取所有可用工具列表。
|
||||
|
||||
**查询参数**:
|
||||
- `category` (string, 可选): 工具分类
|
||||
- `limit` (int, 可选, 默认: 100): 返回数量限制
|
||||
当前端点为占位实现，返回空数组（工具清单由 Data Ingestion 服务维护）。
|
||||
|
||||
**请求示例**:
|
||||
```bash
|
||||
curl -X GET "http://localhost:8000/tools" \
|
||||
curl -X GET "http://localhost:8002/tools" \
|
||||
-H "Accept: application/json"
|
||||
```
|
||||
|
||||
**响应示例**:
|
||||
```json
|
||||
[
|
||||
{
|
||||
"name": "get_weather",
|
||||
"description": "Retrieve current weather information",
|
||||
"category": "weather",
|
||||
"parameters": [
|
||||
{
|
||||
"name": "location",
|
||||
"type": "string",
|
||||
"description": "City name or coordinates",
|
||||
"required": true,
|
||||
"default": null,
|
||||
"enum": null
|
||||
}
|
||||
],
|
||||
"returns": {
|
||||
"type": "object",
|
||||
"properties": {
|
||||
"temperature": {"type": "number", "description": "Current temperature in Celsius"},
|
||||
"condition": {"type": "string", "description": "Weather condition summary"}
|
||||
}
|
||||
},
|
||||
"endpoint": "https://api.example.com/weather",
|
||||
"method": "GET",
|
||||
"headers": {
|
||||
"Authorization": "Bearer <token>"
|
||||
},
|
||||
"rate_limit": 60,
|
||||
"timeout": 30,
|
||||
"cost_per_call": 0.001
|
||||
}
|
||||
]
|
||||
[]
|
||||
```
|
||||
|
||||
---
|
||||
@@ -844,7 +841,7 @@ curl -X GET "http://localhost:8000/tools" \
|
||||
|
||||
**请求示例**:
|
||||
```bash
|
||||
curl -X GET "http://localhost:8000/metrics"
|
||||
curl -X GET "http://localhost:8002/metrics"
|
||||
```
|
||||
|
||||
**示例**:
|
||||
@@ -863,7 +860,7 @@ tool_calls_total{tool_type="function",status="success"} 18
|
||||
|
||||
## MCP 监控 API
|
||||
|
||||
**基础URL**: `http://localhost:8000`
|
||||
**基础URL**: `http://localhost:8002`
|
||||
|
||||
### 1. 获取系统性能指标
|
||||
|
||||
@@ -873,7 +870,7 @@ tool_calls_total{tool_type="function",status="success"} 18
|
||||
|
||||
**请求示例**:
|
||||
```bash
|
||||
curl -X GET "http://localhost:8000/api/v1/monitoring/metrics" \
|
||||
curl -X GET "http://localhost:8002/api/v1/monitoring/metrics" \
|
||||
-H "Accept: application/json"
|
||||
```
|
||||
|
||||
@@ -915,7 +912,7 @@ curl -X GET "http://localhost:8000/api/v1/monitoring/metrics" \
|
||||
|
||||
**请求示例**:
|
||||
```bash
|
||||
curl -X GET "http://localhost:8000/api/v1/monitoring/stats?service=all" \
|
||||
curl -X GET "http://localhost:8002/api/v1/monitoring/stats?service=all" \
|
||||
-H "Accept: application/json"
|
||||
```
|
||||
|
||||
@@ -968,7 +965,7 @@ curl -X GET "http://localhost:8000/api/v1/monitoring/stats?service=all" \
|
||||
|
||||
**请求示例**:
|
||||
```bash
|
||||
curl -X GET "http://localhost:8000/api/v1/monitoring/trends?metric=executions&period=24h&interval=1h" \
|
||||
curl -X GET "http://localhost:8002/api/v1/monitoring/trends?metric=executions&period=24h&interval=1h" \
|
||||
-H "Accept: application/json"
|
||||
```
|
||||
|
||||
@@ -1006,7 +1003,7 @@ curl -X GET "http://localhost:8000/api/v1/monitoring/trends?metric=executions&pe
|
||||
|
||||
**请求示例**:
|
||||
```bash
|
||||
curl -X GET "http://localhost:8000/api/v1/monitoring/alerts?severity=warning" \
|
||||
curl -X GET "http://localhost:8002/api/v1/monitoring/alerts?severity=warning" \
|
||||
-H "Accept: application/json"
|
||||
```
|
||||
|
||||
@@ -1034,7 +1031,7 @@ curl -X GET "http://localhost:8000/api/v1/monitoring/alerts?severity=warning" \
|
||||
|
||||
**请求示例**:
|
||||
```bash
|
||||
curl -X GET "http://localhost:8000/api/v1/monitoring/dashboard" \
|
||||
curl -X GET "http://localhost:8002/api/v1/monitoring/dashboard" \
|
||||
-H "Accept: application/json"
|
||||
```
|
||||
|
||||
@@ -1090,11 +1087,11 @@ curl -X GET "http://localhost:8000/api/v1/monitoring/dashboard" \
|
||||
|
||||
### MCP Protocol WebSocket
|
||||
|
||||
**WebSocket URL**: `ws://localhost:8000/ws/{agent_id}`
|
||||
**WebSocket URL**: `ws://localhost:8002/ws/{agent_name_or_id}`（`agent_name_or_id` 可为 Agent UUID 或名称）
|
||||
|
||||
**连接示例**:
|
||||
```javascript
|
||||
const ws = new WebSocket('ws://localhost:8000/ws/agent_123456');
|
||||
const ws = new WebSocket('ws://localhost:8002/ws/agent_123456');
|
||||
```
|
||||
|
||||
**消息格式**:
|
||||
@@ -1183,17 +1180,17 @@ const ws = new WebSocket('ws://localhost:8000/ws/agent_123456');
|
||||
### Swagger UI
|
||||
|
||||
- Data Ingestion: `http://localhost:8001/docs`
|
||||
- MCP Server: `http://localhost:8000/docs`
|
||||
- MCP Server: `http://localhost:8002/docs`
|
||||
|
||||
### ReDoc
|
||||
|
||||
- Data Ingestion: `http://localhost:8001/redoc`
|
||||
- MCP Server: `http://localhost:8000/redoc`
|
||||
- MCP Server: `http://localhost:8002/redoc`
|
||||
|
||||
### OpenAPI JSON
|
||||
|
||||
- Data Ingestion: `http://localhost:8001/openapi.json`
|
||||
- MCP Server: `http://localhost:8000/openapi.json`
|
||||
- MCP Server: `http://localhost:8002/openapi.json`
|
||||
|
||||
---
|
||||
|
||||
@@ -1297,12 +1294,13 @@ curl http://localhost:8001/tools?category=weather
|
||||
|
||||
---
|
||||
|
||||
**文档版本**: v1.2.1
|
||||
**最后更新**: 2025年12月22日
|
||||
**文档版本**: v1.2.2
|
||||
**最后更新**: 2025年12月24日
|
||||
**维护者**: taiji-AI-PAD 项目组
|
||||
|
||||
## 更新日志
|
||||
|
||||
- **v1.2.2** (2025-12-24): 同步 APILLAMA 参数、工具生成 Schema、WebSocket 端点等代码改动
|
||||
- **v1.2.1** (2025-12-22): 添加 MCP Server 函数工具调用说明
|
||||
- **v1.2.0** (2025-12-22): 初始版本，包含所有 API 端点文档
|
||||
|
||||
|
||||
@@ -0,0 +1,277 @@
|
||||
#!/usr/bin/env python3
|
||||
"""End-to-end flow tester for taiji-AI-PAD services."""
|
||||
from __future__ import annotations
|
||||
|
||||
import argparse
|
||||
import sys
|
||||
import uuid
|
||||
from dataclasses import dataclass
|
||||
from typing import Any, Dict, Iterable, Optional
|
||||
|
||||
import requests
|
||||
|
||||
|
||||
@dataclass
|
||||
class ServiceConfig:
|
||||
data_ingestion_url: str = "http://localhost:8001"
|
||||
mcp_server_url: str = "http://localhost:8002"
|
||||
timeout: int = 30
|
||||
|
||||
|
||||
class ApiFlowTester:
|
||||
def __init__(self, config: ServiceConfig, verbose: bool = True) -> None:
|
||||
self.config = config
|
||||
self.session = requests.Session()
|
||||
self.verbose = verbose
|
||||
|
||||
def run(self, skip_data_ingestion: bool, skip_mcp: bool) -> None:
|
||||
if not skip_data_ingestion:
|
||||
self._log("Running Data Ingestion flow")
|
||||
self._test_data_ingestion_flow()
|
||||
else:
|
||||
self._log("Skipping Data Ingestion flow")
|
||||
|
||||
if not skip_mcp:
|
||||
self._log("Running MCP Server flow")
|
||||
self._test_mcp_flow()
|
||||
else:
|
||||
self._log("Skipping MCP Server flow")
|
||||
|
||||
def _test_data_ingestion_flow(self) -> None:
|
||||
base = self.config.data_ingestion_url
|
||||
|
||||
self._log("Checking Data Ingestion health endpoint")
|
||||
health = self._json_request("GET", f"{base}/health")
|
||||
self._ensure_service_health(
|
||||
health,
|
||||
critical_keys=["data_ingestion"],
|
||||
context="Data Ingestion",
|
||||
)
|
||||
|
||||
self._log("Triggering RapidAPI sync job")
|
||||
sync_response = self._json_request(
|
||||
"POST",
|
||||
f"{base}/rapidapi/sync",
|
||||
params={"category": "weather", "limit": 1},
|
||||
)
|
||||
self._require("message" in sync_response, "RapidAPI sync did not return confirmation", sync_response)
|
||||
|
||||
self._log("Parsing reference OpenAPI specification")
|
||||
openapi_url = "https://raw.githubusercontent.com/OAI/OpenAPI-Specification/main/examples/v3.0/petstore.yaml"
|
||||
openapi = self._json_request("POST", f"{base}/openapi/parse", params={"url": openapi_url})
|
||||
self._require(openapi.get("parsed_data"), "OpenAPI parse missing parsed_data", openapi)
|
||||
|
||||
self._log("Running APILLAMA processing step")
|
||||
apillama_payload = {
|
||||
"api_doc": {
|
||||
"title": "Weather API",
|
||||
"description": "Returns forecast information",
|
||||
"parameters": [
|
||||
{"name": "location", "type": "string", "description": "City name", "required": True}
|
||||
],
|
||||
},
|
||||
"context": {"service": "weather", "version": "1.0"},
|
||||
"output_format": "json_schema",
|
||||
}
|
||||
apillama = self._json_request("POST", f"{base}/apillama/process", json=apillama_payload)
|
||||
self._require(apillama.get("processed"), "APILLAMA processing failed", apillama)
|
||||
|
||||
self._log("Requesting tool generation task")
|
||||
tool_payload = {
|
||||
"url": "https://api.example.com/weather",
|
||||
"method": "GET",
|
||||
"name": f"diag_get_weather_{uuid.uuid4().hex[:8]}",
|
||||
"description": "Diagnostic weather fetch tool",
|
||||
"parameters": [
|
||||
{"name": "location", "type": "string", "required": True},
|
||||
{"name": "unit", "type": "string", "required": False},
|
||||
],
|
||||
"headers": {"Authorization": "Bearer demo-token"},
|
||||
}
|
||||
tool_gen = self._json_request("POST", f"{base}/tools/generate", json=tool_payload)
|
||||
self._require(
|
||||
tool_gen.get("message"),
|
||||
"Tool generation endpoint did not acknowledge request",
|
||||
tool_gen,
|
||||
)
|
||||
|
||||
self._log("Listing generated tools")
|
||||
tools = self._json_request("GET", f"{base}/tools", params={"limit": 5})
|
||||
self._require(isinstance(tools, list), "Tools endpoint did not return a list", tools)
|
||||
if tools:
|
||||
self._log("Fetching first tool definition for verification")
|
||||
first_tool = tools[0]["name"]
|
||||
definition = self._json_request("GET", f"{base}/tools/{first_tool}")
|
||||
self._require(
|
||||
definition.get("name") == first_tool,
|
||||
"Fetched tool definition does not match",
|
||||
definition,
|
||||
)
|
||||
|
||||
self._log("Reading Data Ingestion metrics endpoint")
|
||||
metrics_text = self._request("GET", f"{base}/metrics").text
|
||||
self._require("http_requests_total" in metrics_text, "Metrics output missing expected counters")
|
||||
|
||||
def _test_mcp_flow(self) -> None:
|
||||
base = self.config.mcp_server_url
|
||||
|
||||
self._log("Checking MCP health endpoint")
|
||||
health = self._json_request("GET", f"{base}/health")
|
||||
self._ensure_service_health(
|
||||
health,
|
||||
critical_keys=["database"],
|
||||
context="MCP",
|
||||
)
|
||||
|
||||
self._log("Fetching current MCP agent list")
|
||||
existing_agents = self._json_request("GET", f"{base}/agents", params={"limit": 100, "offset": 0})
|
||||
self._require(isinstance(existing_agents, list), "Agent list did not return a list", existing_agents)
|
||||
existing_ids = {agent.get("id") for agent in existing_agents if agent.get("id")}
|
||||
|
||||
self._log("Registering diagnostic agent")
|
||||
agent_name = f"auto-agent-{uuid.uuid4().hex[:8]}"
|
||||
agent_payload = {
|
||||
"name": agent_name,
|
||||
"description": "Auto-generated diagnostic agent",
|
||||
"role": "assistant",
|
||||
"goal": "Validate MCP server flows",
|
||||
"tools": ["math_add"],
|
||||
"config": {"default_model": "gpt-4o-mini"},
|
||||
"capabilities": ["diagnostics"],
|
||||
}
|
||||
created_agent = self._json_request("POST", f"{base}/agents", json=agent_payload)
|
||||
agent_id = created_agent.get("id")
|
||||
self._require(agent_id, "Agent creation response missing id", created_agent)
|
||||
|
||||
self._log("Verifying new agent presence in list")
|
||||
refreshed_agents = self._json_request("GET", f"{base}/agents", params={"limit": 100, "offset": 0})
|
||||
after_ids = {agent.get("id") for agent in refreshed_agents if agent.get("id")}
|
||||
self._require(agent_id in after_ids, "New agent not found in list after creation")
|
||||
|
||||
self._log("Fetching newly created agent details")
|
||||
fetched_agent = self._json_request("GET", f"{base}/agents/{agent_id}")
|
||||
self._require(fetched_agent.get("name") == agent_name, "Fetched agent does not match created agent", fetched_agent)
|
||||
|
||||
self._log("Executing math_add tool via MCP agent")
|
||||
execution_payload = {
|
||||
"jsonrpc": "2.0",
|
||||
"id": f"exec-{uuid.uuid4().hex[:8]}",
|
||||
"method": "tools/call",
|
||||
"params": {
|
||||
"tool": {"name": "math_add", "function_name": "math_add"},
|
||||
"arguments": {"a": 1, "b": 2},
|
||||
"context": {"session_id": f"session-{agent_id[:8]}"},
|
||||
},
|
||||
}
|
||||
execution = self._json_request("POST", f"{base}/agents/{agent_id}/execute", json=execution_payload)
|
||||
self._require(execution.get("success"), "Agent tool execution failed", execution)
|
||||
self._require(execution.get("result") == 3, "math_add result mismatch", execution)
|
||||
|
||||
self._log("Listing MCP tools for visibility")
|
||||
tools = self._json_request("GET", f"{base}/tools", params={"limit": 5})
|
||||
self._require(isinstance(tools, list), "MCP tools endpoint did not return a list", tools)
|
||||
|
||||
self._log("Reading MCP metrics endpoint")
|
||||
metrics_text = self._request("GET", f"{base}/metrics").text
|
||||
self._require("http_requests_total" in metrics_text, "MCP metrics missing expected counters")
|
||||
|
||||
self._log("Fetching monitoring metrics snapshot")
|
||||
monitoring = self._json_request("GET", f"{base}/api/v1/monitoring/metrics")
|
||||
self._require(
|
||||
monitoring.get("system"),
|
||||
"Monitoring metrics missing system section",
|
||||
monitoring,
|
||||
)
|
||||
|
||||
def _request(
|
||||
self,
|
||||
method: str,
|
||||
url: str,
|
||||
*,
|
||||
expected_status: Optional[Iterable[int]] = None,
|
||||
**kwargs: Any,
|
||||
) -> requests.Response:
|
||||
response = self.session.request(method, url, timeout=self.config.timeout, **kwargs)
|
||||
acceptable = list(expected_status) if expected_status is not None else []
|
||||
if expected_status is None and not 200 <= response.status_code < 300:
|
||||
raise AssertionError(
|
||||
f"Request to {url} failed with status {response.status_code}: {response.text[:200]}"
|
||||
)
|
||||
if expected_status is not None and response.status_code not in acceptable:
|
||||
raise AssertionError(
|
||||
f"Request to {url} expected {acceptable} but received {response.status_code}: {response.text[:200]}"
|
||||
)
|
||||
return response
|
||||
|
||||
def _json_request(self, method: str, url: str, **kwargs: Any) -> Dict[str, Any] | Any:
|
||||
response = self._request(method, url, **kwargs)
|
||||
try:
|
||||
return response.json()
|
||||
except ValueError as exc: # pragma: no cover - defensive guard
|
||||
raise AssertionError(f"Response from {url} is not valid JSON: {response.text[:200]}") from exc
|
||||
|
||||
def _require(self, condition: bool, message: str, payload: Optional[Any] = None) -> None:
|
||||
if not condition:
|
||||
detail = f" | payload={payload}" if payload is not None else ""
|
||||
raise AssertionError(f"{message}{detail}")
|
||||
|
||||
def _ensure_service_health(
|
||||
self,
|
||||
payload: Dict[str, Any],
|
||||
*,
|
||||
critical_keys: Optional[Iterable[str]] = None,
|
||||
context: str,
|
||||
) -> None:
|
||||
status = str(payload.get("status", "")).lower()
|
||||
acceptable = {"healthy", "ok", "degraded"}
|
||||
services = payload.get("services") or {}
|
||||
if status not in acceptable:
|
||||
raise AssertionError(f"{context} health status unacceptable: {status} | payload={payload}")
|
||||
if critical_keys:
|
||||
missing = [svc for svc in critical_keys if services.get(svc) not in {"healthy", "ok"}]
|
||||
if missing:
|
||||
raise AssertionError(
|
||||
f"{context} critical services unhealthy: {missing} | payload={payload}"
|
||||
)
|
||||
degraded = [name for name, svc_status in services.items() if svc_status == "degraded"]
|
||||
if degraded:
|
||||
self._log(
|
||||
f"{context} warning: degraded dependencies detected: {', '.join(degraded)}"
|
||||
)
|
||||
|
||||
def _log(self, message: str) -> None:
|
||||
if self.verbose:
|
||||
print(f"[api-flow] {message}")
|
||||
|
||||
|
||||
def parse_args() -> argparse.Namespace:
|
||||
parser = argparse.ArgumentParser(description="Validate taiji-AI-PAD API flows")
|
||||
parser.add_argument("--data-ingestion-url", default="http://localhost:8001", help="Data Ingestion base URL")
|
||||
parser.add_argument("--mcp-server-url", default="http://localhost:8002", help="MCP Server base URL")
|
||||
parser.add_argument("--timeout", type=int, default=30, help="HTTP timeout in seconds")
|
||||
parser.add_argument("--skip-data", action="store_true", help="Skip Data Ingestion flow")
|
||||
parser.add_argument("--skip-mcp", action="store_true", help="Skip MCP flow")
|
||||
parser.add_argument("--quiet", action="store_true", help="Suppress verbose logs")
|
||||
return parser.parse_args()
|
||||
|
||||
|
||||
def main() -> None:
|
||||
args = parse_args()
|
||||
config = ServiceConfig(
|
||||
data_ingestion_url=args.data_ingestion_url,
|
||||
mcp_server_url=args.mcp_server_url,
|
||||
timeout=args.timeout,
|
||||
)
|
||||
tester = ApiFlowTester(config, verbose=not args.quiet)
|
||||
tester.run(skip_data_ingestion=args.skip_data, skip_mcp=args.skip_mcp)
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
try:
|
||||
main()
|
||||
except (AssertionError, requests.RequestException) as exc:
|
||||
print(f"[api-flow] ❌ {exc}", file=sys.stderr)
|
||||
sys.exit(1)
|
||||
except KeyboardInterrupt:
|
||||
print("[api-flow] Interrupted", file=sys.stderr)
|
||||
sys.exit(130)
|
||||
+184
-2
@@ -13,8 +13,15 @@ SET timezone = 'UTC';
|
||||
-- 创建一些基础索引（如果表已存在的话，模型会自动创建）
|
||||
-- 这里可以添加一些额外的性能优化索引
|
||||
|
||||
-- 创建全文搜索配置
|
||||
CREATE TEXT SEARCH CONFIGURATION IF NOT EXISTS simple_english (COPY = english);
|
||||
DO $$
|
||||
BEGIN
|
||||
IF NOT EXISTS (
|
||||
SELECT 1 FROM pg_catalog.pg_ts_config WHERE cfgname = 'simple_english'
|
||||
) THEN
|
||||
EXECUTE 'CREATE TEXT SEARCH CONFIGURATION simple_english (COPY = english)';
|
||||
END IF;
|
||||
END;
|
||||
$$;
|
||||
|
||||
-- 创建一些有用的函数
|
||||
CREATE OR REPLACE FUNCTION update_updated_at_column()
|
||||
@@ -49,6 +56,7 @@ CREATE TABLE IF NOT EXISTS system_config (
|
||||
);
|
||||
|
||||
-- 创建触发器
|
||||
DROP TRIGGER IF EXISTS update_system_config_updated_at ON system_config;
|
||||
CREATE TRIGGER update_system_config_updated_at
|
||||
BEFORE UPDATE ON system_config
|
||||
FOR EACH ROW EXECUTE FUNCTION update_updated_at_column();
|
||||
@@ -105,3 +113,177 @@ SELECT
|
||||
FROM pg_stat_database
|
||||
WHERE datname = current_database();
|
||||
|
||||
-- 核心业务表：用户
|
||||
CREATE TABLE IF NOT EXISTS users (
|
||||
id UUID PRIMARY KEY DEFAULT uuid_generate_v4(),
|
||||
created_at TIMESTAMPTZ NOT NULL DEFAULT NOW(),
|
||||
updated_at TIMESTAMPTZ NOT NULL DEFAULT NOW(),
|
||||
username VARCHAR(50) NOT NULL UNIQUE,
|
||||
email VARCHAR(255) NOT NULL UNIQUE,
|
||||
hashed_password VARCHAR(255) NOT NULL,
|
||||
full_name VARCHAR(100),
|
||||
is_active BOOLEAN NOT NULL DEFAULT TRUE,
|
||||
is_admin BOOLEAN NOT NULL DEFAULT FALSE
|
||||
);
|
||||
|
||||
CREATE INDEX IF NOT EXISTS idx_user_username ON users(username);
|
||||
CREATE INDEX IF NOT EXISTS idx_user_email ON users(email);
|
||||
|
||||
-- 工具表
|
||||
CREATE TABLE IF NOT EXISTS tools (
|
||||
id UUID PRIMARY KEY DEFAULT uuid_generate_v4(),
|
||||
created_at TIMESTAMPTZ NOT NULL DEFAULT NOW(),
|
||||
updated_at TIMESTAMPTZ NOT NULL DEFAULT NOW(),
|
||||
name VARCHAR(100) NOT NULL,
|
||||
description TEXT,
|
||||
category VARCHAR(50),
|
||||
schema JSONB NOT NULL DEFAULT '{}'::jsonb,
|
||||
endpoint VARCHAR(500),
|
||||
method VARCHAR(10) NOT NULL DEFAULT 'POST',
|
||||
auth_type VARCHAR(20),
|
||||
auth_config JSONB NOT NULL DEFAULT '{}'::jsonb,
|
||||
rate_limit INTEGER NOT NULL DEFAULT 100,
|
||||
cost_per_call DOUBLE PRECISION NOT NULL DEFAULT 0,
|
||||
timeout INTEGER NOT NULL DEFAULT 30,
|
||||
is_active BOOLEAN NOT NULL DEFAULT TRUE,
|
||||
is_public BOOLEAN NOT NULL DEFAULT FALSE,
|
||||
total_calls INTEGER NOT NULL DEFAULT 0,
|
||||
success_rate DOUBLE PRECISION NOT NULL DEFAULT 0,
|
||||
avg_response_time DOUBLE PRECISION NOT NULL DEFAULT 0,
|
||||
owner_id UUID REFERENCES users(id) ON DELETE SET NULL
|
||||
);
|
||||
|
||||
CREATE INDEX IF NOT EXISTS idx_tool_name ON tools(name);
|
||||
CREATE INDEX IF NOT EXISTS idx_tool_category ON tools(category);
|
||||
CREATE INDEX IF NOT EXISTS idx_tool_active ON tools(is_active);
|
||||
|
||||
-- Agent表
|
||||
CREATE TABLE IF NOT EXISTS agents (
|
||||
id UUID PRIMARY KEY DEFAULT uuid_generate_v4(),
|
||||
created_at TIMESTAMPTZ NOT NULL DEFAULT NOW(),
|
||||
updated_at TIMESTAMPTZ NOT NULL DEFAULT NOW(),
|
||||
name VARCHAR(100) NOT NULL,
|
||||
description TEXT,
|
||||
role VARCHAR(200) NOT NULL,
|
||||
goal TEXT NOT NULL,
|
||||
config JSONB NOT NULL DEFAULT '{}'::jsonb,
|
||||
tools JSONB NOT NULL DEFAULT '[]'::jsonb,
|
||||
capabilities JSONB NOT NULL DEFAULT '[]'::jsonb,
|
||||
status VARCHAR(20) NOT NULL DEFAULT 'active',
|
||||
version VARCHAR(20) NOT NULL DEFAULT '1.0.0',
|
||||
total_executions INTEGER NOT NULL DEFAULT 0,
|
||||
success_rate DOUBLE PRECISION NOT NULL DEFAULT 0,
|
||||
avg_execution_time DOUBLE PRECISION NOT NULL DEFAULT 0,
|
||||
owner_id UUID NOT NULL REFERENCES users(id) ON DELETE CASCADE,
|
||||
CONSTRAINT uq_agent_name_owner UNIQUE (name, owner_id)
|
||||
);
|
||||
|
||||
CREATE INDEX IF NOT EXISTS idx_agent_name ON agents(name);
|
||||
CREATE INDEX IF NOT EXISTS idx_agent_owner ON agents(owner_id);
|
||||
CREATE INDEX IF NOT EXISTS idx_agent_status ON agents(status);
|
||||
|
||||
-- 会话表
|
||||
CREATE TABLE IF NOT EXISTS sessions (
|
||||
id UUID PRIMARY KEY DEFAULT uuid_generate_v4(),
|
||||
created_at TIMESTAMPTZ NOT NULL DEFAULT NOW(),
|
||||
updated_at TIMESTAMPTZ NOT NULL DEFAULT NOW(),
|
||||
session_id VARCHAR(100) NOT NULL UNIQUE,
|
||||
context JSONB NOT NULL DEFAULT '{}'::jsonb,
|
||||
session_metadata JSONB NOT NULL DEFAULT '{}'::jsonb,
|
||||
status VARCHAR(20) NOT NULL DEFAULT 'active',
|
||||
user_id UUID NOT NULL REFERENCES users(id) ON DELETE CASCADE
|
||||
);
|
||||
|
||||
CREATE INDEX IF NOT EXISTS idx_session_id ON sessions(session_id);
|
||||
CREATE INDEX IF NOT EXISTS idx_session_user ON sessions(user_id);
|
||||
CREATE INDEX IF NOT EXISTS idx_session_status ON sessions(status);
|
||||
|
||||
-- 执行记录表
|
||||
CREATE TABLE IF NOT EXISTS executions (
|
||||
id UUID PRIMARY KEY DEFAULT uuid_generate_v4(),
|
||||
created_at TIMESTAMPTZ NOT NULL DEFAULT NOW(),
|
||||
updated_at TIMESTAMPTZ NOT NULL DEFAULT NOW(),
|
||||
execution_id VARCHAR(100) NOT NULL UNIQUE,
|
||||
method VARCHAR(50) NOT NULL,
|
||||
params JSONB NOT NULL DEFAULT '{}'::jsonb,
|
||||
result JSONB NOT NULL DEFAULT '{}'::jsonb,
|
||||
error TEXT,
|
||||
started_at TIMESTAMPTZ NOT NULL DEFAULT NOW(),
|
||||
completed_at TIMESTAMPTZ,
|
||||
execution_time DOUBLE PRECISION,
|
||||
status VARCHAR(20) NOT NULL,
|
||||
cpu_usage DOUBLE PRECISION NOT NULL DEFAULT 0,
|
||||
memory_usage DOUBLE PRECISION NOT NULL DEFAULT 0,
|
||||
network_io DOUBLE PRECISION NOT NULL DEFAULT 0,
|
||||
eu_consumed DOUBLE PRECISION NOT NULL DEFAULT 0,
|
||||
agent_id UUID NOT NULL REFERENCES agents(id) ON DELETE CASCADE,
|
||||
session_id UUID REFERENCES sessions(id) ON DELETE SET NULL
|
||||
);
|
||||
|
||||
CREATE INDEX IF NOT EXISTS idx_execution_id ON executions(execution_id);
|
||||
CREATE INDEX IF NOT EXISTS idx_execution_agent ON executions(agent_id);
|
||||
CREATE INDEX IF NOT EXISTS idx_execution_status ON executions(status);
|
||||
CREATE INDEX IF NOT EXISTS idx_execution_started ON executions(started_at);
|
||||
|
||||
-- 计费记录
|
||||
CREATE TABLE IF NOT EXISTS billing (
|
||||
id UUID PRIMARY KEY DEFAULT uuid_generate_v4(),
|
||||
created_at TIMESTAMPTZ NOT NULL DEFAULT NOW(),
|
||||
updated_at TIMESTAMPTZ NOT NULL DEFAULT NOW(),
|
||||
eu_consumed DOUBLE PRECISION NOT NULL,
|
||||
cost DOUBLE PRECISION NOT NULL,
|
||||
currency VARCHAR(3) NOT NULL DEFAULT 'USD',
|
||||
cpu_time DOUBLE PRECISION NOT NULL DEFAULT 0,
|
||||
memory_max DOUBLE PRECISION NOT NULL DEFAULT 0,
|
||||
network_io DOUBLE PRECISION NOT NULL DEFAULT 0,
|
||||
storage_io DOUBLE PRECISION NOT NULL DEFAULT 0,
|
||||
execution_id UUID NOT NULL REFERENCES executions(id) ON DELETE CASCADE,
|
||||
user_id UUID NOT NULL REFERENCES users(id) ON DELETE CASCADE
|
||||
);
|
||||
|
||||
CREATE INDEX IF NOT EXISTS idx_billing_execution ON billing(execution_id);
|
||||
CREATE INDEX IF NOT EXISTS idx_billing_user ON billing(user_id);
|
||||
CREATE INDEX IF NOT EXISTS idx_billing_created ON billing(created_at);
|
||||
|
||||
-- API密钥
|
||||
CREATE TABLE IF NOT EXISTS api_keys (
|
||||
id UUID PRIMARY KEY DEFAULT uuid_generate_v4(),
|
||||
created_at TIMESTAMPTZ NOT NULL DEFAULT NOW(),
|
||||
updated_at TIMESTAMPTZ NOT NULL DEFAULT NOW(),
|
||||
name VARCHAR(100) NOT NULL,
|
||||
key_hash VARCHAR(255) NOT NULL,
|
||||
prefix VARCHAR(20) NOT NULL,
|
||||
scopes JSONB NOT NULL DEFAULT '[]'::jsonb,
|
||||
rate_limit INTEGER NOT NULL DEFAULT 1000,
|
||||
is_active BOOLEAN NOT NULL DEFAULT TRUE,
|
||||
expires_at TIMESTAMPTZ,
|
||||
last_used_at TIMESTAMPTZ,
|
||||
total_requests INTEGER NOT NULL DEFAULT 0,
|
||||
user_id UUID NOT NULL REFERENCES users(id) ON DELETE CASCADE
|
||||
);
|
||||
|
||||
CREATE UNIQUE INDEX IF NOT EXISTS idx_api_key_hash ON api_keys(key_hash);
|
||||
CREATE INDEX IF NOT EXISTS idx_api_key_prefix ON api_keys(prefix);
|
||||
CREATE INDEX IF NOT EXISTS idx_api_key_user ON api_keys(user_id);
|
||||
|
||||
-- 审计日志
|
||||
CREATE TABLE IF NOT EXISTS audit_logs (
|
||||
id UUID PRIMARY KEY DEFAULT uuid_generate_v4(),
|
||||
created_at TIMESTAMPTZ NOT NULL DEFAULT NOW(),
|
||||
updated_at TIMESTAMPTZ NOT NULL DEFAULT NOW(),
|
||||
action VARCHAR(50) NOT NULL,
|
||||
resource_type VARCHAR(50) NOT NULL,
|
||||
resource_id VARCHAR(100),
|
||||
details JSONB NOT NULL DEFAULT '{}'::jsonb,
|
||||
ip_address VARCHAR(45),
|
||||
user_agent TEXT,
|
||||
success BOOLEAN NOT NULL,
|
||||
error_message TEXT,
|
||||
user_id UUID REFERENCES users(id) ON DELETE SET NULL
|
||||
);
|
||||
|
||||
CREATE INDEX IF NOT EXISTS idx_audit_action ON audit_logs(action);
|
||||
CREATE INDEX IF NOT EXISTS idx_audit_resource ON audit_logs(resource_type, resource_id);
|
||||
CREATE INDEX IF NOT EXISTS idx_audit_user ON audit_logs(user_id);
|
||||
CREATE INDEX IF NOT EXISTS idx_audit_created ON audit_logs(created_at);
|
||||
|
||||
|
||||
@@ -1,168 +0,0 @@
|
||||
#!/bin/bash
|
||||
|
||||
# taiji-AI-PAD 启动脚本
|
||||
|
||||
set -e
|
||||
|
||||
echo "🚀 启动 taiji-AI-PAD 平台..."
|
||||
|
||||
# 检查Docker是否运行
|
||||
if ! docker info >/dev/null 2>&1; then
|
||||
echo "❌ Docker 未运行，请先启动Docker"
|
||||
exit 1
|
||||
fi
|
||||
|
||||
# 检查Docker Compose是否可用
|
||||
if ! command -v docker-compose >/dev/null 2>&1; then
|
||||
echo "❌ Docker Compose 未找到，请安装Docker Compose"
|
||||
exit 1
|
||||
fi
|
||||
|
||||
# 创建必要的目录
|
||||
echo "📁 创建必要的目录..."
|
||||
mkdir -p logs
|
||||
mkdir -p config/ssl
|
||||
mkdir -p services/model-gateway/config
|
||||
mkdir -p services/data-ingestion/models
|
||||
mkdir -p services/data-ingestion/cache
|
||||
|
||||
# 设置环境变量（如果.env文件不存在）
|
||||
if [ ! -f .env ]; then
|
||||
echo "⚙️ 创建环境配置文件..."
|
||||
cat > .env << EOF
|
||||
# 环境设置
|
||||
ENVIRONMENT=development
|
||||
|
||||
# 数据库设置
|
||||
POSTGRES_DB=taiji_db
|
||||
POSTGRES_USER=taiji_user
|
||||
POSTGRES_PASSWORD=taiji_pass
|
||||
DATABASE_URL=postgresql+asyncpg://taiji_user:taiji_pass@postgres:5432/taiji_db
|
||||
|
||||
# Redis设置
|
||||
REDIS_URL=redis://redis:6379
|
||||
|
||||
# NATS设置
|
||||
NATS_URL=nats://nats:4222
|
||||
|
||||
# LiteLLM设置
|
||||
LITELLM_MASTER_KEY=sk-taiji-master-key
|
||||
LITELLM_URL=http://litellm-gateway:4000
|
||||
|
||||
# RapidAPI设置（需要实际的API Key）
|
||||
RAPIDAPI_KEY=your-rapidapi-key-here
|
||||
RAPIDAPI_HOST=rapidapi.com
|
||||
|
||||
# APILLAMA模型设置
|
||||
APILLAMA_MODEL_PATH=/app/models/llama-3-8b-instruct
|
||||
APILLAMA_DEVICE=cpu
|
||||
EOF
|
||||
echo "✅ 环境配置文件已创建，请根据需要修改 .env 文件"
|
||||
fi
|
||||
|
||||
# 检查必要的配置文件
|
||||
if [ ! -f config/nginx.conf ]; then
|
||||
echo "❌ Nginx配置文件未找到：config/nginx.conf"
|
||||
exit 1
|
||||
fi
|
||||
|
||||
if [ ! -f scripts/init.sql ]; then
|
||||
echo "❌ 数据库初始化脚本未找到：scripts/init.sql"
|
||||
exit 1
|
||||
fi
|
||||
|
||||
# 拉取基础镜像
|
||||
echo "⬇️ 拉取基础镜像..."
|
||||
docker-compose pull postgres redis nats prometheus grafana api-gateway
|
||||
|
||||
# 构建服务镜像
|
||||
echo "🏗️ 构建服务镜像..."
|
||||
docker-compose build
|
||||
|
||||
# 启动基础设施服务
|
||||
echo "🗄️ 启动基础设施服务..."
|
||||
docker-compose up -d postgres redis nats
|
||||
|
||||
# 等待数据库就绪
|
||||
echo "⏳ 等待数据库就绪..."
|
||||
sleep 10
|
||||
|
||||
# 检查数据库连接
|
||||
echo "🔍 检查数据库连接..."
|
||||
until docker-compose exec -T postgres pg_isready -U taiji_user -d taiji_db; do
|
||||
echo "等待数据库..."
|
||||
sleep 2
|
||||
done
|
||||
|
||||
# 启动应用服务
|
||||
echo "🚀 启动应用服务..."
|
||||
docker-compose up -d
|
||||
|
||||
# 等待服务启动
|
||||
echo "⏳ 等待服务启动..."
|
||||
sleep 15
|
||||
|
||||
# 检查服务状态
|
||||
echo "🔍 检查服务状态..."
|
||||
docker-compose ps
|
||||
|
||||
# 健康检查
|
||||
echo "🏥 执行健康检查..."
|
||||
services=("mcp-server:8002" "data-ingestion:8001" "agent-registry:8003" "billing-engine:8004")
|
||||
|
||||
for service in "${services[@]}"; do
|
||||
service_name=$(echo $service | cut -d':' -f1)
|
||||
port=$(echo $service | cut -d':' -f2)
|
||||
|
||||
echo "检查 $service_name..."
|
||||
if curl -f -s http://localhost:$port/health > /dev/null; then
|
||||
echo "✅ $service_name 健康"
|
||||
else
|
||||
echo "⚠️ $service_name 可能未就绪"
|
||||
fi
|
||||
done
|
||||
|
||||
# 显示访问信息
|
||||
echo ""
|
||||
echo "🎉 taiji-AI-PAD 启动完成！"
|
||||
echo ""
|
||||
echo "📊 服务访问地址："
|
||||
echo " • API网关: http://localhost"
|
||||
echo " • MCP服务器: http://localhost:8002"
|
||||
echo " • 数据接入服务: http://localhost:8001"
|
||||
echo " • Agent注册中心: http://localhost:8003"
|
||||
echo " • 计费引擎: http://localhost:8004"
|
||||
echo " • LiteLLM网关: http://localhost:4000"
|
||||
echo ""
|
||||
echo "📈 监控服务："
|
||||
echo " • Grafana: http://localhost:3000 (admin/admin)"
|
||||
echo " • Prometheus: http://localhost:9090"
|
||||
echo ""
|
||||
echo "🗄️ 数据库服务："
|
||||
echo " • PostgreSQL: localhost:5432"
|
||||
echo " • Redis: localhost:6379"
|
||||
echo " • NATS: localhost:4222"
|
||||
echo ""
|
||||
echo "📚 API文档："
|
||||
echo " • MCP服务器: http://localhost:8002/docs"
|
||||
echo " • 数据接入服务: http://localhost:8001/docs"
|
||||
echo ""
|
||||
echo "🔧 管理命令："
|
||||
echo " • 查看日志: docker-compose logs -f [服务名]"
|
||||
echo " • 停止服务: docker-compose down"
|
||||
echo " • 重启服务: docker-compose restart [服务名]"
|
||||
echo ""
|
||||
|
||||
# 开发环境提示
|
||||
if [ "$ENVIRONMENT" = "development" ]; then
|
||||
echo "🔧 开发环境提示："
|
||||
echo " • 代码变更会自动重载"
|
||||
echo " • 日志级别设置为DEBUG"
|
||||
echo " • 请确保修改.env文件中的API密钥"
|
||||
echo ""
|
||||
fi
|
||||
|
||||
echo "🎯 接下来您可以："
|
||||
echo " 1. 访问 http://localhost:8002/docs 查看MCP API文档"
|
||||
echo " 2. 使用 scripts/test.sh 运行测试"
|
||||
echo " 3. 查看 docs/ 目录了解更多使用方法"
|
||||
@@ -1,58 +0,0 @@
|
||||
#!/bin/bash
|
||||
|
||||
# taiji-AI-PAD 停止脚本
|
||||
|
||||
set -e
|
||||
|
||||
echo "🛑 停止 taiji-AI-PAD 平台..."
|
||||
|
||||
# 检查Docker Compose是否可用
|
||||
if ! command -v docker-compose >/dev/null 2>&1; then
|
||||
echo "❌ Docker Compose 未找到"
|
||||
exit 1
|
||||
fi
|
||||
|
||||
# 显示当前运行的服务
|
||||
echo "📋 当前运行的服务："
|
||||
docker-compose ps
|
||||
|
||||
# 停止所有服务
|
||||
echo "⏹️ 停止所有服务..."
|
||||
docker-compose down
|
||||
|
||||
# 可选：清理数据卷（谨慎使用）
|
||||
if [ "$1" = "--clean" ]; then
|
||||
echo "🧹 清理数据卷..."
|
||||
read -p "⚠️ 这将删除所有数据，是否继续？ (y/N): " -n 1 -r
|
||||
echo
|
||||
if [[ $REPLY =~ ^[Yy]$ ]]; then
|
||||
docker-compose down -v
|
||||
docker system prune -f
|
||||
echo "✅ 数据卷已清理"
|
||||
else
|
||||
echo "❌ 已取消清理操作"
|
||||
fi
|
||||
fi
|
||||
|
||||
# 可选：清理镜像
|
||||
if [ "$1" = "--clean-all" ]; then
|
||||
echo "🧹 清理镜像和数据..."
|
||||
read -p "⚠️ 这将删除所有镜像和数据，是否继续？ (y/N): " -n 1 -r
|
||||
echo
|
||||
if [[ $REPLY =~ ^[Yy]$ ]]; then
|
||||
docker-compose down -v --rmi all
|
||||
docker system prune -a -f
|
||||
echo "✅ 镜像和数据已清理"
|
||||
else
|
||||
echo "❌ 已取消清理操作"
|
||||
fi
|
||||
fi
|
||||
|
||||
echo ""
|
||||
echo "✅ taiji-AI-PAD 已停止"
|
||||
echo ""
|
||||
echo "💡 清理选项："
|
||||
echo " • 清理数据卷: ./scripts/stop.sh --clean"
|
||||
echo " • 清理所有数据: ./scripts/stop.sh --clean-all"
|
||||
echo " • 重新启动: ./scripts/start.sh"
|
||||
|
||||
-242
@@ -1,242 +0,0 @@
|
||||
#!/bin/bash
|
||||
|
||||
# taiji-AI-PAD 测试脚本
|
||||
|
||||
set -e
|
||||
|
||||
echo "🧪 开始测试 taiji-AI-PAD 平台..."
|
||||
|
||||
# 检查服务是否运行
|
||||
check_service() {
|
||||
local service_name=$1
|
||||
local url=$2
|
||||
local expected_status=${3:-200}
|
||||
|
||||
echo "🔍 检查 $service_name..."
|
||||
|
||||
if curl -s -o /dev/null -w "%{http_code}" "$url" | grep -q "$expected_status"; then
|
||||
echo "✅ $service_name 正常运行"
|
||||
return 0
|
||||
else
|
||||
echo "❌ $service_name 无响应"
|
||||
return 1
|
||||
fi
|
||||
}
|
||||
|
||||
# 测试API端点
|
||||
test_api_endpoint() {
|
||||
local name=$1
|
||||
local url=$2
|
||||
local method=${3:-GET}
|
||||
local data=${4:-""}
|
||||
|
||||
echo "🧪 测试 $name..."
|
||||
|
||||
if [ -n "$data" ]; then
|
||||
response=$(curl -s -X "$method" -H "Content-Type: application/json" -d "$data" "$url" 2>/dev/null || echo "ERROR")
|
||||
else
|
||||
response=$(curl -s -X "$method" "$url" 2>/dev/null || echo "ERROR")
|
||||
fi
|
||||
|
||||
if [ "$response" = "ERROR" ]; then
|
||||
echo "❌ $name 测试失败"
|
||||
return 1
|
||||
else
|
||||
echo "✅ $name 测试通过"
|
||||
if command -v jq >/dev/null 2>&1; then
|
||||
echo " 响应: $(echo "$response" | jq -c . 2>/dev/null || echo "$response")"
|
||||
else
|
||||
echo " 响应: $response"
|
||||
fi
|
||||
return 0
|
||||
fi
|
||||
}
|
||||
|
||||
# 等待服务启动
|
||||
wait_for_services() {
|
||||
echo "⏳ 等待服务启动..."
|
||||
sleep 5
|
||||
|
||||
local max_attempts=30
|
||||
local attempt=1
|
||||
|
||||
while [ $attempt -le $max_attempts ]; do
|
||||
if curl -s http://localhost:8002/health > /dev/null 2>&1; then
|
||||
echo "✅ 服务已就绪"
|
||||
break
|
||||
fi
|
||||
|
||||
echo "等待中... ($attempt/$max_attempts)"
|
||||
sleep 2
|
||||
((attempt++))
|
||||
done
|
||||
|
||||
if [ $attempt -gt $max_attempts ]; then
|
||||
echo "❌ 服务启动超时"
|
||||
exit 1
|
||||
fi
|
||||
}
|
||||
|
||||
# 主测试流程
|
||||
main() {
|
||||
echo "🚀 taiji-AI-PAD 平台测试"
|
||||
echo "========================"
|
||||
|
||||
# 等待服务启动
|
||||
wait_for_services
|
||||
|
||||
# 基础健康检查
|
||||
echo ""
|
||||
echo "📋 基础健康检查"
|
||||
echo "----------------"
|
||||
|
||||
local services=(
|
||||
"MCP服务器:http://localhost:8002/health"
|
||||
"数据接入服务:http://localhost:8001/health"
|
||||
"API网关:http://localhost/health"
|
||||
)
|
||||
|
||||
local failed_services=0
|
||||
|
||||
for service_info in "${services[@]}"; do
|
||||
IFS=':' read -r name url <<< "$service_info"
|
||||
if ! check_service "$name" "$url"; then
|
||||
((failed_services++))
|
||||
fi
|
||||
done
|
||||
|
||||
# API功能测试
|
||||
echo ""
|
||||
echo "🔧 API功能测试"
|
||||
echo "---------------"
|
||||
|
||||
local api_tests=(
|
||||
"MCP服务器健康检查:http://localhost:8002/health:GET"
|
||||
"数据接入服务健康检查:http://localhost:8001/health:GET"
|
||||
"MCP工具列表:http://localhost:8002/tools:GET"
|
||||
"数据接入统计:http://localhost:8001/stats:GET"
|
||||
)
|
||||
|
||||
local failed_tests=0
|
||||
|
||||
for test_info in "${api_tests[@]}"; do
|
||||
IFS=':' read -r name url method <<< "$test_info"
|
||||
if ! test_api_endpoint "$name" "$url" "$method"; then
|
||||
((failed_tests++))
|
||||
fi
|
||||
done
|
||||
|
||||
# Agent创建测试
|
||||
echo ""
|
||||
echo "🤖 Agent创建测试"
|
||||
echo "----------------"
|
||||
|
||||
local agent_data='{
|
||||
"name": "test-agent",
|
||||
"description": "测试Agent",
|
||||
"role": "测试助手",
|
||||
"goal": "执行测试任务",
|
||||
"tools": ["web_search"],
|
||||
"config": {}
|
||||
}'
|
||||
|
||||
if test_api_endpoint "创建Agent" "http://localhost:8002/agents" "POST" "$agent_data"; then
|
||||
echo "🎉 Agent创建测试通过"
|
||||
else
|
||||
echo "❌ Agent创建测试失败"
|
||||
((failed_tests++))
|
||||
fi
|
||||
|
||||
# MCP协议测试
|
||||
echo ""
|
||||
echo "🔗 MCP协议测试"
|
||||
echo "--------------"
|
||||
|
||||
local mcp_request='{
|
||||
"jsonrpc": "2.0",
|
||||
"id": "test-1",
|
||||
"method": "tools/list",
|
||||
"params": {}
|
||||
}'
|
||||
|
||||
if test_api_endpoint "MCP工具列表" "http://localhost:8002/agents/test-agent/execute" "POST" "$mcp_request"; then
|
||||
echo "🎉 MCP协议测试通过"
|
||||
else
|
||||
echo "❌ MCP协议测试失败"
|
||||
((failed_tests++))
|
||||
fi
|
||||
|
||||
# 性能测试
|
||||
echo ""
|
||||
echo "⚡ 简单性能测试"
|
||||
echo "---------------"
|
||||
|
||||
echo "🔄 并发请求测试..."
|
||||
local start_time=$(date +%s%N)
|
||||
|
||||
for i in {1..10}; do
|
||||
curl -s http://localhost:8002/health > /dev/null &
|
||||
done
|
||||
wait
|
||||
|
||||
local end_time=$(date +%s%N)
|
||||
local duration=$((($end_time - $start_time) / 1000000))
|
||||
|
||||
echo "✅ 10个并发请求耗时: ${duration}ms"
|
||||
|
||||
# 负载测试（如果安装了ab）
|
||||
if command -v ab >/dev/null 2>&1; then
|
||||
echo "🚀 负载测试 (100个请求，并发10)..."
|
||||
ab -n 100 -c 10 -q http://localhost:8002/health | grep -E "(Requests per second|Time per request)"
|
||||
else
|
||||
echo "💡 提示: 安装 apache2-utils 可进行更详细的性能测试"
|
||||
fi
|
||||
|
||||
# 测试报告
|
||||
echo ""
|
||||
echo "📊 测试报告"
|
||||
echo "==========="
|
||||
|
||||
local total_services=${#services[@]}
|
||||
local total_tests=$((${#api_tests[@]} + 2)) # API测试 + Agent创建 + MCP协议
|
||||
|
||||
echo "服务检查: $((total_services - failed_services))/$total_services 通过"
|
||||
echo "功能测试: $((total_tests - failed_tests))/$total_tests 通过"
|
||||
|
||||
if [ $failed_services -eq 0 ] && [ $failed_tests -eq 0 ]; then
|
||||
echo ""
|
||||
echo "🎉 所有测试通过！taiji-AI-PAD 运行正常"
|
||||
echo ""
|
||||
echo "🔗 快速访问链接："
|
||||
echo " • MCP API文档: http://localhost:8002/docs"
|
||||
echo " • 数据接入API: http://localhost:8001/docs"
|
||||
echo " • Grafana监控: http://localhost:3000"
|
||||
echo ""
|
||||
return 0
|
||||
else
|
||||
echo ""
|
||||
echo "❌ 部分测试失败，请检查服务状态"
|
||||
echo ""
|
||||
echo "🔧 故障排除："
|
||||
echo " • 查看日志: docker-compose logs"
|
||||
echo " • 检查服务状态: docker-compose ps"
|
||||
echo " • 重启服务: docker-compose restart"
|
||||
echo ""
|
||||
return 1
|
||||
fi
|
||||
}
|
||||
|
||||
# 清理函数
|
||||
cleanup() {
|
||||
echo ""
|
||||
echo "🧹 测试清理..."
|
||||
# 删除测试创建的Agent（如果存在）
|
||||
curl -s -X DELETE http://localhost:8002/agents/test-agent > /dev/null 2>&1 || true
|
||||
echo "✅ 清理完成"
|
||||
}
|
||||
|
||||
# 设置清理陷阱
|
||||
trap cleanup EXIT
|
||||
|
||||
# 运行测试
|
||||
main "$@"
|
||||
@@ -0,0 +1,5 @@
|
||||
"""Application factory for the data-ingestion service."""
|
||||
|
||||
from .application import create_app
|
||||
|
||||
__all__ = ["create_app"]
|
||||
@@ -0,0 +1,41 @@
|
||||
"""FastAPI application factory for the data-ingestion service."""
|
||||
|
||||
from fastapi import FastAPI
|
||||
from fastapi.middleware.cors import CORSMiddleware
|
||||
|
||||
from config import Settings
|
||||
from .logging_config import configure_logging
|
||||
from .metrics import register_http_metrics
|
||||
from .lifecycle import register_lifecycle_events
|
||||
from .routes import register_routes
|
||||
from .state import get_state
|
||||
|
||||
|
||||
def create_app() -> FastAPI:
|
||||
"""Create and configure the FastAPI application instance."""
|
||||
configure_logging()
|
||||
|
||||
state = get_state()
|
||||
settings: Settings = state.settings
|
||||
|
||||
app = FastAPI(
|
||||
title="taiji-AI-PAD 数据接入服务",
|
||||
description="全域数据接入与工具化治理服务，支持RapidAPI集成和APILLAMA技术",
|
||||
version="1.0.0",
|
||||
docs_url="/docs",
|
||||
redoc_url="/redoc",
|
||||
)
|
||||
|
||||
app.add_middleware(
|
||||
CORSMiddleware,
|
||||
allow_origins=["*"],
|
||||
allow_credentials=True,
|
||||
allow_methods=["*"],
|
||||
allow_headers=["*"],
|
||||
)
|
||||
|
||||
register_http_metrics(app)
|
||||
register_lifecycle_events(app)
|
||||
register_routes(app)
|
||||
|
||||
return app
|
||||
@@ -0,0 +1,128 @@
|
||||
"""Application lifecycle hooks for the data-ingestion service."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import asyncio
|
||||
import contextlib
|
||||
import structlog
|
||||
from fastapi import FastAPI
|
||||
import nats
|
||||
import redis.asyncio as redis
|
||||
|
||||
from apillama_processor import APILLAMAProcessor
|
||||
from openapi_parser import OpenAPIParser
|
||||
from rapidapi_client import RapidAPIClient
|
||||
from tool_generator import ToolGenerator
|
||||
from .metrics import nats_connections, redis_connections, tools_registry_size
|
||||
from .state import get_state
|
||||
from .tasks.api_sync import start_background_api_sync
|
||||
|
||||
logger = structlog.get_logger(__name__)
|
||||
|
||||
|
||||
def register_lifecycle_events(app: FastAPI) -> None:
|
||||
"""Wire FastAPI startup and shutdown events."""
|
||||
|
||||
@app.on_event("startup")
|
||||
async def on_startup() -> None: # type: ignore[misc]
|
||||
state = get_state()
|
||||
settings = state.settings
|
||||
|
||||
try:
|
||||
state.redis_client = redis.from_url(
|
||||
settings.redis_url,
|
||||
encoding="utf-8",
|
||||
decode_responses=True,
|
||||
)
|
||||
await state.redis_client.ping()
|
||||
redis_connections.set(1)
|
||||
logger.info("Redis连接成功")
|
||||
|
||||
state.nats_client = await nats.connect(settings.nats_url)
|
||||
nats_connections.set(1)
|
||||
logger.info("NATS连接成功")
|
||||
|
||||
state.rapidapi_client = RapidAPIClient(
|
||||
api_key=settings.rapidapi_key,
|
||||
host=settings.rapidapi_host,
|
||||
redis_client=state.redis_client,
|
||||
)
|
||||
logger.info("RapidAPI客户端初始化完成")
|
||||
|
||||
state.apillama_processor = APILLAMAProcessor(
|
||||
model_id=settings.apillama_model_id,
|
||||
openrouter_api_key=settings.openrouter_api_key,
|
||||
openrouter_base_url=settings.openrouter_base_url,
|
||||
max_tokens=settings.apillama_max_tokens,
|
||||
temperature=settings.apillama_temperature,
|
||||
top_p=settings.apillama_top_p,
|
||||
cache_dir=settings.cache_dir,
|
||||
redis_client=state.redis_client,
|
||||
)
|
||||
await state.apillama_processor.initialize()
|
||||
logger.info("APILLAMA处理器初始化完成")
|
||||
|
||||
state.openapi_parser = OpenAPIParser(
|
||||
cache_dir=settings.cache_dir,
|
||||
redis_client=state.redis_client,
|
||||
)
|
||||
logger.info("OpenAPI解析器初始化完成")
|
||||
|
||||
state.tool_generator = ToolGenerator(
|
||||
redis_client=state.redis_client,
|
||||
nats_client=state.nats_client,
|
||||
apillama_processor=state.apillama_processor,
|
||||
)
|
||||
logger.info("工具生成器初始化完成")
|
||||
|
||||
background_task = start_background_api_sync()
|
||||
state.background_tasks.append(background_task)
|
||||
|
||||
if state.redis_client:
|
||||
tool_count = await state.redis_client.scard("tools:registry")
|
||||
tools_registry_size.set(tool_count or 0)
|
||||
|
||||
logger.info("数据接入服务启动完成")
|
||||
|
||||
except Exception as exc: # pragma: no cover - startup failures are critical
|
||||
logger.error("服务启动失败", error=str(exc))
|
||||
raise
|
||||
|
||||
@app.on_event("shutdown")
|
||||
async def on_shutdown() -> None: # type: ignore[misc]
|
||||
state = get_state()
|
||||
|
||||
for task in state.background_tasks:
|
||||
task.cancel()
|
||||
with contextlib.suppress(asyncio.CancelledError):
|
||||
await task
|
||||
state.background_tasks.clear()
|
||||
|
||||
if state.nats_client:
|
||||
with contextlib.suppress(Exception):
|
||||
await state.nats_client.close()
|
||||
nats_connections.set(0)
|
||||
state.nats_client = None
|
||||
|
||||
if state.redis_client:
|
||||
with contextlib.suppress(Exception):
|
||||
await state.redis_client.close()
|
||||
redis_connections.set(0)
|
||||
state.redis_client = None
|
||||
|
||||
if state.apillama_processor:
|
||||
with contextlib.suppress(Exception):
|
||||
await state.apillama_processor.cleanup()
|
||||
state.apillama_processor = None
|
||||
|
||||
if state.rapidapi_client:
|
||||
with contextlib.suppress(Exception):
|
||||
await state.rapidapi_client.close()
|
||||
state.rapidapi_client = None
|
||||
|
||||
if state.openapi_parser:
|
||||
with contextlib.suppress(Exception):
|
||||
await state.openapi_parser.close()
|
||||
state.openapi_parser = None
|
||||
|
||||
logger.info("资源清理完成")
|
||||
@@ -0,0 +1,31 @@
|
||||
"""Logging helpers for the data-ingestion service."""
|
||||
|
||||
import structlog
|
||||
|
||||
|
||||
_LOGGING_CONFIGURED = False
|
||||
|
||||
|
||||
def configure_logging() -> None:
|
||||
"""Configure structlog once for the service."""
|
||||
global _LOGGING_CONFIGURED
|
||||
if _LOGGING_CONFIGURED:
|
||||
return
|
||||
|
||||
structlog.configure(
|
||||
processors=[
|
||||
structlog.stdlib.filter_by_level,
|
||||
structlog.stdlib.add_logger_name,
|
||||
structlog.stdlib.add_log_level,
|
||||
structlog.stdlib.PositionalArgumentsFormatter(),
|
||||
structlog.processors.TimeStamper(fmt="iso"),
|
||||
structlog.processors.StackInfoRenderer(),
|
||||
structlog.processors.format_exc_info,
|
||||
structlog.processors.UnicodeDecoder(),
|
||||
structlog.processors.JSONRenderer(),
|
||||
],
|
||||
context_class=dict,
|
||||
logger_factory=structlog.stdlib.LoggerFactory(),
|
||||
cache_logger_on_first_use=True,
|
||||
)
|
||||
_LOGGING_CONFIGURED = True
|
||||
@@ -0,0 +1,117 @@
|
||||
"""Prometheus metrics definitions and middleware for the data-ingestion service."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import time
|
||||
from fastapi import FastAPI, Request
|
||||
from prometheus_client import Counter, Histogram, Gauge
|
||||
|
||||
|
||||
# HTTP request metrics
|
||||
http_requests_total = Counter(
|
||||
"data_ingestion_http_requests_total",
|
||||
"Total HTTP requests",
|
||||
["method", "endpoint", "status"],
|
||||
)
|
||||
|
||||
http_request_duration = Histogram(
|
||||
"data_ingestion_http_request_duration_seconds",
|
||||
"HTTP request duration",
|
||||
["method", "endpoint"],
|
||||
)
|
||||
|
||||
# API processing metrics
|
||||
rapidapi_sync_total = Counter(
|
||||
"data_ingestion_rapidapi_sync_total",
|
||||
"Total RapidAPI sync operations",
|
||||
["status"],
|
||||
)
|
||||
|
||||
rapidapi_endpoints_synced = Gauge(
|
||||
"data_ingestion_rapidapi_endpoints_synced",
|
||||
"Number of RapidAPI endpoints synced",
|
||||
)
|
||||
|
||||
apillama_processing_total = Counter(
|
||||
"data_ingestion_apillama_processing_total",
|
||||
"Total APILLAMA processing operations",
|
||||
["status"],
|
||||
)
|
||||
|
||||
apillama_processing_duration = Histogram(
|
||||
"data_ingestion_apillama_processing_duration_seconds",
|
||||
"APILLAMA processing duration",
|
||||
)
|
||||
|
||||
openapi_parse_total = Counter(
|
||||
"data_ingestion_openapi_parse_total",
|
||||
"Total OpenAPI parse operations",
|
||||
["status"],
|
||||
)
|
||||
|
||||
openapi_parse_duration = Histogram(
|
||||
"data_ingestion_openapi_parse_duration_seconds",
|
||||
"OpenAPI parse duration",
|
||||
)
|
||||
|
||||
tools_generated_total = Counter(
|
||||
"data_ingestion_tools_generated_total",
|
||||
"Total tools generated",
|
||||
["category"],
|
||||
)
|
||||
|
||||
tools_registry_size = Gauge(
|
||||
"data_ingestion_tools_registry_size",
|
||||
"Number of tools in registry",
|
||||
)
|
||||
|
||||
# Cache metrics
|
||||
cache_hits_total = Counter(
|
||||
"data_ingestion_cache_hits_total",
|
||||
"Total cache hits",
|
||||
["type"],
|
||||
)
|
||||
|
||||
cache_misses_total = Counter(
|
||||
"data_ingestion_cache_misses_total",
|
||||
"Total cache misses",
|
||||
["type"],
|
||||
)
|
||||
|
||||
# System metrics
|
||||
redis_connections = Gauge(
|
||||
"data_ingestion_redis_connections",
|
||||
"Redis connection status (1=connected, 0=disconnected)",
|
||||
)
|
||||
|
||||
nats_connections = Gauge(
|
||||
"data_ingestion_nats_connections",
|
||||
"NATS connection status (1=connected, 0=disconnected)",
|
||||
)
|
||||
|
||||
|
||||
def register_http_metrics(app: FastAPI) -> None:
|
||||
"""Attach HTTP metrics middleware to the app."""
|
||||
|
||||
@app.middleware("http")
|
||||
async def metrics_middleware(request: Request, call_next): # type: ignore[misc]
|
||||
start_time = time.time()
|
||||
method = request.method
|
||||
endpoint = request.url.path
|
||||
|
||||
try:
|
||||
response = await call_next(request)
|
||||
status = response.status_code
|
||||
except Exception:
|
||||
status = 500
|
||||
http_requests_total.labels(method=method, endpoint=endpoint, status=status).inc()
|
||||
http_request_duration.labels(method=method, endpoint=endpoint).observe(
|
||||
time.time() - start_time
|
||||
)
|
||||
raise
|
||||
|
||||
http_requests_total.labels(method=method, endpoint=endpoint, status=status).inc()
|
||||
http_request_duration.labels(method=method, endpoint=endpoint).observe(
|
||||
time.time() - start_time
|
||||
)
|
||||
return response
|
||||
@@ -0,0 +1,13 @@
|
||||
"""Local Pydantic models for the data-ingestion API layer."""
|
||||
|
||||
from typing import Dict
|
||||
from pydantic import BaseModel
|
||||
|
||||
|
||||
class HealthResponse(BaseModel):
|
||||
"""Health check payload."""
|
||||
|
||||
status: str
|
||||
timestamp: str
|
||||
services: Dict[str, str]
|
||||
stats: Dict[str, int]
|
||||
@@ -0,0 +1,19 @@
|
||||
"""Route registration for the data-ingestion service."""
|
||||
|
||||
from fastapi import FastAPI
|
||||
|
||||
from . import apillama, health, metrics, openapi, rapidapi, stats, tools
|
||||
|
||||
|
||||
def register_routes(app: FastAPI) -> None:
|
||||
"""Attach all routers to the FastAPI app."""
|
||||
for router in (
|
||||
health.router,
|
||||
rapidapi.router,
|
||||
openapi.router,
|
||||
apillama.router,
|
||||
tools.router,
|
||||
stats.router,
|
||||
metrics.router,
|
||||
):
|
||||
app.include_router(router)
|
||||
@@ -0,0 +1,73 @@
|
||||
"""APILLAMA processing endpoints."""
|
||||
|
||||
import json
|
||||
import time
|
||||
import structlog
|
||||
from fastapi import APIRouter, HTTPException
|
||||
|
||||
from schemas import APILLAMARequest, APILLAMAResponse
|
||||
from app.metrics import (
|
||||
apillama_processing_duration,
|
||||
apillama_processing_total,
|
||||
cache_hits_total,
|
||||
cache_misses_total,
|
||||
)
|
||||
from app.state import get_state
|
||||
|
||||
logger = structlog.get_logger(__name__)
|
||||
|
||||
router = APIRouter(prefix="/apillama", tags=["apillama"])
|
||||
|
||||
|
||||
@router.post("/process", response_model=APILLAMAResponse)
|
||||
async def process_api(request: APILLAMARequest) -> APILLAMAResponse:
|
||||
"""Convert API docs into structured schemas via APILLAMA."""
|
||||
state = get_state()
|
||||
processor = state.apillama_processor
|
||||
if not processor:
|
||||
raise HTTPException(status_code=500, detail="APILLAMA处理器未初始化")
|
||||
|
||||
start_time = time.time()
|
||||
try:
|
||||
api_doc = request.api_doc
|
||||
if isinstance(api_doc, str):
|
||||
try:
|
||||
api_doc = json.loads(api_doc)
|
||||
except json.JSONDecodeError:
|
||||
api_doc = {"raw": api_doc}
|
||||
|
||||
result = await processor.process_api_doc(
|
||||
api_doc=api_doc,
|
||||
context=request.context,
|
||||
output_format=request.output_format,
|
||||
)
|
||||
|
||||
processing_time = result.get("processing_time", time.time() - start_time)
|
||||
apillama_processing_duration.observe(processing_time)
|
||||
|
||||
if result.get("processed"):
|
||||
apillama_processing_total.labels(status="success").inc()
|
||||
else:
|
||||
apillama_processing_total.labels(status="error").inc()
|
||||
|
||||
if result.get("from_cache"):
|
||||
cache_hits_total.labels(type="apillama").inc()
|
||||
else:
|
||||
cache_misses_total.labels(type="apillama").inc()
|
||||
|
||||
return APILLAMAResponse(
|
||||
processed=result.get("processed", False),
|
||||
output_format=request.output_format,
|
||||
schema=result.get("schema"),
|
||||
description=result.get("description"),
|
||||
parameters=result.get("parameters", []),
|
||||
examples=result.get("examples", []),
|
||||
processing_time=processing_time,
|
||||
confidence_score=result.get("confidence_score"),
|
||||
completeness_score=result.get("completeness_score"),
|
||||
)
|
||||
except Exception as exc:
|
||||
apillama_processing_total.labels(status="error").inc()
|
||||
apillama_processing_duration.observe(time.time() - start_time)
|
||||
logger.error("APILLAMA处理失败", error=str(exc))
|
||||
raise HTTPException(status_code=500, detail=str(exc)) from exc
|
||||
@@ -0,0 +1,78 @@
|
||||
"""Health endpoints."""
|
||||
|
||||
from datetime import datetime
|
||||
import structlog
|
||||
from fastapi import APIRouter
|
||||
|
||||
from app.models import HealthResponse
|
||||
from app.state import get_state
|
||||
|
||||
logger = structlog.get_logger(__name__)
|
||||
|
||||
router = APIRouter(tags=["health"])
|
||||
|
||||
|
||||
@router.get("/health", response_model=HealthResponse)
|
||||
async def health_check() -> HealthResponse:
|
||||
"""Report service health and cache-related stats."""
|
||||
state = get_state()
|
||||
|
||||
services = {
|
||||
"data_ingestion": "healthy",
|
||||
"redis": "unknown",
|
||||
"nats": "unknown",
|
||||
"rapidapi": "unknown",
|
||||
"apillama": "unknown",
|
||||
}
|
||||
stats = {
|
||||
"total_apis": 0,
|
||||
"processed_apis": 0,
|
||||
"generated_tools": 0,
|
||||
"cache_size": 0,
|
||||
}
|
||||
|
||||
redis_client = state.redis_client
|
||||
if redis_client:
|
||||
try:
|
||||
await redis_client.ping()
|
||||
services["redis"] = "healthy"
|
||||
stats["cache_size"] = await redis_client.dbsize()
|
||||
stats["total_apis"] = await redis_client.scard("rapidapi:endpoints") or 0
|
||||
stats["processed_apis"] = await redis_client.scard("processed:apis") or 0
|
||||
stats["generated_tools"] = await redis_client.scard("tools:registry") or 0
|
||||
except Exception as exc: # pragma: no cover - best effort checks
|
||||
services["redis"] = "unhealthy"
|
||||
logger.warning("Redis健康检查失败", error=str(exc))
|
||||
else:
|
||||
services["redis"] = "unhealthy"
|
||||
|
||||
nats_client = state.nats_client
|
||||
if nats_client:
|
||||
services["nats"] = "healthy" if nats_client.is_connected else "unhealthy"
|
||||
else:
|
||||
services["nats"] = "unhealthy"
|
||||
|
||||
rapidapi_client = state.rapidapi_client
|
||||
if rapidapi_client:
|
||||
try:
|
||||
is_ok = await rapidapi_client.test_connection()
|
||||
services["rapidapi"] = "healthy" if is_ok else "degraded"
|
||||
except Exception as exc:
|
||||
services["rapidapi"] = "unhealthy"
|
||||
logger.warning("RapidAPI健康检查失败", error=str(exc))
|
||||
else:
|
||||
services["rapidapi"] = "unhealthy"
|
||||
|
||||
apillama_processor = state.apillama_processor
|
||||
if apillama_processor and apillama_processor.is_ready():
|
||||
services["apillama"] = "healthy"
|
||||
else:
|
||||
services["apillama"] = "unhealthy"
|
||||
|
||||
status = "healthy" if all(value == "healthy" for value in services.values()) else "degraded"
|
||||
return HealthResponse(
|
||||
status=status,
|
||||
timestamp=datetime.utcnow().isoformat(),
|
||||
services=services,
|
||||
stats=stats,
|
||||
)
|
||||
@@ -0,0 +1,43 @@
|
||||
"""Prometheus metrics endpoint."""
|
||||
|
||||
import structlog
|
||||
from fastapi import APIRouter
|
||||
from fastapi.responses import JSONResponse, Response
|
||||
from prometheus_client import CONTENT_TYPE_LATEST, REGISTRY, generate_latest
|
||||
|
||||
from app.metrics import nats_connections, redis_connections, tools_registry_size
|
||||
from app.state import get_state
|
||||
|
||||
logger = structlog.get_logger(__name__)
|
||||
|
||||
router = APIRouter(tags=["metrics"])
|
||||
|
||||
|
||||
@router.get("/metrics")
|
||||
async def get_metrics() -> Response:
|
||||
"""Expose Prometheus metrics with up-to-date gauges."""
|
||||
state = get_state()
|
||||
redis_client = state.redis_client
|
||||
nats_client = state.nats_client
|
||||
|
||||
try:
|
||||
if redis_client:
|
||||
try:
|
||||
await redis_client.ping()
|
||||
redis_connections.set(1)
|
||||
tool_count = await redis_client.scard("tools:registry")
|
||||
tools_registry_size.set(tool_count or 0)
|
||||
except Exception:
|
||||
redis_connections.set(0)
|
||||
else:
|
||||
redis_connections.set(0)
|
||||
|
||||
if nats_client:
|
||||
nats_connections.set(1 if nats_client.is_connected else 0)
|
||||
else:
|
||||
nats_connections.set(0)
|
||||
|
||||
return Response(content=generate_latest(REGISTRY), media_type=CONTENT_TYPE_LATEST)
|
||||
except Exception as exc:
|
||||
logger.error("获取metrics失败", error=str(exc))
|
||||
return JSONResponse({"error": str(exc)}, status_code=500)
|
||||
@@ -0,0 +1,49 @@
|
||||
"""OpenAPI parsing endpoints."""
|
||||
|
||||
import time
|
||||
import structlog
|
||||
from fastapi import APIRouter, BackgroundTasks, HTTPException
|
||||
|
||||
from schemas import APIParsedResponse
|
||||
from app.metrics import openapi_parse_duration, openapi_parse_total
|
||||
from app.state import get_state
|
||||
from app.tasks.tool_generation import generate_tools_from_spec
|
||||
|
||||
logger = structlog.get_logger(__name__)
|
||||
|
||||
router = APIRouter(prefix="/openapi", tags=["openapi"])
|
||||
|
||||
|
||||
@router.post("/parse", response_model=APIParsedResponse)
|
||||
async def parse_openapi_spec(url: str, background_tasks: BackgroundTasks) -> APIParsedResponse:
|
||||
"""Download and parse an OpenAPI document, then schedule tool generation."""
|
||||
state = get_state()
|
||||
parser = state.openapi_parser
|
||||
if not parser:
|
||||
raise HTTPException(status_code=500, detail="OpenAPI解析器未初始化")
|
||||
|
||||
start_time = time.time()
|
||||
try:
|
||||
parsed_result = await parser.parse_spec(url)
|
||||
duration = time.time() - start_time
|
||||
openapi_parse_duration.observe(duration)
|
||||
|
||||
status = "success" if parsed_result.get("parsed") else "error"
|
||||
openapi_parse_total.labels(status=status).inc()
|
||||
|
||||
background_tasks.add_task(generate_tools_from_spec, parsed_result)
|
||||
|
||||
return APIParsedResponse(
|
||||
url=url,
|
||||
title=parsed_result.get("info", {}).get("title", ""),
|
||||
version=parsed_result.get("info", {}).get("version", ""),
|
||||
endpoints_count=len(parsed_result.get("paths", {})),
|
||||
schemas_count=len(parsed_result.get("components", {}).get("schemas", {})),
|
||||
parsed_data=parsed_result,
|
||||
parsing_time=duration,
|
||||
)
|
||||
except Exception as exc:
|
||||
openapi_parse_total.labels(status="error").inc()
|
||||
openapi_parse_duration.observe(time.time() - start_time)
|
||||
logger.error("解析OpenAPI规范失败", error=str(exc))
|
||||
raise HTTPException(status_code=500, detail=str(exc)) from exc
|
||||
@@ -0,0 +1,61 @@
|
||||
"""RapidAPI integration endpoints."""
|
||||
|
||||
import structlog
|
||||
from fastapi import APIRouter, BackgroundTasks, HTTPException
|
||||
|
||||
from schemas import RapidAPIRequest
|
||||
from app.metrics import rapidapi_endpoints_synced, rapidapi_sync_total
|
||||
from app.state import get_state
|
||||
|
||||
logger = structlog.get_logger(__name__)
|
||||
|
||||
router = APIRouter(prefix="/rapidapi", tags=["rapidapi"])
|
||||
|
||||
|
||||
@router.post("/sync")
|
||||
async def sync_endpoints(
|
||||
background_tasks: BackgroundTasks,
|
||||
category: str | None = None,
|
||||
limit: int = 100,
|
||||
) -> dict:
|
||||
"""Trigger a background sync job for RapidAPI endpoints."""
|
||||
state = get_state()
|
||||
client = state.rapidapi_client
|
||||
if not client:
|
||||
raise HTTPException(status_code=500, detail="RapidAPI客户端未初始化")
|
||||
|
||||
async def _sync_task() -> None:
|
||||
try:
|
||||
result = await client.sync_endpoints(category=category, limit=limit)
|
||||
status = result.get("status", "error")
|
||||
rapidapi_sync_total.labels(status=status).inc()
|
||||
if status == "success":
|
||||
rapidapi_endpoints_synced.set(result.get("synced", 0))
|
||||
else:
|
||||
logger.error("RapidAPI同步失败", result=result)
|
||||
except Exception as exc: # pragma: no cover - background task
|
||||
rapidapi_sync_total.labels(status="error").inc()
|
||||
logger.error("后台同步任务失败", error=str(exc))
|
||||
|
||||
background_tasks.add_task(_sync_task)
|
||||
return {"message": "RapidAPI端点同步已启动", "category": category, "limit": limit}
|
||||
|
||||
|
||||
@router.post("/test")
|
||||
async def test_endpoint(request: RapidAPIRequest) -> dict:
|
||||
"""Proxy a test call to a RapidAPI endpoint."""
|
||||
state = get_state()
|
||||
client = state.rapidapi_client
|
||||
if not client:
|
||||
raise HTTPException(status_code=500, detail="RapidAPI客户端未初始化")
|
||||
|
||||
try:
|
||||
return await client.test_endpoint(
|
||||
endpoint=request.endpoint,
|
||||
method=request.method,
|
||||
params=request.params,
|
||||
headers=request.headers,
|
||||
)
|
||||
except Exception as exc:
|
||||
logger.error("测试RapidAPI端点失败", error=str(exc))
|
||||
raise HTTPException(status_code=500, detail=str(exc)) from exc
|
||||
@@ -0,0 +1,69 @@
|
||||
"""Statistics and cache management endpoints."""
|
||||
|
||||
import json
|
||||
import structlog
|
||||
from fastapi import APIRouter, HTTPException
|
||||
|
||||
from app.state import get_state
|
||||
|
||||
logger = structlog.get_logger(__name__)
|
||||
|
||||
router = APIRouter(tags=["stats"])
|
||||
|
||||
|
||||
@router.get("/stats")
|
||||
async def get_statistics() -> dict:
|
||||
"""Aggregate tool and cache statistics from Redis."""
|
||||
state = get_state()
|
||||
redis_client = state.redis_client
|
||||
if not redis_client:
|
||||
raise HTTPException(status_code=500, detail="Redis客户端未初始化")
|
||||
|
||||
try:
|
||||
stats = {
|
||||
"total_apis": await redis_client.scard("rapidapi:endpoints") or 0,
|
||||
"processed_apis": await redis_client.scard("processed:apis") or 0,
|
||||
"generated_tools": await redis_client.scard("tools:registry") or 0,
|
||||
"failed_processes": await redis_client.scard("failed:processes") or 0,
|
||||
"cache_size": await redis_client.dbsize(),
|
||||
"last_sync": await redis_client.get("last_sync_time") or "从未同步",
|
||||
}
|
||||
|
||||
categories: dict[str, int] = {}
|
||||
tool_keys = await redis_client.smembers("tools:registry")
|
||||
for tool_key in tool_keys:
|
||||
tool_data = await redis_client.get(f"tool:{tool_key}")
|
||||
if not tool_data:
|
||||
continue
|
||||
tool = json.loads(tool_data)
|
||||
category = tool.get("category", "unknown")
|
||||
categories[category] = categories.get(category, 0) + 1
|
||||
|
||||
stats["categories"] = categories
|
||||
return stats
|
||||
except Exception as exc:
|
||||
logger.error("获取统计信息失败", error=str(exc))
|
||||
raise HTTPException(status_code=500, detail=str(exc)) from exc
|
||||
|
||||
|
||||
@router.post("/cache/clear")
|
||||
async def clear_cache() -> dict:
|
||||
"""Remove processing caches while keeping tool registry entries."""
|
||||
state = get_state()
|
||||
redis_client = state.redis_client
|
||||
if not redis_client:
|
||||
raise HTTPException(status_code=500, detail="Redis客户端未初始化")
|
||||
|
||||
try:
|
||||
await redis_client.delete("processed:apis")
|
||||
await redis_client.delete("failed:processes")
|
||||
|
||||
tool_keys = await redis_client.smembers("tools:registry")
|
||||
if tool_keys:
|
||||
cache_keys = [f"tool_cache:{key}" for key in tool_keys]
|
||||
await redis_client.delete(*cache_keys)
|
||||
|
||||
return {"message": "缓存已清理"}
|
||||
except Exception as exc:
|
||||
logger.error("清理缓存失败", error=str(exc))
|
||||
raise HTTPException(status_code=500, detail=str(exc)) from exc
|
||||
@@ -0,0 +1,94 @@
|
||||
"""Tool registry endpoints."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import json
|
||||
import structlog
|
||||
from fastapi import APIRouter, BackgroundTasks, HTTPException
|
||||
|
||||
from schemas import APIEndpoint, ToolDefinition
|
||||
from app.metrics import tools_generated_total
|
||||
from app.state import get_state
|
||||
|
||||
logger = structlog.get_logger(__name__)
|
||||
|
||||
router = APIRouter(prefix="/tools", tags=["tools"])
|
||||
|
||||
|
||||
@router.post("/generate")
|
||||
async def generate_tool(endpoint: APIEndpoint, background_tasks: BackgroundTasks) -> dict:
|
||||
"""Schedule tool generation for a specific API endpoint."""
|
||||
state = get_state()
|
||||
generator = state.tool_generator
|
||||
if not generator:
|
||||
raise HTTPException(status_code=500, detail="工具生成器未初始化")
|
||||
|
||||
async def _generate() -> None:
|
||||
try:
|
||||
tool = await generator.generate_tool(endpoint)
|
||||
if tool:
|
||||
category = tool.get("category", "general")
|
||||
tools_generated_total.labels(category=category).inc()
|
||||
except Exception as exc: # pragma: no cover - background task
|
||||
logger.error("生成工具失败", error=str(exc))
|
||||
|
||||
background_tasks.add_task(_generate)
|
||||
return {"message": "工具生成任务已启动", "endpoint": endpoint.url, "method": endpoint.method}
|
||||
|
||||
|
||||
@router.get("", response_model=list[ToolDefinition])
|
||||
async def list_tools(
|
||||
category: str | None = None,
|
||||
limit: int = 100,
|
||||
offset: int = 0,
|
||||
) -> list[ToolDefinition]:
|
||||
"""Return generated tools from Redis."""
|
||||
state = get_state()
|
||||
redis_client = state.redis_client
|
||||
if not redis_client:
|
||||
raise HTTPException(status_code=500, detail="Redis客户端未初始化")
|
||||
|
||||
tool_keys = await redis_client.smembers("tools:registry")
|
||||
tools: list[ToolDefinition] = []
|
||||
|
||||
for tool_key in list(tool_keys)[offset : offset + limit]:
|
||||
tool_data = await redis_client.get(f"tool:{tool_key}")
|
||||
if not tool_data:
|
||||
continue
|
||||
tool_payload = json.loads(tool_data)
|
||||
if category and tool_payload.get("category") != category:
|
||||
continue
|
||||
tools.append(ToolDefinition(**tool_payload))
|
||||
|
||||
return tools
|
||||
|
||||
|
||||
@router.get("/{tool_name}", response_model=ToolDefinition)
|
||||
async def get_tool(tool_name: str) -> ToolDefinition:
|
||||
"""Fetch a single tool definition."""
|
||||
state = get_state()
|
||||
redis_client = state.redis_client
|
||||
if not redis_client:
|
||||
raise HTTPException(status_code=500, detail="Redis客户端未初始化")
|
||||
|
||||
tool_data = await redis_client.get(f"tool:{tool_name}")
|
||||
if not tool_data:
|
||||
raise HTTPException(status_code=404, detail="工具不存在")
|
||||
|
||||
return ToolDefinition(**json.loads(tool_data))
|
||||
|
||||
|
||||
@router.delete("/{tool_name}")
|
||||
async def delete_tool(tool_name: str) -> dict:
|
||||
"""Remove a tool definition from Redis."""
|
||||
state = get_state()
|
||||
redis_client = state.redis_client
|
||||
if not redis_client:
|
||||
raise HTTPException(status_code=500, detail="Redis客户端未初始化")
|
||||
|
||||
deleted = await redis_client.delete(f"tool:{tool_name}")
|
||||
if not deleted:
|
||||
raise HTTPException(status_code=404, detail="工具不存在")
|
||||
|
||||
await redis_client.srem("tools:registry", tool_name)
|
||||
return {"message": f"工具 {tool_name} 已删除"}
|
||||
@@ -0,0 +1,38 @@
|
||||
"""Shared application state for the data-ingestion service."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import asyncio
|
||||
from dataclasses import dataclass, field
|
||||
from typing import List, Optional
|
||||
|
||||
import nats
|
||||
import redis.asyncio as redis
|
||||
|
||||
from config import Settings
|
||||
from rapidapi_client import RapidAPIClient
|
||||
from apillama_processor import APILLAMAProcessor
|
||||
from openapi_parser import OpenAPIParser
|
||||
from tool_generator import ToolGenerator
|
||||
|
||||
|
||||
@dataclass
|
||||
class ServiceState:
|
||||
"""Container for runtime dependencies."""
|
||||
|
||||
settings: Settings = field(default_factory=Settings)
|
||||
redis_client: Optional[redis.Redis] = None
|
||||
nats_client: Optional[nats.NATS] = None
|
||||
rapidapi_client: Optional[RapidAPIClient] = None
|
||||
apillama_processor: Optional[APILLAMAProcessor] = None
|
||||
openapi_parser: Optional[OpenAPIParser] = None
|
||||
tool_generator: Optional[ToolGenerator] = None
|
||||
background_tasks: List[asyncio.Task] = field(default_factory=list)
|
||||
|
||||
|
||||
_state = ServiceState()
|
||||
|
||||
|
||||
def get_state() -> ServiceState:
|
||||
"""Return the singleton service state."""
|
||||
return _state
|
||||
@@ -0,0 +1,6 @@
|
||||
"""Background task helpers for the data-ingestion service."""
|
||||
|
||||
from .api_sync import start_background_api_sync
|
||||
from .tool_generation import generate_tools_from_spec
|
||||
|
||||
__all__ = ["start_background_api_sync", "generate_tools_from_spec"]
|
||||
@@ -0,0 +1,32 @@
|
||||
"""Background RapidAPI sync loop."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import asyncio
|
||||
import structlog
|
||||
|
||||
from app.state import get_state
|
||||
|
||||
logger = structlog.get_logger(__name__)
|
||||
|
||||
|
||||
async def _api_sync_loop(interval_seconds: int) -> None:
|
||||
"""Periodically sync popular RapidAPI endpoints."""
|
||||
while True:
|
||||
await asyncio.sleep(interval_seconds)
|
||||
state = get_state()
|
||||
client = state.rapidapi_client
|
||||
if not client:
|
||||
logger.warning("RapidAPI客户端不可用，跳过同步")
|
||||
continue
|
||||
try:
|
||||
await client.sync_popular_apis()
|
||||
logger.info("后台API同步完成")
|
||||
except Exception as exc: # pragma: no cover - background job
|
||||
logger.error("后台API同步失败", error=str(exc))
|
||||
|
||||
|
||||
def start_background_api_sync(interval_seconds: int = 3600) -> asyncio.Task:
|
||||
"""Create the asyncio task responsible for syncing RapidAPI data."""
|
||||
loop = asyncio.get_event_loop()
|
||||
return loop.create_task(_api_sync_loop(interval_seconds))
|
||||
@@ -0,0 +1,58 @@
|
||||
"""Tool generation helpers invoked via FastAPI background tasks."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
from typing import Any, Dict
|
||||
import structlog
|
||||
|
||||
from schemas import APIEndpoint
|
||||
from app.metrics import tools_generated_total, tools_registry_size
|
||||
from app.state import get_state
|
||||
|
||||
logger = structlog.get_logger(__name__)
|
||||
|
||||
|
||||
async def generate_tools_from_spec(parsed_spec: Dict[str, Any]) -> None:
|
||||
"""Iterate over parsed OpenAPI paths and push them through the tool generator."""
|
||||
state = get_state()
|
||||
generator = state.tool_generator
|
||||
redis_client = state.redis_client
|
||||
if not generator:
|
||||
logger.error("工具生成器未初始化")
|
||||
return
|
||||
|
||||
try:
|
||||
paths = parsed_spec.get("paths", {})
|
||||
servers = parsed_spec.get("servers", [{}])
|
||||
base_url = servers[0].get("url", "") if servers else ""
|
||||
|
||||
for path, methods in paths.items():
|
||||
if not isinstance(methods, dict):
|
||||
continue
|
||||
for method, spec in methods.items():
|
||||
method_upper = method.upper()
|
||||
if method_upper not in {"GET", "POST", "PUT", "DELETE", "PATCH"}:
|
||||
continue
|
||||
|
||||
endpoint = APIEndpoint(
|
||||
url=f"{base_url}{path}",
|
||||
method=method_upper,
|
||||
name=spec.get("operationId", f"{method}_{path}".replace("/", "_")),
|
||||
description=spec.get("summary", spec.get("description", "")),
|
||||
parameters=spec.get("parameters", []),
|
||||
request_body=spec.get("requestBody"),
|
||||
responses=spec.get("responses", {}),
|
||||
)
|
||||
|
||||
tool_result = await generator.generate_tool(endpoint)
|
||||
if tool_result:
|
||||
category = tool_result.get("category", "general")
|
||||
tools_generated_total.labels(category=category).inc()
|
||||
|
||||
if redis_client:
|
||||
tool_count = await redis_client.scard("tools:registry")
|
||||
tools_registry_size.set(tool_count or 0)
|
||||
|
||||
logger.info("OpenAPI工具生成完成", tool_count=len(paths))
|
||||
except Exception as exc:
|
||||
logger.error("从规范生成工具失败", error=str(exc))
|
||||
@@ -107,6 +107,7 @@ class Settings(BaseSettings):
|
||||
env_file = ".env"
|
||||
env_file_encoding = "utf-8"
|
||||
case_sensitive = False
|
||||
extra = "ignore"
|
||||
|
||||
|
||||
class DevelopmentSettings(Settings):
|
||||
|
||||
@@ -1,775 +1,18 @@
|
||||
"""
|
||||
taiji-AI-PAD 数据接入服务
|
||||
负责全域数据接入与工具化治理，包括RapidAPI集成和APILLAMA技术实现
|
||||
"""
|
||||
"""taiji-AI-PAD data-ingestion service entrypoint."""
|
||||
|
||||
import asyncio
|
||||
import json
|
||||
import logging
|
||||
import os
|
||||
import time
|
||||
from datetime import datetime
|
||||
from typing import Any, Dict, List, Optional, Union
|
||||
from app import create_app
|
||||
|
||||
import structlog
|
||||
from fastapi import FastAPI, HTTPException, BackgroundTasks, Depends, Request
|
||||
from fastapi.middleware.cors import CORSMiddleware
|
||||
from fastapi.responses import Response, JSONResponse
|
||||
from pydantic import BaseModel
|
||||
import redis.asyncio as redis
|
||||
import nats
|
||||
import httpx
|
||||
from prometheus_client import (
|
||||
Counter, Histogram, Gauge, generate_latest,
|
||||
CONTENT_TYPE_LATEST, REGISTRY
|
||||
)
|
||||
app = create_app()
|
||||
|
||||
from config import Settings
|
||||
from schemas import (
|
||||
APIEndpoint, ToolDefinition,
|
||||
RapidAPIRequest, APIParsedResponse,
|
||||
APILLAMARequest, APILLAMAResponse
|
||||
)
|
||||
from rapidapi_client import RapidAPIClient
|
||||
from apillama_processor import APILLAMAProcessor
|
||||
from openapi_parser import OpenAPIParser
|
||||
from tool_generator import ToolGenerator
|
||||
|
||||
# 配置日志
|
||||
structlog.configure(
|
||||
processors=[
|
||||
structlog.stdlib.filter_by_level,
|
||||
structlog.stdlib.add_logger_name,
|
||||
structlog.stdlib.add_log_level,
|
||||
structlog.stdlib.PositionalArgumentsFormatter(),
|
||||
structlog.processors.TimeStamper(fmt="iso"),
|
||||
structlog.processors.StackInfoRenderer(),
|
||||
structlog.processors.format_exc_info,
|
||||
structlog.processors.UnicodeDecoder(),
|
||||
structlog.processors.JSONRenderer()
|
||||
],
|
||||
context_class=dict,
|
||||
logger_factory=structlog.stdlib.LoggerFactory(),
|
||||
cache_logger_on_first_use=True,
|
||||
)
|
||||
|
||||
logger = structlog.get_logger()
|
||||
|
||||
# Prometheus Metrics
|
||||
# HTTP请求指标
|
||||
http_requests_total = Counter(
|
||||
'data_ingestion_http_requests_total',
|
||||
'Total HTTP requests',
|
||||
['method', 'endpoint', 'status']
|
||||
)
|
||||
|
||||
http_request_duration = Histogram(
|
||||
'data_ingestion_http_request_duration_seconds',
|
||||
'HTTP request duration',
|
||||
['method', 'endpoint']
|
||||
)
|
||||
|
||||
# API处理指标
|
||||
rapidapi_sync_total = Counter(
|
||||
'data_ingestion_rapidapi_sync_total',
|
||||
'Total RapidAPI sync operations',
|
||||
['status']
|
||||
)
|
||||
|
||||
rapidapi_endpoints_synced = Gauge(
|
||||
'data_ingestion_rapidapi_endpoints_synced',
|
||||
'Number of RapidAPI endpoints synced'
|
||||
)
|
||||
|
||||
apillama_processing_total = Counter(
|
||||
'data_ingestion_apillama_processing_total',
|
||||
'Total APILLAMA processing operations',
|
||||
['status']
|
||||
)
|
||||
|
||||
apillama_processing_duration = Histogram(
|
||||
'data_ingestion_apillama_processing_duration_seconds',
|
||||
'APILLAMA processing duration'
|
||||
)
|
||||
|
||||
openapi_parse_total = Counter(
|
||||
'data_ingestion_openapi_parse_total',
|
||||
'Total OpenAPI parse operations',
|
||||
['status']
|
||||
)
|
||||
|
||||
openapi_parse_duration = Histogram(
|
||||
'data_ingestion_openapi_parse_duration_seconds',
|
||||
'OpenAPI parse duration'
|
||||
)
|
||||
|
||||
tools_generated_total = Counter(
|
||||
'data_ingestion_tools_generated_total',
|
||||
'Total tools generated',
|
||||
['category']
|
||||
)
|
||||
|
||||
tools_registry_size = Gauge(
|
||||
'data_ingestion_tools_registry_size',
|
||||
'Number of tools in registry'
|
||||
)
|
||||
|
||||
# 缓存指标
|
||||
cache_hits_total = Counter(
|
||||
'data_ingestion_cache_hits_total',
|
||||
'Total cache hits',
|
||||
['type']
|
||||
)
|
||||
|
||||
cache_misses_total = Counter(
|
||||
'data_ingestion_cache_misses_total',
|
||||
'Total cache misses',
|
||||
['type']
|
||||
)
|
||||
|
||||
# 系统指标
|
||||
redis_connections = Gauge(
|
||||
'data_ingestion_redis_connections',
|
||||
'Redis connection status (1=connected, 0=disconnected)'
|
||||
)
|
||||
|
||||
nats_connections = Gauge(
|
||||
'data_ingestion_nats_connections',
|
||||
'NATS connection status (1=connected, 0=disconnected)'
|
||||
)
|
||||
|
||||
# 应用设置
|
||||
settings = Settings()
|
||||
app = FastAPI(
|
||||
title="taiji-AI-PAD 数据接入服务",
|
||||
description="全域数据接入与工具化治理服务，支持RapidAPI集成和APILLAMA技术",
|
||||
version="1.0.0",
|
||||
docs_url="/docs",
|
||||
redoc_url="/redoc"
|
||||
)
|
||||
|
||||
# CORS配置
|
||||
app.add_middleware(
|
||||
CORSMiddleware,
|
||||
allow_origins=["*"],
|
||||
allow_credentials=True,
|
||||
allow_methods=["*"],
|
||||
allow_headers=["*"],
|
||||
)
|
||||
|
||||
# Prometheus Metrics中间件
|
||||
@app.middleware("http")
|
||||
async def metrics_middleware(request: Request, call_next):
|
||||
"""收集HTTP请求指标"""
|
||||
start_time = time.time()
|
||||
method = request.method
|
||||
endpoint = request.url.path
|
||||
|
||||
try:
|
||||
response = await call_next(request)
|
||||
status = response.status_code
|
||||
|
||||
# 记录指标
|
||||
http_requests_total.labels(method=method, endpoint=endpoint, status=status).inc()
|
||||
http_request_duration.labels(method=method, endpoint=endpoint).observe(time.time() - start_time)
|
||||
|
||||
return response
|
||||
except Exception as e:
|
||||
status = 500
|
||||
http_requests_total.labels(method=method, endpoint=endpoint, status=status).inc()
|
||||
http_request_duration.labels(method=method, endpoint=endpoint).observe(time.time() - start_time)
|
||||
raise
|
||||
|
||||
# 全局变量
|
||||
redis_client: Optional[redis.Redis] = None
|
||||
nats_client: Optional[nats.NATS] = None
|
||||
rapidapi_client: Optional[RapidAPIClient] = None
|
||||
apillama_processor: Optional[APILLAMAProcessor] = None
|
||||
openapi_parser: Optional[OpenAPIParser] = None
|
||||
tool_generator: Optional[ToolGenerator] = None
|
||||
|
||||
class HealthResponse(BaseModel):
|
||||
status: str
|
||||
timestamp: str
|
||||
services: Dict[str, str]
|
||||
stats: Dict[str, int]
|
||||
|
||||
@app.on_event("startup")
|
||||
async def startup_event():
|
||||
"""应用启动初始化"""
|
||||
global redis_client, nats_client, rapidapi_client
|
||||
global apillama_processor, openapi_parser, tool_generator
|
||||
|
||||
try:
|
||||
# 连接Redis
|
||||
redis_client = redis.from_url(
|
||||
settings.redis_url,
|
||||
encoding="utf-8",
|
||||
decode_responses=True
|
||||
)
|
||||
await redis_client.ping()
|
||||
redis_connections.set(1)
|
||||
logger.info("Redis连接成功")
|
||||
|
||||
# 连接NATS
|
||||
nats_client = await nats.connect(settings.nats_url)
|
||||
nats_connections.set(1)
|
||||
logger.info("NATS连接成功")
|
||||
|
||||
# 初始化RapidAPI客户端
|
||||
rapidapi_client = RapidAPIClient(
|
||||
api_key=settings.rapidapi_key,
|
||||
host=settings.rapidapi_host,
|
||||
redis_client=redis_client
|
||||
)
|
||||
logger.info("RapidAPI客户端初始化完成")
|
||||
|
||||
# 初始化APILLAMA处理器（使用OpenRouter API）
|
||||
apillama_processor = APILLAMAProcessor(
|
||||
model_id=settings.apillama_model_id,
|
||||
openrouter_api_key=settings.openrouter_api_key,
|
||||
openrouter_base_url=settings.openrouter_base_url,
|
||||
max_tokens=settings.apillama_max_tokens,
|
||||
temperature=settings.apillama_temperature,
|
||||
top_p=settings.apillama_top_p,
|
||||
cache_dir=settings.cache_dir,
|
||||
redis_client=redis_client
|
||||
)
|
||||
await apillama_processor.initialize()
|
||||
logger.info("APILLAMA处理器初始化完成")
|
||||
|
||||
# 初始化OpenAPI解析器
|
||||
openapi_parser = OpenAPIParser(
|
||||
cache_dir=settings.cache_dir,
|
||||
redis_client=redis_client
|
||||
)
|
||||
logger.info("OpenAPI解析器初始化完成")
|
||||
|
||||
# 初始化工具生成器
|
||||
tool_generator = ToolGenerator(
|
||||
redis_client=redis_client,
|
||||
nats_client=nats_client,
|
||||
apillama_processor=apillama_processor
|
||||
)
|
||||
logger.info("工具生成器初始化完成")
|
||||
|
||||
# 启动后台任务
|
||||
asyncio.create_task(background_api_sync())
|
||||
|
||||
logger.info("数据接入服务启动完成")
|
||||
|
||||
except Exception as e:
|
||||
logger.error(f"服务启动失败: {e}")
|
||||
raise
|
||||
|
||||
@app.on_event("shutdown")
|
||||
async def shutdown_event():
|
||||
"""应用关闭清理"""
|
||||
global redis_client, nats_client, apillama_processor
|
||||
global rapidapi_client, openapi_parser
|
||||
|
||||
try:
|
||||
# 关闭NATS连接
|
||||
if nats_client:
|
||||
await nats_client.close()
|
||||
nats_connections.set(0)
|
||||
|
||||
# 关闭Redis连接
|
||||
if redis_client:
|
||||
await redis_client.close()
|
||||
redis_connections.set(0)
|
||||
|
||||
# 清理APILLAMA处理器
|
||||
if apillama_processor:
|
||||
await apillama_processor.cleanup()
|
||||
|
||||
# 关闭RapidAPI客户端
|
||||
if rapidapi_client:
|
||||
await rapidapi_client.close()
|
||||
|
||||
# 关闭OpenAPI解析器
|
||||
if openapi_parser:
|
||||
await openapi_parser.close()
|
||||
|
||||
logger.info("资源清理完成")
|
||||
|
||||
except Exception as e:
|
||||
logger.error(f"资源清理失败: {e}")
|
||||
|
||||
@app.get("/health", response_model=HealthResponse)
|
||||
async def health_check():
|
||||
"""健康检查端点"""
|
||||
services = {
|
||||
"data_ingestion": "healthy",
|
||||
"redis": "unknown",
|
||||
"nats": "unknown",
|
||||
"rapidapi": "unknown",
|
||||
"apillama": "unknown"
|
||||
}
|
||||
|
||||
stats = {
|
||||
"total_apis": 0,
|
||||
"processed_apis": 0,
|
||||
"generated_tools": 0,
|
||||
"cache_size": 0
|
||||
}
|
||||
|
||||
try:
|
||||
# 检查Redis
|
||||
if redis_client:
|
||||
await redis_client.ping()
|
||||
services["redis"] = "healthy"
|
||||
|
||||
# 获取统计信息
|
||||
stats["cache_size"] = await redis_client.dbsize()
|
||||
stats["total_apis"] = await redis_client.scard("rapidapi:endpoints") or 0
|
||||
stats["processed_apis"] = await redis_client.scard("processed:apis") or 0
|
||||
stats["generated_tools"] = await redis_client.scard("tools:registry") or 0
|
||||
except Exception:
|
||||
services["redis"] = "unhealthy"
|
||||
|
||||
try:
|
||||
# 检查NATS
|
||||
if nats_client and nats_client.is_connected:
|
||||
services["nats"] = "healthy"
|
||||
except Exception:
|
||||
services["nats"] = "unhealthy"
|
||||
|
||||
try:
|
||||
# 检查RapidAPI
|
||||
if rapidapi_client:
|
||||
await rapidapi_client.test_connection()
|
||||
services["rapidapi"] = "healthy"
|
||||
except Exception:
|
||||
services["rapidapi"] = "unhealthy"
|
||||
|
||||
try:
|
||||
# 检查APILLAMA
|
||||
if apillama_processor and apillama_processor.is_ready():
|
||||
services["apillama"] = "healthy"
|
||||
except Exception:
|
||||
services["apillama"] = "unhealthy"
|
||||
|
||||
return HealthResponse(
|
||||
status="healthy" if all(s == "healthy" for s in services.values()) else "degraded",
|
||||
timestamp=datetime.utcnow().isoformat(),
|
||||
services=services,
|
||||
stats=stats
|
||||
)
|
||||
|
||||
@app.post("/rapidapi/sync")
|
||||
async def sync_rapidapi_endpoints(
|
||||
background_tasks: BackgroundTasks,
|
||||
category: Optional[str] = None,
|
||||
limit: int = 100
|
||||
):
|
||||
"""同步RapidAPI端点"""
|
||||
try:
|
||||
if not rapidapi_client:
|
||||
raise HTTPException(status_code=500, detail="RapidAPI客户端未初始化")
|
||||
|
||||
# 启动后台同步任务
|
||||
async def sync_task():
|
||||
try:
|
||||
result = await rapidapi_client.sync_endpoints(
|
||||
category=category,
|
||||
limit=limit
|
||||
)
|
||||
if result.get("status") == "success":
|
||||
rapidapi_sync_total.labels(status="success").inc()
|
||||
rapidapi_endpoints_synced.set(result.get("synced", 0))
|
||||
else:
|
||||
rapidapi_sync_total.labels(status="error").inc()
|
||||
except Exception as e:
|
||||
rapidapi_sync_total.labels(status="error").inc()
|
||||
logger.error(f"后台同步任务失败: {e}")
|
||||
|
||||
background_tasks.add_task(sync_task)
|
||||
|
||||
return {
|
||||
"message": "RapidAPI端点同步已启动",
|
||||
"category": category,
|
||||
"limit": limit
|
||||
}
|
||||
|
||||
except Exception as e:
|
||||
logger.error(f"同步RapidAPI端点失败: {e}")
|
||||
rapidapi_sync_total.labels(status="error").inc()
|
||||
raise HTTPException(status_code=500, detail=str(e))
|
||||
|
||||
@app.post("/rapidapi/test")
|
||||
async def test_rapidapi_endpoint(request: RapidAPIRequest):
|
||||
"""测试RapidAPI端点"""
|
||||
try:
|
||||
if not rapidapi_client:
|
||||
raise HTTPException(status_code=500, detail="RapidAPI客户端未初始化")
|
||||
|
||||
result = await rapidapi_client.test_endpoint(
|
||||
endpoint=request.endpoint,
|
||||
method=request.method,
|
||||
params=request.params,
|
||||
headers=request.headers
|
||||
)
|
||||
|
||||
return result
|
||||
|
||||
except Exception as e:
|
||||
logger.error(f"测试RapidAPI端点失败: {e}")
|
||||
raise HTTPException(status_code=500, detail=str(e))
|
||||
|
||||
@app.post("/openapi/parse", response_model=APIParsedResponse)
|
||||
async def parse_openapi_spec(
|
||||
url: str,
|
||||
background_tasks: BackgroundTasks
|
||||
):
|
||||
"""解析OpenAPI规范文档"""
|
||||
start_time = time.time()
|
||||
try:
|
||||
if not openapi_parser:
|
||||
raise HTTPException(status_code=500, detail="OpenAPI解析器未初始化")
|
||||
|
||||
# 解析OpenAPI文档
|
||||
parsed_result = await openapi_parser.parse_spec(url)
|
||||
|
||||
parse_duration = time.time() - start_time
|
||||
openapi_parse_duration.observe(parse_duration)
|
||||
|
||||
if parsed_result.get("parsed"):
|
||||
openapi_parse_total.labels(status="success").inc()
|
||||
else:
|
||||
openapi_parse_total.labels(status="error").inc()
|
||||
|
||||
# 启动后台工具生成任务
|
||||
background_tasks.add_task(
|
||||
generate_tools_from_spec,
|
||||
parsed_result
|
||||
)
|
||||
|
||||
return APIParsedResponse(
|
||||
url=url,
|
||||
title=parsed_result.get("info", {}).get("title", ""),
|
||||
version=parsed_result.get("info", {}).get("version", ""),
|
||||
endpoints_count=len(parsed_result.get("paths", {})),
|
||||
schemas_count=len(parsed_result.get("components", {}).get("schemas", {})),
|
||||
parsed_data=parsed_result,
|
||||
parsing_time=parse_duration
|
||||
)
|
||||
|
||||
except Exception as e:
|
||||
openapi_parse_total.labels(status="error").inc()
|
||||
openapi_parse_duration.observe(time.time() - start_time)
|
||||
logger.error(f"解析OpenAPI规范失败: {e}")
|
||||
raise HTTPException(status_code=500, detail=str(e))
|
||||
|
||||
@app.post("/apillama/process", response_model=APILLAMAResponse)
|
||||
async def process_api_with_apillama(request: APILLAMARequest):
|
||||
"""使用APILLAMA处理API文档"""
|
||||
start_time = time.time()
|
||||
try:
|
||||
if not apillama_processor:
|
||||
raise HTTPException(status_code=500, detail="APILLAMA处理器未初始化")
|
||||
|
||||
# 处理api_doc（可能是字符串或字典）
|
||||
api_doc = request.api_doc
|
||||
if isinstance(api_doc, str):
|
||||
try:
|
||||
api_doc = json.loads(api_doc)
|
||||
except:
|
||||
api_doc = {"raw": api_doc}
|
||||
|
||||
result = await apillama_processor.process_api_doc(
|
||||
api_doc=api_doc,
|
||||
context=request.context,
|
||||
output_format=request.output_format
|
||||
)
|
||||
|
||||
processing_time = result.get("processing_time", time.time() - start_time)
|
||||
apillama_processing_duration.observe(processing_time)
|
||||
|
||||
if result.get("processed"):
|
||||
apillama_processing_total.labels(status="success").inc()
|
||||
else:
|
||||
apillama_processing_total.labels(status="error").inc()
|
||||
|
||||
# 记录缓存命中
|
||||
if result.get("from_cache"):
|
||||
cache_hits_total.labels(type="apillama").inc()
|
||||
else:
|
||||
cache_misses_total.labels(type="apillama").inc()
|
||||
|
||||
return APILLAMAResponse(
|
||||
processed=result.get("processed", False),
|
||||
output_format=request.output_format,
|
||||
schema=result.get("schema"),
|
||||
description=result.get("description"),
|
||||
parameters=result.get("parameters", []),
|
||||
examples=result.get("examples", []),
|
||||
processing_time=processing_time,
|
||||
confidence_score=result.get("confidence_score"),
|
||||
completeness_score=result.get("completeness_score")
|
||||
)
|
||||
|
||||
except Exception as e:
|
||||
apillama_processing_total.labels(status="error").inc()
|
||||
apillama_processing_duration.observe(time.time() - start_time)
|
||||
logger.error(f"APILLAMA处理失败: {e}")
|
||||
raise HTTPException(status_code=500, detail=str(e))
|
||||
|
||||
@app.post("/tools/generate")
|
||||
async def generate_tool_from_endpoint(
|
||||
endpoint: APIEndpoint,
|
||||
background_tasks: BackgroundTasks
|
||||
):
|
||||
"""从API端点生成工具定义"""
|
||||
try:
|
||||
if not tool_generator:
|
||||
raise HTTPException(status_code=500, detail="工具生成器未初始化")
|
||||
|
||||
# 启动后台工具生成任务
|
||||
background_tasks.add_task(
|
||||
tool_generator.generate_tool,
|
||||
endpoint
|
||||
)
|
||||
|
||||
return {
|
||||
"message": "工具生成任务已启动",
|
||||
"endpoint": endpoint.url,
|
||||
"method": endpoint.method
|
||||
}
|
||||
|
||||
except Exception as e:
|
||||
logger.error(f"生成工具失败: {e}")
|
||||
raise HTTPException(status_code=500, detail=str(e))
|
||||
|
||||
@app.get("/tools", response_model=List[ToolDefinition])
|
||||
async def list_generated_tools(
|
||||
category: Optional[str] = None,
|
||||
limit: int = 100,
|
||||
offset: int = 0
|
||||
):
|
||||
"""获取生成的工具列表"""
|
||||
try:
|
||||
if not redis_client:
|
||||
raise HTTPException(status_code=500, detail="Redis客户端未初始化")
|
||||
|
||||
tools = []
|
||||
tool_keys = await redis_client.smembers("tools:registry")
|
||||
|
||||
for tool_key in list(tool_keys)[offset:offset+limit]:
|
||||
tool_data = await redis_client.get(f"tool:{tool_key}")
|
||||
if tool_data:
|
||||
tool = json.loads(tool_data)
|
||||
if not category or tool.get("category") == category:
|
||||
tools.append(ToolDefinition(**tool))
|
||||
|
||||
return tools
|
||||
|
||||
except Exception as e:
|
||||
logger.error(f"获取工具列表失败: {e}")
|
||||
raise HTTPException(status_code=500, detail=str(e))
|
||||
|
||||
@app.get("/tools/{tool_name}", response_model=ToolDefinition)
|
||||
async def get_tool_definition(tool_name: str):
|
||||
"""获取特定工具定义"""
|
||||
try:
|
||||
if not redis_client:
|
||||
raise HTTPException(status_code=500, detail="Redis客户端未初始化")
|
||||
|
||||
tool_data = await redis_client.get(f"tool:{tool_name}")
|
||||
if not tool_data:
|
||||
raise HTTPException(status_code=404, detail="工具不存在")
|
||||
|
||||
tool = json.loads(tool_data)
|
||||
return ToolDefinition(**tool)
|
||||
|
||||
except HTTPException:
|
||||
raise
|
||||
except Exception as e:
|
||||
logger.error(f"获取工具定义失败: {e}")
|
||||
raise HTTPException(status_code=500, detail=str(e))
|
||||
|
||||
@app.delete("/tools/{tool_name}")
|
||||
async def delete_tool(tool_name: str):
|
||||
"""删除工具定义"""
|
||||
try:
|
||||
if not redis_client:
|
||||
raise HTTPException(status_code=500, detail="Redis客户端未初始化")
|
||||
|
||||
# 删除工具数据
|
||||
deleted = await redis_client.delete(f"tool:{tool_name}")
|
||||
if not deleted:
|
||||
raise HTTPException(status_code=404, detail="工具不存在")
|
||||
|
||||
# 从注册表中移除
|
||||
await redis_client.srem("tools:registry", tool_name)
|
||||
|
||||
return {"message": f"工具 {tool_name} 已删除"}
|
||||
|
||||
except HTTPException:
|
||||
raise
|
||||
except Exception as e:
|
||||
logger.error(f"删除工具失败: {e}")
|
||||
raise HTTPException(status_code=500, detail=str(e))
|
||||
|
||||
@app.get("/stats")
|
||||
async def get_statistics():
|
||||
"""获取统计信息"""
|
||||
try:
|
||||
if not redis_client:
|
||||
raise HTTPException(status_code=500, detail="Redis客户端未初始化")
|
||||
|
||||
stats = {
|
||||
"total_apis": await redis_client.scard("rapidapi:endpoints") or 0,
|
||||
"processed_apis": await redis_client.scard("processed:apis") or 0,
|
||||
"generated_tools": await redis_client.scard("tools:registry") or 0,
|
||||
"failed_processes": await redis_client.scard("failed:processes") or 0,
|
||||
"cache_size": await redis_client.dbsize(),
|
||||
"last_sync": await redis_client.get("last_sync_time") or "从未同步"
|
||||
}
|
||||
|
||||
# 获取分类统计
|
||||
categories = {}
|
||||
tool_keys = await redis_client.smembers("tools:registry")
|
||||
for tool_key in tool_keys:
|
||||
tool_data = await redis_client.get(f"tool:{tool_key}")
|
||||
if tool_data:
|
||||
tool = json.loads(tool_data)
|
||||
category = tool.get("category", "unknown")
|
||||
categories[category] = categories.get(category, 0) + 1
|
||||
|
||||
stats["categories"] = categories
|
||||
|
||||
return stats
|
||||
|
||||
except Exception as e:
|
||||
logger.error(f"获取统计信息失败: {e}")
|
||||
raise HTTPException(status_code=500, detail=str(e))
|
||||
|
||||
@app.post("/cache/clear")
|
||||
async def clear_cache():
|
||||
"""清理缓存"""
|
||||
try:
|
||||
if not redis_client:
|
||||
raise HTTPException(status_code=500, detail="Redis客户端未初始化")
|
||||
|
||||
# 清理处理缓存
|
||||
await redis_client.delete("processed:apis")
|
||||
await redis_client.delete("failed:processes")
|
||||
|
||||
# 清理工具缓存（保留工具注册表）
|
||||
tool_keys = await redis_client.smembers("tools:registry")
|
||||
if tool_keys:
|
||||
cache_keys = [f"tool_cache:{key}" for key in tool_keys]
|
||||
await redis_client.delete(*cache_keys)
|
||||
|
||||
return {"message": "缓存已清理"}
|
||||
|
||||
except Exception as e:
|
||||
logger.error(f"清理缓存失败: {e}")
|
||||
raise HTTPException(status_code=500, detail=str(e))
|
||||
|
||||
async def generate_tools_from_spec(parsed_spec: Dict[str, Any]):
|
||||
"""从解析的OpenAPI规范生成工具"""
|
||||
try:
|
||||
if not tool_generator:
|
||||
logger.error("工具生成器未初始化")
|
||||
return
|
||||
|
||||
paths = parsed_spec.get("paths", {})
|
||||
|
||||
for path, methods in paths.items():
|
||||
for method, spec in methods.items():
|
||||
if method.upper() in ["GET", "POST", "PUT", "DELETE", "PATCH"]:
|
||||
endpoint = APIEndpoint(
|
||||
url=f"{parsed_spec.get('servers', [{}])[0].get('url', '')}{path}",
|
||||
method=method.upper(),
|
||||
name=spec.get("operationId", f"{method}_{path}".replace("/", "_")),
|
||||
description=spec.get("summary", spec.get("description", "")),
|
||||
parameters=spec.get("parameters", []),
|
||||
request_body=spec.get("requestBody"),
|
||||
responses=spec.get("responses", {})
|
||||
)
|
||||
|
||||
tool_result = await tool_generator.generate_tool(endpoint)
|
||||
if tool_result:
|
||||
category = tool_result.get("category", "general")
|
||||
tools_generated_total.labels(category=category).inc()
|
||||
|
||||
# 更新工具注册表大小
|
||||
if redis_client:
|
||||
tool_count = await redis_client.scard("tools:registry")
|
||||
tools_registry_size.set(tool_count)
|
||||
|
||||
logger.info(f"从OpenAPI规范生成了 {len(paths)} 个工具")
|
||||
|
||||
except Exception as e:
|
||||
logger.error(f"从规范生成工具失败: {e}")
|
||||
|
||||
async def background_api_sync():
|
||||
"""后台API同步任务"""
|
||||
while True:
|
||||
try:
|
||||
await asyncio.sleep(3600) # 每小时同步一次
|
||||
|
||||
if rapidapi_client:
|
||||
await rapidapi_client.sync_popular_apis()
|
||||
logger.info("后台API同步完成")
|
||||
|
||||
except Exception as e:
|
||||
logger.error(f"后台API同步失败: {e}")
|
||||
|
||||
@app.get("/metrics")
|
||||
async def get_metrics():
|
||||
"""Prometheus metrics端点"""
|
||||
try:
|
||||
# 更新动态指标
|
||||
if redis_client:
|
||||
try:
|
||||
await redis_client.ping()
|
||||
redis_connections.set(1)
|
||||
except:
|
||||
redis_connections.set(0)
|
||||
else:
|
||||
redis_connections.set(0)
|
||||
|
||||
if nats_client:
|
||||
try:
|
||||
if nats_client.is_connected:
|
||||
nats_connections.set(1)
|
||||
else:
|
||||
nats_connections.set(0)
|
||||
except:
|
||||
nats_connections.set(0)
|
||||
else:
|
||||
nats_connections.set(0)
|
||||
|
||||
# 更新工具注册表大小
|
||||
if redis_client:
|
||||
try:
|
||||
tool_count = await redis_client.scard("tools:registry")
|
||||
tools_registry_size.set(tool_count)
|
||||
except:
|
||||
pass
|
||||
|
||||
# 生成Prometheus格式的指标
|
||||
return Response(
|
||||
content=generate_latest(REGISTRY),
|
||||
media_type=CONTENT_TYPE_LATEST
|
||||
)
|
||||
except Exception as e:
|
||||
logger.error(f"获取metrics失败: {e}")
|
||||
return JSONResponse(
|
||||
{"error": str(e)},
|
||||
status_code=500
|
||||
)
|
||||
|
||||
if __name__ == "__main__":
|
||||
import uvicorn
|
||||
|
||||
uvicorn.run(
|
||||
"main:app",
|
||||
host="0.0.0.0",
|
||||
port=8000,
|
||||
reload=True,
|
||||
log_level="info"
|
||||
log_level="info",
|
||||
)
|
||||
|
||||
|
||||
@@ -0,0 +1,5 @@
|
||||
"""Application factory for the MCP server."""
|
||||
|
||||
from .application import create_app
|
||||
|
||||
__all__ = ["create_app"]
|
||||
@@ -0,0 +1,38 @@
|
||||
"""FastAPI application factory for the MCP server."""
|
||||
|
||||
from fastapi import FastAPI
|
||||
from fastapi.middleware.cors import CORSMiddleware
|
||||
|
||||
from .logging_config import configure_logging
|
||||
from .metrics import register_http_metrics
|
||||
from .lifecycle import register_lifecycle_events
|
||||
from .routes import register_routes
|
||||
from .state import get_state
|
||||
|
||||
|
||||
def create_app() -> FastAPI:
|
||||
"""Create and configure the FastAPI application."""
|
||||
configure_logging()
|
||||
settings = get_state().settings
|
||||
|
||||
app = FastAPI(
|
||||
title="taiji-AI-PAD MCP Server",
|
||||
description="Model Context Protocol Server for Agent Management",
|
||||
version="1.0.0",
|
||||
docs_url="/docs",
|
||||
redoc_url="/redoc",
|
||||
)
|
||||
|
||||
app.add_middleware(
|
||||
CORSMiddleware,
|
||||
allow_origins=settings.cors_origins or ["*"],
|
||||
allow_credentials=True,
|
||||
allow_methods=["*"],
|
||||
allow_headers=["*"],
|
||||
)
|
||||
|
||||
register_http_metrics(app)
|
||||
register_lifecycle_events(app)
|
||||
register_routes(app)
|
||||
|
||||
return app
|
||||
@@ -0,0 +1,53 @@
|
||||
"""NATS event handlers for the MCP server."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import json
|
||||
import structlog
|
||||
|
||||
from .state import get_state
|
||||
|
||||
logger = structlog.get_logger(__name__)
|
||||
|
||||
|
||||
async def setup_nats_handlers() -> None:
|
||||
"""Subscribe to NATS subjects once the client is connected."""
|
||||
state = get_state()
|
||||
client = state.nats_client
|
||||
if not client:
|
||||
return
|
||||
|
||||
await client.subscribe("agent.execution.*", cb=handle_agent_execution)
|
||||
await client.subscribe("billing.*", cb=handle_billing_event)
|
||||
await client.subscribe("system.*", cb=handle_system_event)
|
||||
|
||||
|
||||
async def handle_agent_execution(msg) -> None:
|
||||
"""Broadcast agent execution events to WebSocket clients."""
|
||||
state = get_state()
|
||||
try:
|
||||
payload = json.loads(msg.data.decode())
|
||||
logger.info("收到Agent执行事件", payload=payload)
|
||||
|
||||
for websocket in list(state.active_websockets.values()):
|
||||
await websocket.send_json({"type": "agent_execution", "data": payload})
|
||||
except Exception as exc: # pragma: no cover - defensive logging
|
||||
logger.error("处理Agent执行事件失败", error=str(exc))
|
||||
|
||||
|
||||
async def handle_billing_event(msg) -> None:
|
||||
"""Log billing events published on NATS."""
|
||||
try:
|
||||
payload = json.loads(msg.data.decode())
|
||||
logger.info("收到计费事件", payload=payload)
|
||||
except Exception as exc:
|
||||
logger.error("处理计费事件失败", error=str(exc))
|
||||
|
||||
|
||||
async def handle_system_event(msg) -> None:
|
||||
"""Log system-wide events."""
|
||||
try:
|
||||
payload = json.loads(msg.data.decode())
|
||||
logger.info("收到系统事件", payload=payload)
|
||||
except Exception as exc:
|
||||
logger.error("处理系统事件失败", error=str(exc))
|
||||
@@ -0,0 +1,94 @@
|
||||
"""Startup and shutdown hooks for the MCP server."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import contextlib
|
||||
import structlog
|
||||
from fastapi import FastAPI
|
||||
import nats
|
||||
import redis.asyncio as redis
|
||||
|
||||
from config import Settings
|
||||
from database import init_db
|
||||
from mcp_protocol import MCPProtocolHandler
|
||||
from .events import setup_nats_handlers
|
||||
from .metrics import (
|
||||
database_connections,
|
||||
function_registry_size,
|
||||
nats_connections,
|
||||
redis_connections,
|
||||
)
|
||||
from .state import get_state
|
||||
|
||||
logger = structlog.get_logger(__name__)
|
||||
|
||||
|
||||
def register_lifecycle_events(app: FastAPI) -> None:
|
||||
"""Bind FastAPI lifecycle events to resource initialization."""
|
||||
|
||||
@app.on_event("startup")
|
||||
async def on_startup() -> None: # type: ignore[misc]
|
||||
state = get_state()
|
||||
settings: Settings = state.settings
|
||||
|
||||
try:
|
||||
await init_db()
|
||||
database_connections.set(1)
|
||||
logger.info("数据库初始化完成")
|
||||
|
||||
state.redis_client = redis.from_url(
|
||||
settings.redis_url,
|
||||
encoding="utf-8",
|
||||
decode_responses=True,
|
||||
)
|
||||
await state.redis_client.ping()
|
||||
redis_connections.set(1)
|
||||
logger.info("Redis连接成功")
|
||||
|
||||
state.nats_client = await nats.connect(settings.nats_url)
|
||||
nats_connections.set(1)
|
||||
logger.info("NATS连接成功")
|
||||
|
||||
state.mcp_handler = MCPProtocolHandler(
|
||||
redis_client=state.redis_client,
|
||||
nats_client=state.nats_client,
|
||||
litellm_url=settings.litellm_url,
|
||||
)
|
||||
function_count = len(state.mcp_handler.function_registry.list_all())
|
||||
function_registry_size.set(function_count)
|
||||
logger.info("MCP协议处理器初始化完成", function_count=function_count)
|
||||
|
||||
await setup_nats_handlers()
|
||||
logger.info("MCP Server启动完成")
|
||||
except Exception as exc: # pragma: no cover - startup is critical
|
||||
logger.error("服务启动失败", error=str(exc))
|
||||
raise
|
||||
|
||||
@app.on_event("shutdown")
|
||||
async def on_shutdown() -> None: # type: ignore[misc]
|
||||
state = get_state()
|
||||
|
||||
for websocket in list(state.active_websockets.values()):
|
||||
with contextlib.suppress(Exception):
|
||||
await websocket.close()
|
||||
state.active_websockets.clear()
|
||||
database_connections.set(0)
|
||||
|
||||
if state.mcp_handler:
|
||||
with contextlib.suppress(Exception):
|
||||
await state.mcp_handler.close()
|
||||
state.mcp_handler = None
|
||||
|
||||
if state.nats_client:
|
||||
with contextlib.suppress(Exception):
|
||||
await state.nats_client.close()
|
||||
nats_connections.set(0)
|
||||
state.nats_client = None
|
||||
|
||||
if state.redis_client:
|
||||
with contextlib.suppress(Exception):
|
||||
await state.redis_client.close()
|
||||
redis_connections.set(0)
|
||||
state.redis_client = None
|
||||
|
||||
logger.info("资源清理完成")
|
||||
@@ -0,0 +1,31 @@
|
||||
"""Logging helpers for the MCP server."""
|
||||
|
||||
import structlog
|
||||
|
||||
|
||||
_LOGGING_CONFIGURED = False
|
||||
|
||||
|
||||
def configure_logging() -> None:
|
||||
"""Configure structlog once per process."""
|
||||
global _LOGGING_CONFIGURED
|
||||
if _LOGGING_CONFIGURED:
|
||||
return
|
||||
|
||||
structlog.configure(
|
||||
processors=[
|
||||
structlog.stdlib.filter_by_level,
|
||||
structlog.stdlib.add_logger_name,
|
||||
structlog.stdlib.add_log_level,
|
||||
structlog.stdlib.PositionalArgumentsFormatter(),
|
||||
structlog.processors.TimeStamper(fmt="iso"),
|
||||
structlog.processors.StackInfoRenderer(),
|
||||
structlog.processors.format_exc_info,
|
||||
structlog.processors.UnicodeDecoder(),
|
||||
structlog.processors.JSONRenderer(),
|
||||
],
|
||||
context_class=dict,
|
||||
logger_factory=structlog.stdlib.LoggerFactory(),
|
||||
cache_logger_on_first_use=True,
|
||||
)
|
||||
_LOGGING_CONFIGURED = True
|
||||
@@ -0,0 +1,142 @@
|
||||
"""Prometheus metrics for the MCP server."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import time
|
||||
from fastapi import FastAPI, Request
|
||||
from prometheus_client import Counter, Gauge, Histogram
|
||||
|
||||
# HTTP metrics
|
||||
http_requests_total = Counter(
|
||||
"http_requests_total",
|
||||
"HTTP请求总数",
|
||||
["method", "endpoint", "status"],
|
||||
)
|
||||
|
||||
http_request_duration = Histogram(
|
||||
"http_request_duration_seconds",
|
||||
"HTTP请求耗时（秒）",
|
||||
["method", "endpoint"],
|
||||
)
|
||||
|
||||
# Agent metrics
|
||||
agents_registered_total = Counter(
|
||||
"agents_registered_total",
|
||||
"注册的Agent总数",
|
||||
["status"],
|
||||
)
|
||||
|
||||
agents_queries_total = Counter(
|
||||
"agents_queries_total",
|
||||
"Agent查询总数",
|
||||
["operation"],
|
||||
)
|
||||
|
||||
agents_active_websockets = Gauge(
|
||||
"agents_active_websockets",
|
||||
"活跃的WebSocket连接数",
|
||||
)
|
||||
|
||||
# Tool metrics
|
||||
tool_calls_total = Counter(
|
||||
"tool_calls_total",
|
||||
"工具调用总数",
|
||||
["tool_type", "status"],
|
||||
)
|
||||
|
||||
tool_call_duration = Histogram(
|
||||
"tool_call_duration_seconds",
|
||||
"工具调用耗时（秒）",
|
||||
["tool_type"],
|
||||
)
|
||||
|
||||
function_tool_calls_total = Counter(
|
||||
"function_tool_calls_total",
|
||||
"函数工具调用总数",
|
||||
["function_name", "status"],
|
||||
)
|
||||
|
||||
function_tool_call_duration = Histogram(
|
||||
"function_tool_call_duration_seconds",
|
||||
"函数工具调用耗时（秒）",
|
||||
["function_name"],
|
||||
)
|
||||
|
||||
# MCP metrics
|
||||
mcp_requests_total = Counter(
|
||||
"mcp_requests_total",
|
||||
"MCP请求总数",
|
||||
["method", "status"],
|
||||
)
|
||||
|
||||
mcp_request_duration = Histogram(
|
||||
"mcp_request_duration_seconds",
|
||||
"MCP请求耗时（秒）",
|
||||
["method"],
|
||||
)
|
||||
|
||||
# WebSocket metrics
|
||||
websocket_connections_total = Counter(
|
||||
"websocket_connections_total",
|
||||
"WebSocket连接总数",
|
||||
["status"],
|
||||
)
|
||||
|
||||
websocket_connections_active = Gauge(
|
||||
"websocket_connections_active",
|
||||
"活跃的WebSocket连接数",
|
||||
)
|
||||
|
||||
websocket_messages_total = Counter(
|
||||
"websocket_messages_total",
|
||||
"WebSocket消息总数",
|
||||
["direction"],
|
||||
)
|
||||
|
||||
# Infrastructure metrics
|
||||
redis_connections = Gauge(
|
||||
"redis_connections",
|
||||
"Redis连接状态（1=连接，0=断开）",
|
||||
)
|
||||
|
||||
nats_connections = Gauge(
|
||||
"nats_connections",
|
||||
"NATS连接状态（1=连接，0=断开）",
|
||||
)
|
||||
|
||||
database_connections = Gauge(
|
||||
"database_connections",
|
||||
"数据库连接状态（1=连接，0=断开）",
|
||||
)
|
||||
|
||||
function_registry_size = Gauge(
|
||||
"function_registry_size",
|
||||
"函数注册表大小",
|
||||
)
|
||||
|
||||
|
||||
def register_http_metrics(app: FastAPI) -> None:
|
||||
"""Attach HTTP middleware that records latency and counts."""
|
||||
|
||||
@app.middleware("http")
|
||||
async def metrics_middleware(request: Request, call_next): # type: ignore[misc]
|
||||
start_time = time.time()
|
||||
method = request.method
|
||||
endpoint = request.url.path
|
||||
|
||||
try:
|
||||
response = await call_next(request)
|
||||
status = response.status_code
|
||||
except Exception:
|
||||
status = 500
|
||||
http_requests_total.labels(method=method, endpoint=endpoint, status=status).inc()
|
||||
http_request_duration.labels(method=method, endpoint=endpoint).observe(
|
||||
time.time() - start_time
|
||||
)
|
||||
raise
|
||||
|
||||
http_requests_total.labels(method=method, endpoint=endpoint, status=status).inc()
|
||||
http_request_duration.labels(method=method, endpoint=endpoint).observe(
|
||||
time.time() - start_time
|
||||
)
|
||||
return response
|
||||
@@ -0,0 +1,17 @@
|
||||
"""Attach all FastAPI routers for the MCP server."""
|
||||
|
||||
from fastapi import FastAPI
|
||||
|
||||
from . import agents, health, metrics, monitoring, tools, websocket
|
||||
|
||||
|
||||
def register_routes(app: FastAPI) -> None:
|
||||
for router in (
|
||||
health.router,
|
||||
agents.router,
|
||||
tools.router,
|
||||
monitoring.router,
|
||||
metrics.router,
|
||||
websocket.router,
|
||||
):
|
||||
app.include_router(router)
|
||||
@@ -0,0 +1,239 @@
|
||||
"""Agent CRUD and execution endpoints."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import json
|
||||
import time
|
||||
import uuid
|
||||
import structlog
|
||||
from datetime import datetime
|
||||
from typing import List
|
||||
|
||||
from fastapi import APIRouter, Depends, HTTPException
|
||||
from sqlalchemy import select
|
||||
from sqlalchemy.ext.asyncio import AsyncSession
|
||||
|
||||
from database import get_db
|
||||
from models import Agent, User
|
||||
from schemas import AgentCard, AgentCreateRequest, ExecutionResult, MCPRequest
|
||||
from ..metrics import (
|
||||
agents_queries_total,
|
||||
agents_registered_total,
|
||||
mcp_request_duration,
|
||||
mcp_requests_total,
|
||||
)
|
||||
from ..state import get_state
|
||||
from ..utils import record_tool_metrics
|
||||
|
||||
logger = structlog.get_logger(__name__)
|
||||
router = APIRouter(prefix="/agents", tags=["agents"])
|
||||
|
||||
TEST_USER_ID = uuid.UUID("00000000-0000-0000-0000-000000000001")
|
||||
|
||||
|
||||
def _build_agent_card(agent: Agent) -> AgentCard:
|
||||
return AgentCard(
|
||||
id=agent.id,
|
||||
name=agent.name,
|
||||
description=agent.description,
|
||||
role=agent.role,
|
||||
goal=agent.goal,
|
||||
tools=agent.tools or [],
|
||||
capabilities=agent.capabilities or [],
|
||||
endpoints={
|
||||
"mcp": f"mcp://localhost:8002/agents/{agent.id}",
|
||||
"http": f"http://localhost:8002/agents/{agent.id}",
|
||||
"websocket": f"ws://localhost:8002/agents/{agent.id}/ws",
|
||||
},
|
||||
status=agent.status,
|
||||
version=agent.version,
|
||||
total_executions=agent.total_executions,
|
||||
success_rate=agent.success_rate,
|
||||
avg_execution_time=agent.avg_execution_time,
|
||||
created_at=agent.created_at,
|
||||
updated_at=agent.updated_at,
|
||||
)
|
||||
|
||||
|
||||
async def _ensure_test_user(db: AsyncSession) -> User:
|
||||
result = await db.execute(select(User).where(User.id == TEST_USER_ID))
|
||||
test_user = result.scalar_one_or_none()
|
||||
if test_user is None:
|
||||
test_user = User(
|
||||
id=TEST_USER_ID,
|
||||
username="test_user",
|
||||
email="test@taiji-ai.com",
|
||||
hashed_password="",
|
||||
full_name="测试用户",
|
||||
is_active=True,
|
||||
is_admin=False,
|
||||
)
|
||||
db.add(test_user)
|
||||
await db.flush()
|
||||
return test_user
|
||||
|
||||
|
||||
@router.post("", response_model=AgentCard)
|
||||
async def create_agent(
|
||||
request: AgentCreateRequest, db: AsyncSession = Depends(get_db)
|
||||
) -> AgentCard:
|
||||
"""Create a new agent bound to the fixed test user."""
|
||||
state = get_state()
|
||||
|
||||
try:
|
||||
await _ensure_test_user(db)
|
||||
|
||||
agent = Agent(
|
||||
name=request.name,
|
||||
description=request.description,
|
||||
role=request.role,
|
||||
goal=request.goal,
|
||||
tools=request.tools,
|
||||
config=request.config,
|
||||
capabilities=request.capabilities,
|
||||
owner_id=TEST_USER_ID,
|
||||
)
|
||||
db.add(agent)
|
||||
await db.commit()
|
||||
await db.refresh(agent)
|
||||
|
||||
card = _build_agent_card(agent)
|
||||
redis_client = state.redis_client
|
||||
if redis_client:
|
||||
await redis_client.setex(
|
||||
f"agent:{agent.id}", 3600, json.dumps(card.model_dump(mode="json"), ensure_ascii=False)
|
||||
)
|
||||
|
||||
nats_client = state.nats_client
|
||||
if nats_client:
|
||||
await nats_client.publish(
|
||||
"agent.created",
|
||||
json.dumps(
|
||||
{
|
||||
"agent_id": str(agent.id),
|
||||
"name": agent.name,
|
||||
"timestamp": datetime.utcnow().isoformat(),
|
||||
}
|
||||
).encode(),
|
||||
)
|
||||
|
||||
agents_registered_total.labels(status="success").inc()
|
||||
logger.info("Agent创建成功", agent_id=str(agent.id))
|
||||
return card
|
||||
except Exception as exc:
|
||||
await db.rollback()
|
||||
agents_registered_total.labels(status="error").inc()
|
||||
logger.error("创建Agent失败", error=str(exc))
|
||||
raise HTTPException(status_code=500, detail=str(exc)) from exc
|
||||
|
||||
|
||||
@router.get("", response_model=List[AgentCard])
|
||||
async def list_agents(
|
||||
skip: int = 0, limit: int = 100, db: AsyncSession = Depends(get_db)
|
||||
) -> List[AgentCard]:
|
||||
"""Return paginated agent cards."""
|
||||
state = get_state()
|
||||
redis_client = state.redis_client
|
||||
|
||||
try:
|
||||
agents_queries_total.labels(operation="list").inc()
|
||||
result = await db.execute(
|
||||
select(Agent).order_by(Agent.created_at.desc()).offset(skip).limit(limit)
|
||||
)
|
||||
agent_rows = result.scalars().all()
|
||||
|
||||
cards: List[AgentCard] = []
|
||||
for agent in agent_rows:
|
||||
card = _build_agent_card(agent)
|
||||
cards.append(card)
|
||||
if redis_client:
|
||||
await redis_client.setex(
|
||||
f"agent:{agent.id}", 3600, json.dumps(card.model_dump(mode="json"), ensure_ascii=False)
|
||||
)
|
||||
return cards
|
||||
except Exception as exc:
|
||||
logger.error("获取Agent列表失败", error=str(exc))
|
||||
raise HTTPException(status_code=500, detail=str(exc)) from exc
|
||||
|
||||
|
||||
@router.get("/{agent_id}", response_model=AgentCard)
|
||||
async def get_agent(agent_id: str, db: AsyncSession = Depends(get_db)) -> AgentCard:
|
||||
"""Fetch a specific agent, using Redis cache when available."""
|
||||
state = get_state()
|
||||
redis_client = state.redis_client
|
||||
|
||||
try:
|
||||
agents_queries_total.labels(operation="get").inc()
|
||||
|
||||
if redis_client:
|
||||
cached = await redis_client.get(f"agent:{agent_id}")
|
||||
if cached:
|
||||
return AgentCard.parse_raw(cached)
|
||||
|
||||
agent_uuid = uuid.UUID(agent_id)
|
||||
agent = await db.get(Agent, agent_uuid)
|
||||
if not agent:
|
||||
raise HTTPException(status_code=404, detail="Agent not found")
|
||||
|
||||
card = _build_agent_card(agent)
|
||||
if redis_client:
|
||||
await redis_client.setex(
|
||||
f"agent:{agent.id}", 3600, json.dumps(card.model_dump(mode="json"), ensure_ascii=False)
|
||||
)
|
||||
return card
|
||||
except ValueError as exc:
|
||||
raise HTTPException(status_code=400, detail="Invalid agent ID") from exc
|
||||
except HTTPException:
|
||||
raise
|
||||
except Exception as exc:
|
||||
logger.error("获取Agent失败", error=str(exc))
|
||||
raise HTTPException(status_code=500, detail=str(exc)) from exc
|
||||
|
||||
|
||||
@router.post("/{agent_id}/execute", response_model=ExecutionResult)
|
||||
async def execute_agent(
|
||||
agent_id: str, request: MCPRequest, db: AsyncSession = Depends(get_db)
|
||||
) -> ExecutionResult:
|
||||
"""Execute an MCP request for a given agent."""
|
||||
state = get_state()
|
||||
handler = state.mcp_handler
|
||||
if not handler:
|
||||
raise HTTPException(status_code=500, detail="MCP handler not initialized")
|
||||
|
||||
start_time = time.time()
|
||||
|
||||
try:
|
||||
mcp_requests_total.labels(method=request.method, status="processing").inc()
|
||||
result = await handler.execute_request(agent_id, request)
|
||||
|
||||
duration = time.time() - start_time
|
||||
mcp_requests_total.labels(method=request.method, status="success").inc()
|
||||
mcp_request_duration.labels(method=request.method).observe(duration)
|
||||
|
||||
record_tool_metrics(request, duration, success=True)
|
||||
|
||||
nats_client = state.nats_client
|
||||
if nats_client:
|
||||
await nats_client.publish(
|
||||
f"agent.execution.{agent_id}",
|
||||
json.dumps(
|
||||
{
|
||||
"agent_id": agent_id,
|
||||
"request_id": request.id,
|
||||
"method": request.method,
|
||||
"timestamp": datetime.utcnow().isoformat(),
|
||||
"success": result.success,
|
||||
}
|
||||
).encode(),
|
||||
)
|
||||
|
||||
return result
|
||||
except Exception as exc:
|
||||
duration = time.time() - start_time
|
||||
mcp_requests_total.labels(method=request.method, status="error").inc()
|
||||
mcp_request_duration.labels(method=request.method).observe(duration)
|
||||
record_tool_metrics(request, duration, success=False)
|
||||
logger.error("执行Agent任务失败", error=str(exc))
|
||||
raise HTTPException(status_code=500, detail=str(exc)) from exc
|
||||
|
||||
|
||||
@@ -0,0 +1,15 @@
|
||||
"""Health endpoints."""
|
||||
|
||||
from fastapi import APIRouter
|
||||
|
||||
from monitoring import system_monitor
|
||||
from schemas import HealthCheck
|
||||
|
||||
router = APIRouter(tags=["health"])
|
||||
|
||||
|
||||
@router.get("/health", response_model=HealthCheck)
|
||||
async def health_check() -> HealthCheck:
|
||||
"""Return the current system health snapshot."""
|
||||
data = await system_monitor.get_system_health()
|
||||
return HealthCheck(**data)
|
||||
@@ -0,0 +1,58 @@
|
||||
"""Prometheus metrics endpoint for the MCP server."""
|
||||
|
||||
import structlog
|
||||
from fastapi import APIRouter
|
||||
from fastapi.responses import JSONResponse, Response
|
||||
from prometheus_client import CONTENT_TYPE_LATEST, REGISTRY, generate_latest
|
||||
|
||||
from ..state import get_state
|
||||
from ..metrics import (
|
||||
database_connections,
|
||||
function_registry_size,
|
||||
nats_connections,
|
||||
redis_connections,
|
||||
websocket_connections_active,
|
||||
)
|
||||
|
||||
logger = structlog.get_logger(__name__)
|
||||
|
||||
router = APIRouter(tags=["metrics"])
|
||||
|
||||
|
||||
@router.get("/metrics")
|
||||
async def get_metrics() -> Response:
|
||||
"""Expose Prometheus metrics with refreshed gauges."""
|
||||
state = get_state()
|
||||
redis_client = state.redis_client
|
||||
nats_client = state.nats_client
|
||||
handler = state.mcp_handler
|
||||
|
||||
try:
|
||||
if redis_client:
|
||||
try:
|
||||
await redis_client.ping()
|
||||
redis_connections.set(1)
|
||||
except Exception:
|
||||
redis_connections.set(0)
|
||||
else:
|
||||
redis_connections.set(0)
|
||||
|
||||
if nats_client:
|
||||
nats_connections.set(1 if nats_client.is_connected else 0)
|
||||
else:
|
||||
nats_connections.set(0)
|
||||
|
||||
database_connections.set(1 if handler else 0)
|
||||
|
||||
if handler:
|
||||
try:
|
||||
function_registry_size.set(len(handler.function_registry.list_all()))
|
||||
except Exception:
|
||||
pass
|
||||
|
||||
websocket_connections_active.set(len(state.active_websockets))
|
||||
|
||||
return Response(content=generate_latest(REGISTRY), media_type=CONTENT_TYPE_LATEST)
|
||||
except Exception as exc:
|
||||
logger.error("获取metrics失败", error=str(exc))
|
||||
return JSONResponse({"error": str(exc)}, status_code=500)
|
||||
@@ -0,0 +1,97 @@
|
||||
"""Monitoring endpoints that expose aggregated stats."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import asyncio
|
||||
import structlog
|
||||
from fastapi import APIRouter, HTTPException
|
||||
|
||||
from monitoring import system_monitor
|
||||
|
||||
logger = structlog.get_logger(__name__)
|
||||
|
||||
router = APIRouter(prefix="/api/v1/monitoring", tags=["monitoring"])
|
||||
|
||||
|
||||
@router.get("/metrics")
|
||||
async def get_system_metrics() -> dict:
|
||||
"""Return live system metrics (CPU, memory, etc.)."""
|
||||
try:
|
||||
return await system_monitor.get_system_metrics()
|
||||
except Exception as exc:
|
||||
logger.error("获取系统指标失败", error=str(exc))
|
||||
raise HTTPException(status_code=500, detail=str(exc)) from exc
|
||||
|
||||
|
||||
@router.get("/stats")
|
||||
async def get_service_stats(service: str = "all") -> dict:
|
||||
"""Return aggregate statistics for a specific subsystem."""
|
||||
try:
|
||||
return await system_monitor.get_service_stats(service)
|
||||
except Exception as exc:
|
||||
logger.error("获取服务统计失败", error=str(exc))
|
||||
raise HTTPException(status_code=500, detail=str(exc)) from exc
|
||||
|
||||
|
||||
@router.get("/trends")
|
||||
async def get_performance_trends(
|
||||
metric: str = "executions", period: str = "24h", interval: str = "1h"
|
||||
) -> dict:
|
||||
"""Return trend data for executions or EU consumption."""
|
||||
try:
|
||||
return await system_monitor.get_performance_trends(metric, period, interval)
|
||||
except Exception as exc:
|
||||
logger.error("获取性能趋势失败", error=str(exc))
|
||||
raise HTTPException(status_code=500, detail=str(exc)) from exc
|
||||
|
||||
|
||||
@router.get("/alerts")
|
||||
async def get_system_alerts(severity: str | None = None) -> dict:
|
||||
"""Return alert summaries with optional severity filtering."""
|
||||
try:
|
||||
alerts = await system_monitor.get_alerts(severity)
|
||||
return {
|
||||
"timestamp": await _current_timestamp(),
|
||||
"alerts": alerts,
|
||||
"count": len(alerts),
|
||||
}
|
||||
except Exception as exc:
|
||||
logger.error("获取系统告警失败", error=str(exc))
|
||||
raise HTTPException(status_code=500, detail=str(exc)) from exc
|
||||
|
||||
|
||||
@router.get("/dashboard")
|
||||
async def get_monitoring_dashboard() -> dict:
|
||||
"""Aggregate health, metrics, stats, and alerts for dashboards."""
|
||||
try:
|
||||
health_task = system_monitor.get_system_health()
|
||||
metrics_task = system_monitor.get_system_metrics()
|
||||
stats_task = system_monitor.get_service_stats("all")
|
||||
alerts_task = system_monitor.get_alerts()
|
||||
|
||||
health, metrics, stats, alerts = await asyncio.gather(
|
||||
health_task, metrics_task, stats_task, alerts_task
|
||||
)
|
||||
|
||||
return {
|
||||
"timestamp": await _current_timestamp(),
|
||||
"health": health,
|
||||
"metrics": metrics,
|
||||
"stats": stats.get("stats", {}),
|
||||
"alerts": {
|
||||
"items": alerts,
|
||||
"count": len(alerts),
|
||||
"critical_count": len([a for a in alerts if a.get("severity") == "critical"]),
|
||||
"warning_count": len([a for a in alerts if a.get("severity") == "warning"]),
|
||||
},
|
||||
}
|
||||
except Exception as exc:
|
||||
logger.error("获取监控仪表板失败", error=str(exc))
|
||||
raise HTTPException(status_code=500, detail=str(exc)) from exc
|
||||
|
||||
|
||||
async def _current_timestamp() -> str:
|
||||
"""Helper returning the current UTC timestamp string."""
|
||||
from datetime import datetime
|
||||
|
||||
return datetime.utcnow().isoformat()
|
||||
@@ -0,0 +1,16 @@
|
||||
"""Tool catalogue endpoints (placeholder)."""
|
||||
|
||||
from fastapi import APIRouter, Depends, HTTPException
|
||||
from sqlalchemy.ext.asyncio import AsyncSession
|
||||
|
||||
from database import get_db
|
||||
from schemas import ToolDefinition
|
||||
|
||||
router = APIRouter(prefix="/tools", tags=["tools"])
|
||||
|
||||
|
||||
@router.get("", response_model=list[ToolDefinition])
|
||||
async def list_tools(db: AsyncSession = Depends(get_db)) -> list[ToolDefinition]:
|
||||
"""Return a list of tools (to be implemented)."""
|
||||
# TODO: fetch actual tool entries from the database
|
||||
return []
|
||||
@@ -0,0 +1,103 @@
|
||||
"""WebSocket endpoint for live MCP interactions."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import json
|
||||
import time
|
||||
import uuid
|
||||
import structlog
|
||||
from fastapi import APIRouter, Depends, HTTPException, WebSocket, WebSocketDisconnect
|
||||
from sqlalchemy import select
|
||||
from sqlalchemy.ext.asyncio import AsyncSession
|
||||
|
||||
from database import get_db
|
||||
from models import Agent
|
||||
from schemas import MCPRequest
|
||||
from ..metrics import (
|
||||
mcp_request_duration,
|
||||
mcp_requests_total,
|
||||
websocket_connections_active,
|
||||
websocket_connections_total,
|
||||
websocket_messages_total,
|
||||
)
|
||||
from ..state import get_state
|
||||
from ..utils import record_tool_metrics
|
||||
|
||||
logger = structlog.get_logger(__name__)
|
||||
router = APIRouter(tags=["websocket"])
|
||||
|
||||
|
||||
@router.websocket("/ws/{agent_name_or_id}")
|
||||
async def websocket_endpoint(
|
||||
websocket: WebSocket, agent_name_or_id: str, db: AsyncSession = Depends(get_db)
|
||||
) -> None:
|
||||
state = get_state()
|
||||
handler = state.mcp_handler
|
||||
if not handler:
|
||||
await websocket.close(code=1011)
|
||||
return
|
||||
|
||||
agent = await _resolve_agent(db, agent_name_or_id)
|
||||
if agent is None:
|
||||
await websocket.close(code=1008, reason=f"Agent not found: {agent_name_or_id}")
|
||||
websocket_connections_total.labels(status="rejected").inc()
|
||||
return
|
||||
|
||||
agent_id = str(agent.id)
|
||||
await websocket.accept()
|
||||
state.active_websockets[agent_id] = websocket
|
||||
websocket_connections_total.labels(status="connected").inc()
|
||||
websocket_connections_active.inc()
|
||||
logger.info("WebSocket连接建立", agent_id=agent_id)
|
||||
|
||||
try:
|
||||
while True:
|
||||
data = await websocket.receive_json()
|
||||
websocket_messages_total.labels(direction="inbound").inc()
|
||||
|
||||
if data.get("type") != "mcp_request":
|
||||
continue
|
||||
|
||||
request = MCPRequest(**data["payload"])
|
||||
mcp_requests_total.labels(method=request.method, status="processing").inc()
|
||||
start_time = time.time()
|
||||
|
||||
try:
|
||||
result = await handler.execute_request(agent_id, request)
|
||||
duration = time.time() - start_time
|
||||
mcp_requests_total.labels(method=request.method, status="success").inc()
|
||||
mcp_request_duration.labels(method=request.method).observe(duration)
|
||||
record_tool_metrics(request, duration, success=True)
|
||||
except Exception as exc:
|
||||
duration = time.time() - start_time
|
||||
mcp_requests_total.labels(method=request.method, status="error").inc()
|
||||
mcp_request_duration.labels(method=request.method).observe(duration)
|
||||
record_tool_metrics(request, duration, success=False)
|
||||
logger.error("WebSocket执行失败", error=str(exc))
|
||||
raise
|
||||
|
||||
await websocket.send_json(
|
||||
{"type": "mcp_response", "payload": result.model_dump(mode="json")}
|
||||
)
|
||||
websocket_messages_total.labels(direction="outbound").inc()
|
||||
except WebSocketDisconnect:
|
||||
websocket_connections_total.labels(status="disconnected").inc()
|
||||
logger.info("WebSocket连接断开", agent_id=agent_id)
|
||||
except Exception as exc:
|
||||
websocket_connections_total.labels(status="error").inc()
|
||||
logger.error("WebSocket错误", error=str(exc))
|
||||
raise
|
||||
finally:
|
||||
state.active_websockets.pop(agent_id, None)
|
||||
websocket_connections_active.dec()
|
||||
|
||||
|
||||
async def _resolve_agent(db: AsyncSession, agent_name_or_id: str) -> Agent | None:
|
||||
"""Resolve an agent by UUID or by name."""
|
||||
try:
|
||||
agent_uuid = uuid.UUID(agent_name_or_id)
|
||||
result = await db.execute(select(Agent).where(Agent.id == agent_uuid))
|
||||
return result.scalar_one_or_none()
|
||||
except ValueError:
|
||||
result = await db.execute(select(Agent).where(Agent.name == agent_name_or_id))
|
||||
return result.scalar_one_or_none()
|
||||
@@ -0,0 +1,34 @@
|
||||
"""Shared state container for the MCP server."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
from dataclasses import dataclass, field
|
||||
from typing import Dict, Optional, TYPE_CHECKING
|
||||
|
||||
import nats
|
||||
import redis.asyncio as redis
|
||||
|
||||
from config import Settings
|
||||
from mcp_protocol import MCPProtocolHandler
|
||||
|
||||
if TYPE_CHECKING: # pragma: no cover - typing helpers
|
||||
from fastapi import WebSocket
|
||||
|
||||
|
||||
@dataclass
|
||||
class ServiceState:
|
||||
"""Runtime resources shared across routes and background tasks."""
|
||||
|
||||
settings: Settings = field(default_factory=Settings)
|
||||
redis_client: Optional[redis.Redis] = None
|
||||
nats_client: Optional[nats.NATS] = None
|
||||
mcp_handler: Optional[MCPProtocolHandler] = None
|
||||
active_websockets: Dict[str, "WebSocket"] = field(default_factory=dict)
|
||||
|
||||
|
||||
_state = ServiceState()
|
||||
|
||||
|
||||
def get_state() -> ServiceState:
|
||||
"""Expose the singleton service state."""
|
||||
return _state
|
||||
@@ -0,0 +1,32 @@
|
||||
"""Utility helpers shared across MCP server routes."""
|
||||
|
||||
from schemas import MCPRequest
|
||||
from .metrics import (
|
||||
function_tool_call_duration,
|
||||
function_tool_calls_total,
|
||||
tool_call_duration,
|
||||
tool_calls_total,
|
||||
)
|
||||
|
||||
|
||||
def record_tool_metrics(request: MCPRequest, duration: float, success: bool) -> None:
|
||||
"""Record tool metrics for tools/call MCP requests."""
|
||||
if request.method != "tools/call":
|
||||
return
|
||||
|
||||
tool_type = "api"
|
||||
function_name = None
|
||||
params = request.params or {}
|
||||
tool_info = params.get("tool") if isinstance(params, dict) else None
|
||||
if isinstance(tool_info, dict) and tool_info.get("function_name"):
|
||||
tool_type = "function"
|
||||
function_name = tool_info.get("function_name")
|
||||
|
||||
status = "success" if success else "error"
|
||||
tool_calls_total.labels(tool_type=tool_type, status=status).inc()
|
||||
tool_call_duration.labels(tool_type=tool_type).observe(duration)
|
||||
|
||||
if function_name:
|
||||
function_tool_calls_total.labels(function_name=function_name, status=status).inc()
|
||||
if success:
|
||||
function_tool_call_duration.labels(function_name=function_name).observe(duration)
|
||||
@@ -93,6 +93,7 @@ class Settings(BaseSettings):
|
||||
env_file = ".env"
|
||||
env_file_encoding = "utf-8"
|
||||
case_sensitive = False
|
||||
extra = "ignore"
|
||||
|
||||
|
||||
class DevelopmentSettings(Settings):
|
||||
|
||||
+5
-895
@@ -1,908 +1,18 @@
|
||||
"""
|
||||
taiji-AI-PAD MCP Server
|
||||
核心MCP协议服务器，负责Agent注册、工具管理和协议通信
|
||||
"""
|
||||
"""taiji-AI-PAD MCP server entrypoint."""
|
||||
|
||||
import asyncio
|
||||
import json
|
||||
import logging
|
||||
import os
|
||||
from datetime import datetime
|
||||
from typing import Any, Dict, List, Optional
|
||||
from fastapi import Query
|
||||
from uuid import UUID
|
||||
import uuid
|
||||
from app import create_app
|
||||
|
||||
import structlog
|
||||
import time
|
||||
from fastapi import FastAPI, HTTPException, Depends, WebSocket, WebSocketDisconnect, Request
|
||||
from fastapi.middleware.cors import CORSMiddleware
|
||||
from fastapi.responses import JSONResponse, Response
|
||||
from pydantic import BaseModel
|
||||
import redis.asyncio as redis
|
||||
import nats
|
||||
from sqlalchemy.ext.asyncio import AsyncSession, create_async_engine
|
||||
from sqlalchemy.orm import sessionmaker
|
||||
from sqlalchemy import select
|
||||
from prometheus_client import (
|
||||
Counter, Histogram, Gauge, generate_latest,
|
||||
CONTENT_TYPE_LATEST, REGISTRY
|
||||
)
|
||||
|
||||
from models import Agent, Tool, Session as DBSession, User
|
||||
from schemas import (
|
||||
AgentCard,
|
||||
AgentCreateRequest,
|
||||
ToolDefinition,
|
||||
MCPRequest,
|
||||
MCPResponse,
|
||||
ExecutionResult
|
||||
)
|
||||
from mcp_protocol import MCPProtocolHandler
|
||||
from database import get_db, init_db
|
||||
from config import Settings
|
||||
from monitoring import system_monitor
|
||||
|
||||
# 配置日志
|
||||
structlog.configure(
|
||||
processors=[
|
||||
structlog.stdlib.filter_by_level,
|
||||
structlog.stdlib.add_logger_name,
|
||||
structlog.stdlib.add_log_level,
|
||||
structlog.stdlib.PositionalArgumentsFormatter(),
|
||||
structlog.processors.TimeStamper(fmt="iso"),
|
||||
structlog.processors.StackInfoRenderer(),
|
||||
structlog.processors.format_exc_info,
|
||||
structlog.processors.UnicodeDecoder(),
|
||||
structlog.processors.JSONRenderer()
|
||||
],
|
||||
context_class=dict,
|
||||
logger_factory=structlog.stdlib.LoggerFactory(),
|
||||
cache_logger_on_first_use=True,
|
||||
)
|
||||
|
||||
logger = structlog.get_logger()
|
||||
|
||||
# 应用设置
|
||||
settings = Settings()
|
||||
app = FastAPI(
|
||||
title="taiji-AI-PAD MCP Server",
|
||||
description="Model Context Protocol Server for Agent Management",
|
||||
version="1.0.0",
|
||||
docs_url="/docs",
|
||||
redoc_url="/redoc"
|
||||
)
|
||||
|
||||
# CORS配置
|
||||
app.add_middleware(
|
||||
CORSMiddleware,
|
||||
allow_origins=["*"],
|
||||
allow_credentials=True,
|
||||
allow_methods=["*"],
|
||||
allow_headers=["*"],
|
||||
)
|
||||
|
||||
# Prometheus Metrics定义
|
||||
# HTTP请求指标
|
||||
http_requests_total = Counter(
|
||||
"http_requests_total",
|
||||
"HTTP请求总数",
|
||||
["method", "endpoint", "status"]
|
||||
)
|
||||
http_request_duration = Histogram(
|
||||
"http_request_duration_seconds",
|
||||
"HTTP请求耗时（秒）",
|
||||
["method", "endpoint"]
|
||||
)
|
||||
|
||||
# Agent管理指标
|
||||
agents_registered_total = Counter(
|
||||
"agents_registered_total",
|
||||
"注册的Agent总数",
|
||||
["status"]
|
||||
)
|
||||
agents_queries_total = Counter(
|
||||
"agents_queries_total",
|
||||
"Agent查询总数",
|
||||
["operation"]
|
||||
)
|
||||
agents_active_websockets = Gauge(
|
||||
"agents_active_websockets",
|
||||
"活跃的WebSocket连接数"
|
||||
)
|
||||
|
||||
# 工具调用指标
|
||||
tool_calls_total = Counter(
|
||||
"tool_calls_total",
|
||||
"工具调用总数",
|
||||
["tool_type", "status"]
|
||||
)
|
||||
tool_call_duration = Histogram(
|
||||
"tool_call_duration_seconds",
|
||||
"工具调用耗时（秒）",
|
||||
["tool_type"]
|
||||
)
|
||||
function_tool_calls_total = Counter(
|
||||
"function_tool_calls_total",
|
||||
"函数工具调用总数",
|
||||
["function_name", "status"]
|
||||
)
|
||||
function_tool_call_duration = Histogram(
|
||||
"function_tool_call_duration_seconds",
|
||||
"函数工具调用耗时（秒）",
|
||||
["function_name"]
|
||||
)
|
||||
|
||||
# MCP协议指标
|
||||
mcp_requests_total = Counter(
|
||||
"mcp_requests_total",
|
||||
"MCP请求总数",
|
||||
["method", "status"]
|
||||
)
|
||||
mcp_request_duration = Histogram(
|
||||
"mcp_request_duration_seconds",
|
||||
"MCP请求耗时（秒）",
|
||||
["method"]
|
||||
)
|
||||
|
||||
# WebSocket指标
|
||||
websocket_connections_total = Counter(
|
||||
"websocket_connections_total",
|
||||
"WebSocket连接总数",
|
||||
["status"]
|
||||
)
|
||||
websocket_connections_active = Gauge(
|
||||
"websocket_connections_active",
|
||||
"活跃的WebSocket连接数"
|
||||
)
|
||||
websocket_messages_total = Counter(
|
||||
"websocket_messages_total",
|
||||
"WebSocket消息总数",
|
||||
["direction"]
|
||||
)
|
||||
|
||||
# 系统健康指标
|
||||
redis_connections = Gauge(
|
||||
"redis_connections",
|
||||
"Redis连接状态（1=连接，0=断开）"
|
||||
)
|
||||
nats_connections = Gauge(
|
||||
"nats_connections",
|
||||
"NATS连接状态（1=连接，0=断开）"
|
||||
)
|
||||
database_connections = Gauge(
|
||||
"database_connections",
|
||||
"数据库连接状态（1=连接，0=断开）"
|
||||
)
|
||||
function_registry_size = Gauge(
|
||||
"function_registry_size",
|
||||
"函数注册表大小"
|
||||
)
|
||||
|
||||
# Prometheus Metrics中间件
|
||||
@app.middleware("http")
|
||||
async def metrics_middleware(request: Request, call_next):
|
||||
"""收集HTTP请求指标"""
|
||||
start_time = time.time()
|
||||
method = request.method
|
||||
endpoint = request.url.path
|
||||
|
||||
try:
|
||||
response = await call_next(request)
|
||||
status = response.status_code
|
||||
|
||||
# 记录指标
|
||||
http_requests_total.labels(method=method, endpoint=endpoint, status=status).inc()
|
||||
http_request_duration.labels(method=method, endpoint=endpoint).observe(time.time() - start_time)
|
||||
|
||||
return response
|
||||
except Exception as e:
|
||||
status = 500
|
||||
http_requests_total.labels(method=method, endpoint=endpoint, status=status).inc()
|
||||
http_request_duration.labels(method=method, endpoint=endpoint).observe(time.time() - start_time)
|
||||
raise
|
||||
|
||||
# 全局变量
|
||||
redis_client: Optional[redis.Redis] = None
|
||||
nats_client: Optional[nats.NATS] = None
|
||||
mcp_handler: Optional[MCPProtocolHandler] = None
|
||||
active_websockets: Dict[str, WebSocket] = {}
|
||||
|
||||
|
||||
def _build_agent_card(agent: Agent) -> AgentCard:
|
||||
"""将ORM对象转换为AgentCard用于序列化"""
|
||||
return AgentCard(
|
||||
id=agent.id,
|
||||
name=agent.name,
|
||||
description=agent.description,
|
||||
role=agent.role,
|
||||
goal=agent.goal,
|
||||
tools=agent.tools or [],
|
||||
capabilities=agent.capabilities or [],
|
||||
endpoints={
|
||||
"mcp": f"mcp://localhost:8002/agents/{str(agent.id)}",
|
||||
"http": f"http://localhost:8002/agents/{str(agent.id)}",
|
||||
"websocket": f"ws://localhost:8002/agents/{str(agent.id)}/ws"
|
||||
},
|
||||
status=agent.status,
|
||||
version=agent.version,
|
||||
total_executions=agent.total_executions,
|
||||
success_rate=agent.success_rate,
|
||||
avg_execution_time=agent.avg_execution_time,
|
||||
created_at=agent.created_at,
|
||||
updated_at=agent.updated_at
|
||||
)
|
||||
|
||||
class HealthResponse(BaseModel):
|
||||
status: str
|
||||
timestamp: str
|
||||
services: Dict[str, str]
|
||||
|
||||
@app.on_event("startup")
|
||||
async def startup_event():
|
||||
"""应用启动时初始化服务"""
|
||||
global redis_client, nats_client, mcp_handler
|
||||
|
||||
try:
|
||||
# 初始化数据库
|
||||
await init_db()
|
||||
logger.info("数据库初始化完成")
|
||||
|
||||
# 连接Redis
|
||||
redis_client = redis.from_url(
|
||||
settings.redis_url,
|
||||
encoding="utf-8",
|
||||
decode_responses=True
|
||||
)
|
||||
await redis_client.ping()
|
||||
redis_connections.set(1)
|
||||
logger.info("Redis连接成功")
|
||||
|
||||
# 连接NATS
|
||||
nats_client = await nats.connect(settings.nats_url)
|
||||
nats_connections.set(1)
|
||||
logger.info("NATS连接成功")
|
||||
|
||||
# 初始化数据库连接状态
|
||||
database_connections.set(1)
|
||||
|
||||
# 初始化MCP协议处理器
|
||||
mcp_handler = MCPProtocolHandler(redis_client, nats_client)
|
||||
logger.info("MCP协议处理器初始化完成")
|
||||
|
||||
# 更新函数注册表大小
|
||||
function_count = len(mcp_handler.function_registry.list_all())
|
||||
function_registry_size.set(function_count)
|
||||
|
||||
# 注册NATS事件处理器
|
||||
await setup_nats_handlers()
|
||||
|
||||
logger.info("MCP Server启动完成")
|
||||
|
||||
except Exception as e:
|
||||
logger.error(f"服务启动失败: {e}")
|
||||
raise
|
||||
|
||||
@app.on_event("shutdown")
|
||||
async def shutdown_event():
|
||||
"""应用关闭时清理资源"""
|
||||
global redis_client, nats_client
|
||||
|
||||
try:
|
||||
# 关闭所有WebSocket连接
|
||||
for ws in active_websockets.values():
|
||||
await ws.close()
|
||||
|
||||
# 关闭NATS连接
|
||||
if nats_client:
|
||||
await nats_client.close()
|
||||
|
||||
# 关闭Redis连接
|
||||
if redis_client:
|
||||
await redis_client.close()
|
||||
|
||||
logger.info("资源清理完成")
|
||||
|
||||
except Exception as e:
|
||||
logger.error(f"资源清理失败: {e}")
|
||||
|
||||
async def setup_nats_handlers():
|
||||
"""设置NATS事件处理器"""
|
||||
if not nats_client:
|
||||
return
|
||||
|
||||
# Agent执行事件
|
||||
await nats_client.subscribe("agent.execution.*", cb=handle_agent_execution)
|
||||
|
||||
# 计费事件
|
||||
await nats_client.subscribe("billing.*", cb=handle_billing_event)
|
||||
|
||||
# 系统事件
|
||||
await nats_client.subscribe("system.*", cb=handle_system_event)
|
||||
|
||||
async def handle_agent_execution(msg):
|
||||
"""处理Agent执行事件"""
|
||||
try:
|
||||
data = json.loads(msg.data.decode())
|
||||
logger.info(f"收到Agent执行事件: {data}")
|
||||
|
||||
# 广播给相关的WebSocket连接
|
||||
for ws in active_websockets.values():
|
||||
await ws.send_json({
|
||||
"type": "agent_execution",
|
||||
"data": data
|
||||
})
|
||||
|
||||
except Exception as e:
|
||||
logger.error(f"处理Agent执行事件失败: {e}")
|
||||
|
||||
async def handle_billing_event(msg):
|
||||
"""处理计费事件"""
|
||||
try:
|
||||
data = json.loads(msg.data.decode())
|
||||
logger.info(f"收到计费事件: {data}")
|
||||
|
||||
except Exception as e:
|
||||
logger.error(f"处理计费事件失败: {e}")
|
||||
|
||||
async def handle_system_event(msg):
|
||||
"""处理系统事件"""
|
||||
try:
|
||||
data = json.loads(msg.data.decode())
|
||||
logger.info(f"收到系统事件: {data}")
|
||||
|
||||
except Exception as e:
|
||||
logger.error(f"处理系统事件失败: {e}")
|
||||
|
||||
@app.get("/health", response_model=HealthResponse)
|
||||
async def health_check():
|
||||
"""健康检查端点"""
|
||||
health_data = await system_monitor.get_system_health()
|
||||
return HealthResponse(
|
||||
status=health_data["status"],
|
||||
timestamp=health_data["timestamp"],
|
||||
services=health_data["services"]
|
||||
)
|
||||
|
||||
# 固定的测试用户UUID
|
||||
TEST_USER_ID = uuid.UUID("00000000-0000-0000-0000-000000000001")
|
||||
|
||||
@app.post("/agents", response_model=AgentCard)
|
||||
async def create_agent(
|
||||
request: AgentCreateRequest,
|
||||
db: AsyncSession = Depends(get_db)
|
||||
):
|
||||
"""创建新的Agent"""
|
||||
try:
|
||||
# 使用固定的测试用户ID
|
||||
owner_id = TEST_USER_ID
|
||||
|
||||
# 确保测试用户存在（如果不存在则创建）
|
||||
result = await db.execute(select(User).where(User.id == TEST_USER_ID))
|
||||
test_user = result.scalar_one_or_none()
|
||||
if test_user is None:
|
||||
test_user = User(
|
||||
id=TEST_USER_ID,
|
||||
username="test_user",
|
||||
email="test@taiji-ai.com",
|
||||
hashed_password="", # 测试用户不需要密码
|
||||
full_name="测试用户",
|
||||
is_active=True,
|
||||
is_admin=False
|
||||
)
|
||||
db.add(test_user)
|
||||
await db.flush()
|
||||
|
||||
# 创建Agent记录
|
||||
agent = Agent(
|
||||
name=request.name,
|
||||
description=request.description,
|
||||
role=request.role,
|
||||
goal=request.goal,
|
||||
tools=request.tools,
|
||||
config=request.config,
|
||||
capabilities=request.capabilities,
|
||||
owner_id=owner_id # 使用固定的测试用户ID
|
||||
)
|
||||
|
||||
db.add(agent)
|
||||
await db.commit()
|
||||
await db.refresh(agent)
|
||||
|
||||
# 确保所有UUID对象都是标准类型（转换asyncpg的UUID）
|
||||
def convert_uuid_to_standard(uuid_obj):
|
||||
"""将asyncpg的UUID转换为标准UUID"""
|
||||
if uuid_obj is None:
|
||||
return None
|
||||
try:
|
||||
from asyncpg.pgproto.pgproto import UUID as AsyncUUID
|
||||
if isinstance(uuid_obj, AsyncUUID):
|
||||
return uuid.UUID(str(uuid_obj))
|
||||
except (ImportError, AttributeError):
|
||||
pass
|
||||
if isinstance(uuid_obj, uuid.UUID):
|
||||
return uuid_obj
|
||||
try:
|
||||
return uuid.UUID(str(uuid_obj))
|
||||
except (ValueError, TypeError):
|
||||
return uuid_obj
|
||||
|
||||
# 转换agent.id为字符串，然后让Pydantic处理
|
||||
agent_id_str = str(agent.id)
|
||||
agent_id = uuid.UUID(agent_id_str)
|
||||
|
||||
# 生成Agent Card - 使用字典方式创建，避免Pydantic验证时的问题
|
||||
agent_card_dict = {
|
||||
"id": agent_id,
|
||||
"name": agent.name,
|
||||
"description": agent.description,
|
||||
"role": agent.role,
|
||||
"goal": agent.goal,
|
||||
"tools": agent.tools or [],
|
||||
"capabilities": agent.capabilities or [],
|
||||
"endpoints": {
|
||||
"mcp": f"mcp://localhost:8002/agents/{agent_id_str}",
|
||||
"http": f"http://localhost:8002/agents/{agent_id_str}",
|
||||
"websocket": f"ws://localhost:8002/agents/{agent_id_str}/ws"
|
||||
},
|
||||
"created_at": agent.created_at,
|
||||
"updated_at": agent.updated_at
|
||||
}
|
||||
agent_card = AgentCard(**agent_card_dict)
|
||||
|
||||
# 缓存到Redis
|
||||
if redis_client:
|
||||
cached_agent = agent_card.model_dump(mode="json")
|
||||
await redis_client.setex(
|
||||
f"agent:{str(agent.id)}",
|
||||
3600, # 1小时过期
|
||||
json.dumps(cached_agent, ensure_ascii=False)
|
||||
)
|
||||
|
||||
# 发布Agent创建事件
|
||||
if nats_client:
|
||||
await nats_client.publish(
|
||||
"agent.created",
|
||||
json.dumps({
|
||||
"agent_id": str(agent.id),
|
||||
"name": agent.name,
|
||||
"timestamp": datetime.utcnow().isoformat()
|
||||
}).encode()
|
||||
)
|
||||
|
||||
# 记录指标
|
||||
agents_registered_total.labels(status="success").inc()
|
||||
|
||||
logger.info(f"Agent创建成功: {agent.id}")
|
||||
return agent_card
|
||||
|
||||
except Exception as e:
|
||||
# 记录失败指标
|
||||
agents_registered_total.labels(status="error").inc()
|
||||
import traceback
|
||||
logger.error(f"创建Agent失败: {e}")
|
||||
logger.error(f"错误堆栈:\n{traceback.format_exc()}")
|
||||
raise HTTPException(status_code=500, detail=str(e))
|
||||
|
||||
@app.get("/agents", response_model=List[AgentCard])
|
||||
async def list_agents(
|
||||
skip: int = 0,
|
||||
limit: int = 100,
|
||||
db: AsyncSession = Depends(get_db)
|
||||
):
|
||||
"""获取Agent列表"""
|
||||
try:
|
||||
agents_queries_total.labels(operation="list").inc()
|
||||
|
||||
result = await db.execute(
|
||||
select(Agent)
|
||||
.order_by(Agent.created_at.desc())
|
||||
.offset(skip)
|
||||
.limit(limit)
|
||||
)
|
||||
agent_rows = result.scalars().all()
|
||||
|
||||
agent_cards: List[AgentCard] = []
|
||||
for agent in agent_rows:
|
||||
card = _build_agent_card(agent)
|
||||
agent_cards.append(card)
|
||||
if redis_client:
|
||||
cached_agent = card.model_dump(mode="json")
|
||||
await redis_client.setex(
|
||||
f"agent:{agent.id}",
|
||||
3600,
|
||||
json.dumps(cached_agent, ensure_ascii=False)
|
||||
)
|
||||
|
||||
return agent_cards
|
||||
|
||||
except Exception as e:
|
||||
logger.error(f"获取Agent列表失败: {e}")
|
||||
raise HTTPException(status_code=500, detail=str(e))
|
||||
|
||||
@app.get("/agents/{agent_id}", response_model=AgentCard)
|
||||
async def get_agent(agent_id: str, db: AsyncSession = Depends(get_db)):
|
||||
"""获取特定Agent信息"""
|
||||
try:
|
||||
agents_queries_total.labels(operation="get").inc()
|
||||
# 先从Redis缓存查找
|
||||
if redis_client:
|
||||
cached = await redis_client.get(f"agent:{agent_id}")
|
||||
if cached:
|
||||
return AgentCard.parse_raw(cached)
|
||||
|
||||
try:
|
||||
agent_uuid = UUID(agent_id)
|
||||
except ValueError:
|
||||
raise HTTPException(status_code=400, detail="Invalid agent ID")
|
||||
|
||||
agent = await db.get(Agent, agent_uuid)
|
||||
if not agent:
|
||||
raise HTTPException(status_code=404, detail="Agent not found")
|
||||
|
||||
card = _build_agent_card(agent)
|
||||
if redis_client:
|
||||
cached_agent = card.model_dump(mode="json")
|
||||
await redis_client.setex(
|
||||
f"agent:{agent.id}",
|
||||
3600,
|
||||
json.dumps(cached_agent, ensure_ascii=False)
|
||||
)
|
||||
|
||||
return card
|
||||
|
||||
except HTTPException:
|
||||
raise
|
||||
except Exception as e:
|
||||
logger.error(f"获取Agent失败: {e}")
|
||||
raise HTTPException(status_code=500, detail=str(e))
|
||||
|
||||
@app.post("/agents/{agent_id}/execute", response_model=ExecutionResult)
|
||||
async def execute_agent(
|
||||
agent_id: str,
|
||||
request: MCPRequest,
|
||||
db: AsyncSession = Depends(get_db)
|
||||
):
|
||||
"""执行Agent任务"""
|
||||
start_time = time.time()
|
||||
try:
|
||||
if not mcp_handler:
|
||||
raise HTTPException(status_code=500, detail="MCP handler not initialized")
|
||||
|
||||
# 记录MCP请求指标
|
||||
mcp_requests_total.labels(method=request.method, status="processing").inc()
|
||||
|
||||
# 执行MCP请求
|
||||
result = await mcp_handler.execute_request(agent_id, request)
|
||||
|
||||
# 记录成功指标
|
||||
duration = time.time() - start_time
|
||||
mcp_requests_total.labels(method=request.method, status="success").inc()
|
||||
mcp_request_duration.labels(method=request.method).observe(duration)
|
||||
|
||||
# 判断工具类型并记录指标
|
||||
if request.method == "tools/call":
|
||||
tool_type = "api" # 默认API工具
|
||||
if request.params and isinstance(request.params, dict):
|
||||
tool_info = request.params.get("tool", {})
|
||||
if isinstance(tool_info, dict) and tool_info.get("function_name"):
|
||||
tool_type = "function"
|
||||
function_name = tool_info.get("function_name")
|
||||
function_tool_calls_total.labels(function_name=function_name, status="success").inc()
|
||||
function_tool_call_duration.labels(function_name=function_name).observe(duration)
|
||||
|
||||
tool_calls_total.labels(tool_type=tool_type, status="success").inc()
|
||||
tool_call_duration.labels(tool_type=tool_type).observe(duration)
|
||||
|
||||
# 发布执行事件
|
||||
if nats_client:
|
||||
await nats_client.publish(
|
||||
f"agent.execution.{agent_id}",
|
||||
json.dumps({
|
||||
"agent_id": agent_id,
|
||||
"request_id": request.id,
|
||||
"method": request.method,
|
||||
"timestamp": datetime.utcnow().isoformat(),
|
||||
"success": result.success
|
||||
}).encode()
|
||||
)
|
||||
|
||||
return result
|
||||
|
||||
except Exception as e:
|
||||
# 记录失败指标
|
||||
duration = time.time() - start_time
|
||||
mcp_requests_total.labels(method=request.method, status="error").inc()
|
||||
mcp_request_duration.labels(method=request.method).observe(duration)
|
||||
|
||||
if request.method == "tools/call":
|
||||
tool_type = "api"
|
||||
if request.params and isinstance(request.params, dict):
|
||||
tool_info = request.params.get("tool", {})
|
||||
if isinstance(tool_info, dict) and tool_info.get("function_name"):
|
||||
tool_type = "function"
|
||||
function_name = tool_info.get("function_name")
|
||||
function_tool_calls_total.labels(function_name=function_name, status="error").inc()
|
||||
|
||||
tool_calls_total.labels(tool_type=tool_type, status="error").inc()
|
||||
|
||||
logger.error(f"执行Agent任务失败: {e}")
|
||||
raise HTTPException(status_code=500, detail=str(e))
|
||||
|
||||
@app.get("/tools", response_model=List[ToolDefinition])
|
||||
async def list_tools(db: AsyncSession = Depends(get_db)):
|
||||
"""获取可用工具列表"""
|
||||
try:
|
||||
# 从数据库获取工具列表
|
||||
# 这里应该有实际的工具查询逻辑
|
||||
tools = []
|
||||
|
||||
return tools
|
||||
|
||||
except Exception as e:
|
||||
logger.error(f"获取工具列表失败: {e}")
|
||||
raise HTTPException(status_code=500, detail=str(e))
|
||||
|
||||
@app.websocket("/ws/{agent_name_or_id}")
|
||||
async def websocket_endpoint_by_name(websocket: WebSocket, agent_name_or_id: str, db: AsyncSession = Depends(get_db)):
|
||||
"""Agent WebSocket连接端点（通过名称或ID）"""
|
||||
agent_id = None
|
||||
try:
|
||||
# 尝试通过名称或ID查找Agent
|
||||
agent = None
|
||||
try:
|
||||
# 尝试作为UUID解析
|
||||
agent_uuid = uuid.UUID(agent_name_or_id)
|
||||
result = await db.execute(select(Agent).where(Agent.id == agent_uuid))
|
||||
agent = result.scalar_one_or_none()
|
||||
except ValueError:
|
||||
# 如果不是UUID，则作为名称查找
|
||||
result = await db.execute(select(Agent).where(Agent.name == agent_name_or_id))
|
||||
agent = result.scalar_one_or_none()
|
||||
|
||||
if agent is None:
|
||||
await websocket.close(code=1008, reason=f"Agent not found: {agent_name_or_id}")
|
||||
websocket_connections_total.labels(status="rejected").inc()
|
||||
logger.warning(f"WebSocket连接被拒绝: Agent不存在 - {agent_name_or_id}")
|
||||
return
|
||||
|
||||
# 接受连接
|
||||
await websocket.accept()
|
||||
agent_id = str(agent.id)
|
||||
active_websockets[agent_id] = websocket
|
||||
websocket_connections_total.labels(status="connected").inc()
|
||||
websocket_connections_active.inc()
|
||||
|
||||
logger.info(f"WebSocket连接建立: {agent.name} ({agent_id})")
|
||||
|
||||
while True:
|
||||
# 等待客户端消息
|
||||
data = await websocket.receive_json()
|
||||
websocket_messages_total.labels(direction="inbound").inc()
|
||||
|
||||
# 处理MCP消息
|
||||
if mcp_handler and data.get("type") == "mcp_request":
|
||||
request = MCPRequest(**data["payload"])
|
||||
|
||||
# 记录MCP请求指标
|
||||
mcp_requests_total.labels(method=request.method, status="processing").inc()
|
||||
start_time = time.time()
|
||||
|
||||
try:
|
||||
result = await mcp_handler.execute_request(agent_id, request)
|
||||
duration = time.time() - start_time
|
||||
|
||||
# 记录成功指标
|
||||
mcp_requests_total.labels(method=request.method, status="success").inc()
|
||||
mcp_request_duration.labels(method=request.method).observe(duration)
|
||||
|
||||
# 判断工具类型并记录指标
|
||||
if request.method == "tools/call":
|
||||
tool_type = "api"
|
||||
if request.params and isinstance(request.params, dict):
|
||||
tool_info = request.params.get("tool", {})
|
||||
if isinstance(tool_info, dict) and tool_info.get("function_name"):
|
||||
tool_type = "function"
|
||||
function_name = tool_info.get("function_name")
|
||||
function_tool_calls_total.labels(function_name=function_name, status="success").inc()
|
||||
function_tool_call_duration.labels(function_name=function_name).observe(duration)
|
||||
|
||||
tool_calls_total.labels(tool_type=tool_type, status="success").inc()
|
||||
tool_call_duration.labels(tool_type=tool_type).observe(duration)
|
||||
|
||||
except Exception as e:
|
||||
duration = time.time() - start_time
|
||||
mcp_requests_total.labels(method=request.method, status="error").inc()
|
||||
mcp_request_duration.labels(method=request.method).observe(duration)
|
||||
raise
|
||||
|
||||
await websocket.send_json({
|
||||
"type": "mcp_response",
|
||||
"payload": result.model_dump(mode="json")
|
||||
})
|
||||
websocket_messages_total.labels(direction="outbound").inc()
|
||||
|
||||
except WebSocketDisconnect:
|
||||
websocket_connections_total.labels(status="disconnected").inc()
|
||||
logger.info(f"WebSocket连接断开: {agent_id}")
|
||||
except Exception as e:
|
||||
websocket_connections_total.labels(status="error").inc()
|
||||
logger.error(f"WebSocket错误: {e}")
|
||||
finally:
|
||||
if agent_id in active_websockets:
|
||||
del active_websockets[agent_id]
|
||||
websocket_connections_active.dec()
|
||||
|
||||
@app.get("/metrics")
|
||||
async def get_metrics():
|
||||
"""Prometheus metrics端点"""
|
||||
try:
|
||||
# 更新动态指标
|
||||
if redis_client:
|
||||
try:
|
||||
await redis_client.ping()
|
||||
redis_connections.set(1)
|
||||
except:
|
||||
redis_connections.set(0)
|
||||
else:
|
||||
redis_connections.set(0)
|
||||
|
||||
if nats_client:
|
||||
try:
|
||||
if nats_client.is_connected:
|
||||
nats_connections.set(1)
|
||||
else:
|
||||
nats_connections.set(0)
|
||||
except:
|
||||
nats_connections.set(0)
|
||||
else:
|
||||
nats_connections.set(0)
|
||||
|
||||
# 更新数据库连接状态
|
||||
try:
|
||||
# 这里可以添加实际的数据库连接检查
|
||||
database_connections.set(1)
|
||||
except:
|
||||
database_connections.set(0)
|
||||
|
||||
# 更新函数注册表大小
|
||||
if mcp_handler:
|
||||
try:
|
||||
function_count = len(mcp_handler.function_registry.list_all())
|
||||
function_registry_size.set(function_count)
|
||||
except:
|
||||
pass
|
||||
|
||||
# 更新活跃Agent数量
|
||||
try:
|
||||
if redis_client:
|
||||
# 从Redis获取活跃Agent数量（如果有缓存）
|
||||
# 这里可以根据实际情况实现
|
||||
pass
|
||||
except:
|
||||
pass
|
||||
|
||||
# 更新WebSocket连接数
|
||||
websocket_connections_active.set(len(active_websockets))
|
||||
|
||||
# 生成Prometheus格式的指标
|
||||
return Response(
|
||||
content=generate_latest(REGISTRY),
|
||||
media_type=CONTENT_TYPE_LATEST
|
||||
)
|
||||
except Exception as e:
|
||||
logger.error(f"获取metrics失败: {e}")
|
||||
return JSONResponse(
|
||||
{"error": str(e)},
|
||||
status_code=500
|
||||
)
|
||||
|
||||
# ========== 监控API端点 ==========
|
||||
|
||||
@app.get("/api/v1/monitoring/metrics")
|
||||
async def get_system_metrics():
|
||||
"""获取系统性能指标"""
|
||||
try:
|
||||
metrics = await system_monitor.get_system_metrics()
|
||||
return metrics
|
||||
except Exception as e:
|
||||
logger.error(f"获取系统指标失败: {e}")
|
||||
raise HTTPException(status_code=500, detail=str(e))
|
||||
|
||||
|
||||
@app.get("/api/v1/monitoring/stats")
|
||||
async def get_service_stats(service: str = "all"):
|
||||
"""获取服务统计信息
|
||||
|
||||
Args:
|
||||
service: 服务类型 (all, agents, executions, tools, users)
|
||||
"""
|
||||
try:
|
||||
stats = await system_monitor.get_service_stats(service)
|
||||
return stats
|
||||
except Exception as e:
|
||||
logger.error(f"获取服务统计失败: {e}")
|
||||
raise HTTPException(status_code=500, detail=str(e))
|
||||
|
||||
|
||||
@app.get("/api/v1/monitoring/trends")
|
||||
async def get_performance_trends(
|
||||
metric: str = "executions",
|
||||
period: str = "24h",
|
||||
interval: str = "1h"
|
||||
):
|
||||
"""获取性能趋势数据
|
||||
|
||||
Args:
|
||||
metric: 指标类型 (executions, eu_consumption)
|
||||
period: 时间范围 (24h, 7d, 30d)
|
||||
interval: 时间间隔 (1h, 6h, 1d)
|
||||
"""
|
||||
try:
|
||||
trends = await system_monitor.get_performance_trends(metric, period, interval)
|
||||
return trends
|
||||
except Exception as e:
|
||||
logger.error(f"获取性能趋势失败: {e}")
|
||||
raise HTTPException(status_code=500, detail=str(e))
|
||||
|
||||
|
||||
@app.get("/api/v1/monitoring/alerts")
|
||||
async def get_system_alerts(severity: Optional[str] = None):
|
||||
"""获取系统告警
|
||||
|
||||
Args:
|
||||
severity: 严重程度过滤 (warning, critical, info)
|
||||
"""
|
||||
try:
|
||||
alerts = await system_monitor.get_alerts(severity)
|
||||
return {
|
||||
"timestamp": datetime.utcnow().isoformat(),
|
||||
"alerts": alerts,
|
||||
"count": len(alerts)
|
||||
}
|
||||
except Exception as e:
|
||||
logger.error(f"获取系统告警失败: {e}")
|
||||
raise HTTPException(status_code=500, detail=str(e))
|
||||
|
||||
|
||||
@app.get("/api/v1/monitoring/dashboard")
|
||||
async def get_monitoring_dashboard():
|
||||
"""获取监控仪表板数据（聚合所有监控信息）"""
|
||||
try:
|
||||
# 并行获取所有监控数据
|
||||
health_task = system_monitor.get_system_health()
|
||||
metrics_task = system_monitor.get_system_metrics()
|
||||
stats_task = system_monitor.get_service_stats("all")
|
||||
alerts_task = system_monitor.get_alerts()
|
||||
|
||||
health, metrics, stats, alerts = await asyncio.gather(
|
||||
health_task, metrics_task, stats_task, alerts_task
|
||||
)
|
||||
|
||||
return {
|
||||
"timestamp": datetime.utcnow().isoformat(),
|
||||
"health": health,
|
||||
"metrics": metrics,
|
||||
"stats": stats["stats"],
|
||||
"alerts": {
|
||||
"items": alerts,
|
||||
"count": len(alerts),
|
||||
"critical_count": len([a for a in alerts if a.get("severity") == "critical"]),
|
||||
"warning_count": len([a for a in alerts if a.get("severity") == "warning"]),
|
||||
}
|
||||
}
|
||||
except Exception as e:
|
||||
logger.error(f"获取监控仪表板失败: {e}")
|
||||
raise HTTPException(status_code=500, detail=str(e))
|
||||
app = create_app()
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
import uvicorn
|
||||
|
||||
uvicorn.run(
|
||||
"main:app",
|
||||
host="0.0.0.0",
|
||||
port=8000,
|
||||
reload=True,
|
||||
log_level="info"
|
||||
log_level="info",
|
||||
)
|
||||
|
||||
|
||||
@@ -9,4 +9,6 @@ addopts =
|
||||
--tb=short
|
||||
--strict-markers
|
||||
--disable-warnings
|
||||
markers =
|
||||
integration: tests that hit external or dockerized services
|
||||
|
||||
|
||||
@@ -1,148 +0,0 @@
|
||||
# MCP Server 单元测试
|
||||
|
||||
## 📋 测试概述
|
||||
|
||||
本目录包含 MCP Server 的单元测试，覆盖以下模块：
|
||||
|
||||
1. **function_registry** - 函数注册表测试
|
||||
2. **sandbox_executor** - 沙箱执行器测试
|
||||
3. **mcp_function_tool** - MCP 函数工具调用测试
|
||||
|
||||
## 🚀 运行测试
|
||||
|
||||
### 方法 1: 使用测试脚本
|
||||
|
||||
```bash
|
||||
./run_tests.sh
|
||||
```
|
||||
|
||||
### 方法 2: 使用 pytest 直接运行
|
||||
|
||||
```bash
|
||||
# 运行所有测试
|
||||
pytest tests/ -v
|
||||
|
||||
# 运行特定测试文件
|
||||
pytest tests/test_function_registry.py -v
|
||||
|
||||
# 运行特定测试类
|
||||
pytest tests/test_function_registry.py::TestFunctionRegistry -v
|
||||
|
||||
# 运行特定测试方法
|
||||
pytest tests/test_function_registry.py::TestFunctionRegistry::test_math_add -v
|
||||
```
|
||||
|
||||
### 方法 3: 在 Docker 容器中运行
|
||||
|
||||
```bash
|
||||
docker-compose exec mcp-server pytest tests/ -v
|
||||
```
|
||||
|
||||
## 📊 测试覆盖
|
||||
|
||||
### function_registry 测试
|
||||
|
||||
- ✅ 注册表初始化
|
||||
- ✅ 函数注册和查询
|
||||
- ✅ 内置函数测试（16个函数）
|
||||
- 数学函数（5个）
|
||||
- 字符串函数（4个）
|
||||
- JSON 函数（2个）
|
||||
- 哈希函数（2个）
|
||||
- Base64 函数（2个）
|
||||
- 日期时间函数（1个）
|
||||
|
||||
### sandbox_executor 测试
|
||||
|
||||
- ✅ 简单函数执行
|
||||
- ✅ 参数验证
|
||||
- ✅ 超时控制
|
||||
- ✅ 异常处理
|
||||
- ✅ 并发执行
|
||||
|
||||
### mcp_function_tool 测试
|
||||
|
||||
- ✅ 函数工具调用
|
||||
- ✅ 参数验证
|
||||
- ✅ 错误处理
|
||||
- ✅ 所有内置函数集成测试
|
||||
|
||||
## 📝 测试要求
|
||||
|
||||
- Python 3.8+
|
||||
- pytest 7.4.3+
|
||||
- pytest-asyncio 0.21.1+
|
||||
|
||||
## 🔧 配置
|
||||
|
||||
测试配置在 `pytest.ini` 文件中：
|
||||
|
||||
```ini
|
||||
[pytest]
|
||||
testpaths = tests
|
||||
python_files = test_*.py
|
||||
python_classes = Test*
|
||||
python_functions = test_*
|
||||
asyncio_mode = auto
|
||||
```
|
||||
|
||||
## 📈 测试报告
|
||||
|
||||
运行测试后，可以使用以下命令生成覆盖率报告：
|
||||
|
||||
```bash
|
||||
# 安装 coverage
|
||||
pip install pytest-cov
|
||||
|
||||
# 运行测试并生成覆盖率报告
|
||||
pytest tests/ --cov=. --cov-report=html
|
||||
|
||||
# 查看 HTML 报告
|
||||
open htmlcov/index.html
|
||||
```
|
||||
|
||||
## 🐛 调试测试
|
||||
|
||||
如果测试失败，可以使用以下选项获取更详细的输出：
|
||||
|
||||
```bash
|
||||
# 显示详细输出
|
||||
pytest tests/ -v -s
|
||||
|
||||
# 显示失败测试的完整堆栈跟踪
|
||||
pytest tests/ --tb=long
|
||||
|
||||
# 在第一个失败时停止
|
||||
pytest tests/ -x
|
||||
```
|
||||
|
||||
## 📚 添加新测试
|
||||
|
||||
添加新测试时，请遵循以下规范：
|
||||
|
||||
1. 测试文件命名：`test_<module_name>.py`
|
||||
2. 测试类命名：`Test<ClassName>`
|
||||
3. 测试方法命名：`test_<functionality>`
|
||||
4. 使用 `@pytest.mark.asyncio` 标记异步测试
|
||||
5. 使用 fixtures 共享测试数据
|
||||
|
||||
示例：
|
||||
|
||||
```python
|
||||
import pytest
|
||||
from module import Class
|
||||
|
||||
class TestClass:
|
||||
@pytest.fixture
|
||||
def instance(self):
|
||||
return Class()
|
||||
|
||||
def test_method(self, instance):
|
||||
assert instance.method() == expected
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_async_method(self, instance):
|
||||
result = await instance.async_method()
|
||||
assert result == expected
|
||||
```
|
||||
|
||||
@@ -1,4 +0,0 @@
|
||||
"""
|
||||
MCP Server 单元测试
|
||||
"""
|
||||
|
||||
@@ -1,20 +0,0 @@
|
||||
"""
|
||||
Pytest 配置和共享 fixtures
|
||||
"""
|
||||
|
||||
import pytest
|
||||
import sys
|
||||
import os
|
||||
|
||||
# 添加项目根目录到 Python 路径
|
||||
sys.path.insert(0, os.path.dirname(os.path.dirname(os.path.abspath(__file__))))
|
||||
|
||||
|
||||
@pytest.fixture(scope="session")
|
||||
def event_loop():
|
||||
"""创建事件循环"""
|
||||
import asyncio
|
||||
loop = asyncio.get_event_loop_policy().new_event_loop()
|
||||
yield loop
|
||||
loop.close()
|
||||
|
||||
@@ -1,242 +0,0 @@
|
||||
"""
|
||||
函数注册表单元测试
|
||||
"""
|
||||
|
||||
import pytest
|
||||
from function_registry import FunctionRegistry, get_function_registry
|
||||
|
||||
|
||||
class TestFunctionRegistry:
|
||||
"""函数注册表测试类"""
|
||||
|
||||
def test_registry_initialization(self):
|
||||
"""测试注册表初始化"""
|
||||
registry = FunctionRegistry()
|
||||
assert registry is not None
|
||||
assert len(registry.list_functions()) > 0
|
||||
|
||||
def test_list_functions(self):
|
||||
"""测试列出所有函数"""
|
||||
registry = get_function_registry()
|
||||
functions = registry.list_all()
|
||||
assert isinstance(functions, list)
|
||||
assert len(functions) > 0
|
||||
|
||||
# 检查函数结构
|
||||
for func_info in functions:
|
||||
assert "name" in func_info
|
||||
assert "description" in func_info
|
||||
assert "parameters" in func_info
|
||||
assert "returns" in func_info
|
||||
assert "func" not in func_info # 不应该包含实际函数对象
|
||||
|
||||
def test_get_function(self):
|
||||
"""测试获取函数信息"""
|
||||
registry = get_function_registry()
|
||||
# 测试存在的函数
|
||||
func_info = registry.get("math_add")
|
||||
assert func_info is not None
|
||||
assert func_info["name"] == "math_add"
|
||||
assert "func" in func_info
|
||||
assert "description" in func_info
|
||||
assert "parameters" in func_info
|
||||
|
||||
# 测试不存在的函数
|
||||
func_info = registry.get("non_existent_function")
|
||||
assert func_info is None
|
||||
|
||||
def test_register_function(self):
|
||||
"""测试注册新函数"""
|
||||
registry = FunctionRegistry()
|
||||
|
||||
def test_func(x: int, y: int) -> int:
|
||||
"""测试函数"""
|
||||
return x + y
|
||||
|
||||
registry.register(
|
||||
name="test_add",
|
||||
func=test_func,
|
||||
description="测试加法函数",
|
||||
parameters=[
|
||||
{"name": "x", "type": "integer", "description": "第一个数"},
|
||||
{"name": "y", "type": "integer", "description": "第二个数"}
|
||||
],
|
||||
returns={"type": "integer", "description": "两数之和"}
|
||||
)
|
||||
|
||||
func_info = registry.get_function("test_add")
|
||||
assert func_info is not None
|
||||
assert func_info["name"] == "test_add"
|
||||
assert func_info["description"] == "测试加法函数"
|
||||
|
||||
# 测试函数可以执行
|
||||
result = func_info["func"](10, 20)
|
||||
assert result == 30
|
||||
|
||||
def test_register_duplicate_function(self):
|
||||
"""测试注册重复函数（应该覆盖）"""
|
||||
registry = FunctionRegistry()
|
||||
|
||||
def func1():
|
||||
return 1
|
||||
|
||||
def func2():
|
||||
return 2
|
||||
|
||||
registry.register(
|
||||
name="duplicate_test",
|
||||
func=func1,
|
||||
description="第一个函数",
|
||||
parameters=[],
|
||||
returns={"type": "integer"}
|
||||
)
|
||||
|
||||
registry.register(
|
||||
name="duplicate_test",
|
||||
func=func2,
|
||||
description="第二个函数",
|
||||
parameters=[],
|
||||
returns={"type": "integer"}
|
||||
)
|
||||
|
||||
func_info = registry.get_function("duplicate_test")
|
||||
assert func_info["func"]() == 2 # 应该是第二个函数
|
||||
|
||||
def test_builtin_math_functions(self):
|
||||
"""测试内置数学函数"""
|
||||
registry = get_function_registry()
|
||||
# math_add
|
||||
func_info = registry.get("math_add")
|
||||
assert func_info is not None
|
||||
result = func_info["func"](10, 20)
|
||||
assert result == 30.0
|
||||
|
||||
# math_subtract
|
||||
func_info = registry.get("math_subtract")
|
||||
assert func_info is not None
|
||||
result = func_info["func"](20, 10)
|
||||
assert result == 10.0
|
||||
|
||||
# math_multiply
|
||||
func_info = registry.get("math_multiply")
|
||||
assert func_info is not None
|
||||
result = func_info["func"](5, 6)
|
||||
assert result == 30.0
|
||||
|
||||
# math_divide
|
||||
func_info = registry.get("math_divide")
|
||||
assert func_info is not None
|
||||
result = func_info["func"](20, 4)
|
||||
assert result == 5.0
|
||||
|
||||
# math_power
|
||||
func_info = registry.get("math_power")
|
||||
assert func_info is not None
|
||||
result = func_info["func"](2, 3)
|
||||
assert result == 8.0
|
||||
|
||||
def test_builtin_string_functions(self):
|
||||
"""测试内置字符串函数"""
|
||||
registry = get_function_registry()
|
||||
# string_upper
|
||||
func_info = registry.get("string_upper")
|
||||
assert func_info is not None
|
||||
result = func_info["func"]("hello")
|
||||
assert result == "HELLO"
|
||||
|
||||
# string_lower
|
||||
func_info = registry.get("string_lower")
|
||||
assert func_info is not None
|
||||
result = func_info["func"]("WORLD")
|
||||
assert result == "world"
|
||||
|
||||
# string_length
|
||||
func_info = registry.get("string_length")
|
||||
assert func_info is not None
|
||||
result = func_info["func"]("test")
|
||||
assert result == 4
|
||||
|
||||
# string_replace
|
||||
func_info = registry.get("string_replace")
|
||||
assert func_info is not None
|
||||
result = func_info["func"]("hello world", "world", "python")
|
||||
assert result == "hello python"
|
||||
|
||||
def test_builtin_json_functions(self):
|
||||
"""测试内置JSON函数"""
|
||||
registry = get_function_registry()
|
||||
# json_stringify
|
||||
func_info = registry.get("json_stringify")
|
||||
assert func_info is not None
|
||||
result = func_info["func"]({"key": "value"})
|
||||
assert result == '{"key": "value"}'
|
||||
|
||||
# json_parse
|
||||
func_info = registry.get("json_parse")
|
||||
assert func_info is not None
|
||||
result = func_info["func"]('{"key": "value"}')
|
||||
assert result == {"key": "value"}
|
||||
|
||||
def test_builtin_hash_functions(self):
|
||||
"""测试内置哈希函数"""
|
||||
registry = get_function_registry()
|
||||
# hash_md5
|
||||
func_info = registry.get("hash_md5")
|
||||
assert func_info is not None
|
||||
result = func_info["func"]("test")
|
||||
assert len(result) == 32 # MD5 哈希长度为 32
|
||||
assert isinstance(result, str)
|
||||
|
||||
# hash_sha256
|
||||
func_info = registry.get("hash_sha256")
|
||||
assert func_info is not None
|
||||
result = func_info["func"]("test")
|
||||
assert len(result) == 64 # SHA256 哈希长度为 64
|
||||
assert isinstance(result, str)
|
||||
|
||||
def test_builtin_base64_functions(self):
|
||||
"""测试内置Base64函数"""
|
||||
registry = get_function_registry()
|
||||
# base64_encode
|
||||
func_info = registry.get("base64_encode")
|
||||
assert func_info is not None
|
||||
result = func_info["func"]("hello")
|
||||
assert isinstance(result, str)
|
||||
|
||||
# base64_decode
|
||||
func_info = registry.get("base64_decode")
|
||||
assert func_info is not None
|
||||
encoded = registry.get("base64_encode")["func"]("hello")
|
||||
result = func_info["func"](encoded)
|
||||
assert result == "hello"
|
||||
|
||||
def test_builtin_datetime_function(self):
|
||||
"""测试内置日期时间函数"""
|
||||
registry = get_function_registry()
|
||||
func_info = registry.get("datetime_now")
|
||||
assert func_info is not None
|
||||
result = func_info["func"]()
|
||||
assert isinstance(result, str)
|
||||
assert "T" in result or "-" in result # ISO格式
|
||||
|
||||
def test_function_count(self):
|
||||
"""测试函数数量"""
|
||||
registry = get_function_registry()
|
||||
functions = registry.list_all()
|
||||
# 应该至少有16个内置函数
|
||||
assert len(functions) >= 16
|
||||
|
||||
# 检查所有预期的函数都存在
|
||||
expected_functions = [
|
||||
"math_add", "math_subtract", "math_multiply", "math_divide", "math_power",
|
||||
"string_upper", "string_lower", "string_length", "string_replace",
|
||||
"datetime_now",
|
||||
"json_parse", "json_stringify",
|
||||
"hash_md5", "hash_sha256",
|
||||
"base64_encode", "base64_decode"
|
||||
]
|
||||
|
||||
function_names = [f["name"] for f in functions]
|
||||
for expected_func in expected_functions:
|
||||
assert expected_func in function_names, f"函数 {expected_func} 未找到"
|
||||
|
||||
@@ -1,216 +0,0 @@
|
||||
"""
|
||||
MCP 协议函数工具调用单元测试
|
||||
"""
|
||||
|
||||
import pytest
|
||||
from unittest.mock import Mock, AsyncMock, patch
|
||||
from mcp_protocol import MCPProtocolHandler
|
||||
from function_registry import get_function_registry
|
||||
from sandbox_executor import get_sandbox_executor
|
||||
|
||||
|
||||
class TestMCPFunctionTool:
|
||||
"""MCP 函数工具调用测试类"""
|
||||
|
||||
@pytest.fixture
|
||||
def mcp_server(self):
|
||||
"""创建 MCP Protocol Handler 实例"""
|
||||
from unittest.mock import MagicMock
|
||||
redis_mock = MagicMock()
|
||||
nats_mock = MagicMock()
|
||||
handler = MCPProtocolHandler(redis_client=redis_mock, nats_client=nats_mock)
|
||||
return handler
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_execute_function_tool_math_add(self, mcp_server):
|
||||
"""测试执行数学加法函数"""
|
||||
tool_info = {
|
||||
"name": "math_add",
|
||||
"timeout": 5
|
||||
}
|
||||
arguments = {"a": 10, "b": 20}
|
||||
|
||||
result = await mcp_server._execute_function_tool(tool_info, arguments)
|
||||
assert result == 30.0
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_execute_function_tool_string_upper(self, mcp_server):
|
||||
"""测试执行字符串大写函数"""
|
||||
tool_info = {
|
||||
"name": "string_upper",
|
||||
"timeout": 5
|
||||
}
|
||||
arguments = {"s": "hello world"}
|
||||
|
||||
result = await mcp_server._execute_function_tool(tool_info, arguments)
|
||||
assert result == "HELLO WORLD"
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_execute_function_tool_missing_function_name(self, mcp_server):
|
||||
"""测试缺少函数名称的情况"""
|
||||
tool_info = {}
|
||||
arguments = {"a": 10, "b": 20}
|
||||
|
||||
with pytest.raises(ValueError, match="函数工具必须指定"):
|
||||
await mcp_server._execute_function_tool(tool_info, arguments)
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_execute_function_tool_unregistered_function(self, mcp_server):
|
||||
"""测试未注册的函数"""
|
||||
tool_info = {
|
||||
"name": "non_existent_function",
|
||||
"timeout": 5
|
||||
}
|
||||
arguments = {"a": 10}
|
||||
|
||||
with pytest.raises(ValueError, match="未在注册表中"):
|
||||
await mcp_server._execute_function_tool(tool_info, arguments)
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_execute_function_tool_missing_required_parameter(self, mcp_server):
|
||||
"""测试缺少必需参数"""
|
||||
tool_info = {
|
||||
"name": "math_add",
|
||||
"timeout": 5
|
||||
}
|
||||
arguments = {"a": 10} # 缺少 b 参数
|
||||
|
||||
# 注意：由于参数验证可能允许部分参数，这里可能需要调整
|
||||
# 如果函数定义中 b 是必需的，应该会抛出异常
|
||||
try:
|
||||
result = await mcp_server._execute_function_tool(tool_info, arguments)
|
||||
# 如果没有抛出异常，说明参数验证允许部分参数
|
||||
# 这种情况下函数可能会使用默认值或报错
|
||||
except (ValueError, TypeError) as e:
|
||||
# 参数验证失败是预期的
|
||||
assert "参数" in str(e) or "missing" in str(e).lower()
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_execute_function_tool_timeout(self, mcp_server):
|
||||
"""测试函数执行超时"""
|
||||
# 注册一个会超时的函数
|
||||
def slow_function():
|
||||
import time
|
||||
time.sleep(10)
|
||||
return "done"
|
||||
|
||||
registry = get_function_registry()
|
||||
registry.register(
|
||||
name="slow_test",
|
||||
func=slow_function,
|
||||
description="慢速测试函数",
|
||||
parameters=[],
|
||||
returns={"type": "string"}
|
||||
)
|
||||
|
||||
tool_info = {
|
||||
"name": "slow_test",
|
||||
"timeout": 1 # 1秒超时
|
||||
}
|
||||
arguments = {}
|
||||
|
||||
with pytest.raises(Exception): # 可能是 TimeoutError 或其他异常
|
||||
await mcp_server._execute_function_tool(tool_info, arguments)
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_execute_function_tool_with_all_builtin_functions(self, mcp_server):
|
||||
"""测试所有内置函数"""
|
||||
test_cases = [
|
||||
("math_add", {"a": 5, "b": 3}, 8.0),
|
||||
("math_subtract", {"a": 10, "b": 4}, 6.0),
|
||||
("math_multiply", {"a": 6, "b": 7}, 42.0),
|
||||
("math_divide", {"a": 20, "b": 4}, 5.0),
|
||||
("math_power", {"a": 2, "b": 3}, 8.0),
|
||||
("string_upper", {"s": "hello"}, "HELLO"),
|
||||
("string_lower", {"s": "WORLD"}, "world"),
|
||||
("string_length", {"s": "test"}, 4),
|
||||
("string_replace", {"s": "hello", "old": "l", "new": "L"}, "heLLo"),
|
||||
]
|
||||
|
||||
for func_name, args, expected in test_cases:
|
||||
tool_info = {
|
||||
"name": func_name,
|
||||
"timeout": 5
|
||||
}
|
||||
result = await mcp_server._execute_function_tool(tool_info, args)
|
||||
assert result == expected, f"函数 {func_name} 执行结果不正确"
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_execute_function_tool_json_functions(self, mcp_server):
|
||||
"""测试JSON函数"""
|
||||
# json_stringify
|
||||
tool_info = {
|
||||
"name": "json_stringify",
|
||||
"timeout": 5
|
||||
}
|
||||
arguments = {"obj": {"key": "value", "number": 123}}
|
||||
result = await mcp_server._execute_function_tool(tool_info, arguments)
|
||||
assert isinstance(result, str)
|
||||
assert "key" in result
|
||||
assert "value" in result
|
||||
|
||||
# json_parse
|
||||
tool_info = {
|
||||
"name": "json_parse",
|
||||
"timeout": 5
|
||||
}
|
||||
arguments = {"s": '{"key": "value"}'}
|
||||
result = await mcp_server._execute_function_tool(tool_info, arguments)
|
||||
assert isinstance(result, dict)
|
||||
assert result["key"] == "value"
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_execute_function_tool_hash_functions(self, mcp_server):
|
||||
"""测试哈希函数"""
|
||||
# hash_md5
|
||||
tool_info = {
|
||||
"name": "hash_md5",
|
||||
"timeout": 5
|
||||
}
|
||||
arguments = {"s": "test"}
|
||||
result = await mcp_server._execute_function_tool(tool_info, arguments)
|
||||
assert isinstance(result, str)
|
||||
assert len(result) == 32
|
||||
|
||||
# hash_sha256
|
||||
tool_info = {
|
||||
"name": "hash_sha256",
|
||||
"timeout": 5
|
||||
}
|
||||
result = await mcp_server._execute_function_tool(tool_info, arguments)
|
||||
assert isinstance(result, str)
|
||||
assert len(result) == 64
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_execute_function_tool_base64_functions(self, mcp_server):
|
||||
"""测试Base64函数"""
|
||||
# base64_encode
|
||||
tool_info = {
|
||||
"name": "base64_encode",
|
||||
"timeout": 5
|
||||
}
|
||||
arguments = {"s": "hello"}
|
||||
encoded = await mcp_server._execute_function_tool(tool_info, arguments)
|
||||
assert isinstance(encoded, str)
|
||||
|
||||
# base64_decode
|
||||
tool_info = {
|
||||
"name": "base64_decode",
|
||||
"timeout": 5
|
||||
}
|
||||
arguments = {"s": encoded}
|
||||
decoded = await mcp_server._execute_function_tool(tool_info, arguments)
|
||||
assert decoded == "hello"
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_execute_function_tool_datetime_function(self, mcp_server):
|
||||
"""测试日期时间函数"""
|
||||
tool_info = {
|
||||
"name": "datetime_now",
|
||||
"timeout": 5
|
||||
}
|
||||
arguments = {}
|
||||
result = await mcp_server._execute_function_tool(tool_info, arguments)
|
||||
assert isinstance(result, str)
|
||||
assert "T" in result or "-" in result # ISO格式
|
||||
|
||||
@@ -1,173 +0,0 @@
|
||||
"""
|
||||
沙箱执行器单元测试
|
||||
"""
|
||||
|
||||
import pytest
|
||||
import asyncio
|
||||
from sandbox_executor import SandboxExecutor
|
||||
|
||||
|
||||
class TestSandboxExecutor:
|
||||
"""沙箱执行器测试类"""
|
||||
|
||||
@pytest.fixture
|
||||
def executor(self):
|
||||
"""创建沙箱执行器实例"""
|
||||
return SandboxExecutor(timeout=2.0, max_memory_mb=100)
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_execute_simple_function(self, executor):
|
||||
"""测试执行简单函数"""
|
||||
def add(a, b):
|
||||
return a + b
|
||||
|
||||
result = await executor.execute(add, {"a": 10, "b": 20}, "add")
|
||||
assert result == 30
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_execute_with_correct_parameters(self, executor):
|
||||
"""测试使用正确参数执行函数"""
|
||||
def multiply(x, y):
|
||||
return x * y
|
||||
|
||||
result = await executor.execute(multiply, {"x": 5, "y": 6}, "multiply")
|
||||
assert result == 30
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_execute_with_missing_parameters(self, executor):
|
||||
"""测试缺少参数的情况"""
|
||||
def func(a, b, c):
|
||||
return a + b + c
|
||||
|
||||
with pytest.raises(TypeError):
|
||||
await executor.execute(func, {"a": 1, "b": 2}, "func")
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_execute_with_extra_parameters(self, executor):
|
||||
"""测试多余参数的情况（应该被忽略）"""
|
||||
def func(a, b):
|
||||
return a + b
|
||||
|
||||
# 传递额外参数应该不影响执行
|
||||
result = await executor.execute(func, {"a": 1, "b": 2, "c": 3}, "func")
|
||||
assert result == 3
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_execute_timeout(self, executor):
|
||||
"""测试执行超时"""
|
||||
def slow_function():
|
||||
import time
|
||||
time.sleep(5) # 休眠5秒
|
||||
return "done"
|
||||
|
||||
with pytest.raises(asyncio.TimeoutError):
|
||||
await executor.execute(slow_function, {}, "slow_function")
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_execute_with_exception(self, executor):
|
||||
"""测试函数抛出异常的情况"""
|
||||
def error_function():
|
||||
raise ValueError("测试错误")
|
||||
|
||||
with pytest.raises(ValueError, match="测试错误"):
|
||||
await executor.execute(error_function, {}, "error_function")
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_execute_string_function(self, executor):
|
||||
"""测试字符串处理函数"""
|
||||
def upper_case(s):
|
||||
return s.upper()
|
||||
|
||||
result = await executor.execute(upper_case, {"s": "hello"}, "upper_case")
|
||||
assert result == "HELLO"
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_execute_list_function(self, executor):
|
||||
"""测试列表处理函数"""
|
||||
def sum_list(numbers):
|
||||
return sum(numbers)
|
||||
|
||||
result = await executor.execute(sum_list, {"numbers": [1, 2, 3, 4, 5]}, "sum_list")
|
||||
assert result == 15
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_execute_dict_function(self, executor):
|
||||
"""测试字典处理函数"""
|
||||
def get_value(data, key):
|
||||
return data.get(key)
|
||||
|
||||
result = await executor.execute(
|
||||
get_value,
|
||||
{"data": {"name": "test", "value": 123}, "key": "value"},
|
||||
"get_value"
|
||||
)
|
||||
assert result == 123
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_execute_nested_function(self, executor):
|
||||
"""测试嵌套函数调用"""
|
||||
def outer(x):
|
||||
def inner(y):
|
||||
return y * 2
|
||||
return inner(x) + 10
|
||||
|
||||
result = await executor.execute(outer, {"x": 5}, "outer")
|
||||
assert result == 20
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_execute_with_type_conversion(self, executor):
|
||||
"""测试类型转换"""
|
||||
def add_strings(a, b):
|
||||
return str(a) + str(b)
|
||||
|
||||
result = await executor.execute(add_strings, {"a": 123, "b": 456}, "add_strings")
|
||||
assert result == "123456"
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_execute_with_none(self, executor):
|
||||
"""测试处理None值"""
|
||||
def return_none():
|
||||
return None
|
||||
|
||||
result = await executor.execute(return_none, {}, "return_none")
|
||||
assert result is None
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_execute_with_empty_dict(self, executor):
|
||||
"""测试空参数字典"""
|
||||
def no_params():
|
||||
return "success"
|
||||
|
||||
result = await executor.execute(no_params, {}, "no_params")
|
||||
assert result == "success"
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_custom_timeout(self):
|
||||
"""测试自定义超时时间"""
|
||||
executor = SandboxExecutor(timeout=0.5)
|
||||
|
||||
def slow_function():
|
||||
import time
|
||||
time.sleep(1)
|
||||
return "done"
|
||||
|
||||
with pytest.raises(asyncio.TimeoutError):
|
||||
await executor.execute(slow_function, {}, "slow_function")
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_concurrent_executions(self, executor):
|
||||
"""测试并发执行"""
|
||||
def add(a, b):
|
||||
return a + b
|
||||
|
||||
# 并发执行多个函数
|
||||
tasks = [
|
||||
executor.execute(add, {"a": i, "b": i+1}, f"add_{i}")
|
||||
for i in range(10)
|
||||
]
|
||||
|
||||
results = await asyncio.gather(*tasks)
|
||||
assert len(results) == 10
|
||||
assert results[0] == 1 # 0 + 1
|
||||
assert results[9] == 19 # 9 + 10
|
||||
|
||||
Reference in New Issue
Block a user