taijibaga/taiji-AI-PAD
9
0
Fork 0
forked from xiaohei/taiji-AI-PAD
Code Pull Requests Activity

feat: 完成taiji-AI-PAD核心架构和代码实现

Browse Source 
✨ 新功能:
- 实现五层技术架构的容器化部署
- 完成MCP Server核心服务 (Python/FastAPI)
- 实现数据接入服务框架 (RapidAPI/APILLAMA)
- 配置LiteLLM模型网关 (100+模型支持)
- 设置完整的监控体系 (Prometheus/Grafana)

🏗️ 基础设施:
- Docker Compose编排配置
- Nginx API网关
- PostgreSQL + Redis + NATS
- 完整的启动/停止/测试脚本

📚 文档:
- 工程排期计划
- 任务拆分与分工
- 系统运作流程图
- 项目开发状态文档

🔧 开发工具:
- 容器化开发环境
- 自动化测试脚本
- API文档自动生成
- 健康检查机制

该提交包含了约60%的项目核心功能实现，可以进行基础的Agent管理和MCP协议通信。
This commit is contained in:
xiaohei
2025-12-20 05:31:03 +00:00
parent dd4417e34c
commit cd0719e1cb
26 changed files with 5928 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files
Docs/任务拆分与分工.md
+230
View File
@@ -0,0 +1,230 @@
# taiji-AI-PAD 任务拆分与分工
## 📋 任务分解结构 (WBS)
### 1️⃣ 第一平面：全域数据接入与工具化治理
#### 1.1 RapidAPI生态集成模块
| 任务ID | 任务名称 | 负责角色 | 预计工时 | 优先级 | 依赖关系 |
|--------|----------|----------|----------|--------|----------|
| T1.1.1 | RapidAPI SDK集成与认证 | 后端开发工程师 | 40h | P0 | - |
| T1.1.2 | 统一API Key代理服务 | 后端开发工程师 | 32h | P0 | T1.1.1 |
| T1.1.3 | API调用成本跟踪 | 后端开发工程师 | 24h | P1 | T1.1.2 |
| T1.1.4 | 16000+ API元数据管理 | 数据工程师 | 56h | P1 | T1.1.1 |
#### 1.2 APILLAMA技术栈
| 任务ID | 任务名称 | 负责角色 | 预计工时 | 优先级 | 依赖关系 |
|--------|----------|----------|----------|--------|----------|
| T1.2.1 | Llama-3-8B-Instruct模型部署 | AI工程师 | 48h | P0 | - |
| T1.2.2 | 软提示技术实现 | AI工程师 | 40h | P0 | T1.2.1 |
| T1.2.3 | API文档→Pydantic转换器 | 后端开发工程师 | 64h | P0 | T1.2.2 |
| T1.2.4 | JSON Schema生成引擎 | 后端开发工程师 | 32h | P1 | T1.2.3 |
| T1.2.5 | 语义增强与幻觉消除 | AI工程师 | 56h | P1 | T1.2.3 |
#### 1.3 异构数据源管理
| 任务ID | 任务名称 | 负责角色 | 预计工时 | 优先级 | 依赖关系 |
|--------|----------|----------|----------|--------|----------|
| T1.3.1 | OpenAPI/Swagger解析器 | 后端开发工程师 | 40h | P0 | - |
| T1.3.2 | FastMCP工具集成 | 后端开发工程师 | 32h | P1 | T1.3.1 |
| T1.3.3 | 动态热加载机制 | 后端开发工程师 | 48h | P1 | T1.3.2 |
| T1.3.4 | 私有API接入框架 | 后端开发工程师 | 40h | P2 | T1.3.1 |
---
### 2️⃣ 第二平面：模型抽象层与动态治理
#### 2.1 LiteLLM网关集成
| 任务ID | 任务名称 | 负责角色 | 预计工时 | 优先级 | 依赖关系 |
|--------|----------|----------|----------|--------|----------|
| T2.1.1 | LiteLLM Proxy服务搭建 | DevOps工程师 | 32h | P0 | - |
| T2.1.2 | 100+模型API适配 | 后端开发工程师 | 80h | P0 | T2.1.1 |
| T2.1.3 | OpenAI兼容端点开发 | 后端开发工程师 | 40h | P0 | T2.1.2 |
| T2.1.4 | 模型组(Model Groups)配置 | 后端开发工程师 | 24h | P1 | T2.1.3 |
#### 2.2 高可用路由系统
| 任务ID | 任务名称 | 负责角色 | 预计工时 | 优先级 | 依赖关系 |
|--------|----------|----------|----------|--------|----------|
| T2.2.1 | 负载均衡算法实现 | 后端开发工程师 | 48h | P0 | T2.1.4 |
| T2.2.2 | 故障转移机制 | 后端开发工程师 | 56h | P0 | T2.2.1 |
| T2.2.3 | 跨服务商切换逻辑 | 后端开发工程师 | 40h | P1 | T2.2.2 |
| T2.2.4 | 健康检查与监控 | DevOps工程师 | 32h | P1 | T2.2.3 |
#### 2.3 上下文管理
| 任务ID | 任务名称 | 负责角色 | 预计工时 | 优先级 | 依赖关系 |
|--------|----------|----------|----------|--------|----------|
| T2.3.1 | Token限制检测器 | 后端开发工程师 | 32h | P0 | - |
| T2.3.2 | 会话截断算法 | AI工程师 | 48h | P0 | T2.3.1 |
| T2.3.3 | 上下文总结逻辑 | AI工程师 | 40h | P1 | T2.3.2 |
| T2.3.4 | 成本归因分析 | 后端开发工程师 | 36h | P1 | T2.3.1 |
---
### 3️⃣ 第三平面：单体Agent协议化封装
#### 3.1 MCP协议实现
| 任务ID | 任务名称 | 负责角色 | 预计工时 | 优先级 | 依赖关系 |
|--------|----------|----------|----------|--------|----------|
| T3.1.1 | MCP Server核心框架 | 后端开发工程师 | 64h | P0 | - |
| T3.1.2 | JSON-RPC 2.0通信层 | 后端开发工程师 | 48h | P0 | T3.1.1 |
| T3.1.3 | stdio传输支持 | 后端开发工程师 | 32h | P0 | T3.1.2 |
| T3.1.4 | SSE流式传输 | 后端开发工程师 | 40h | P0 | T3.1.2 |
| T3.1.5 | MCP Client适配器 | 后端开发工程师 | 48h | P1 | T3.1.4 |
#### 3.2 A2A通信协议
| 任务ID | 任务名称 | 负责角色 | 预计工时 | 优先级 | 依赖关系 |
|--------|----------|----------|----------|--------|----------|
| T3.2.1 | Agent Card生成器 | 后端开发工程师 | 40h | P0 | - |
| T3.2.2 | 代理发现机制 | 后端开发工程师 | 48h | P0 | T3.2.1 |
| T3.2.3 | 任务生命周期管理 | 后端开发工程师 | 56h | P0 | T3.2.2 |
| T3.2.4 | 工件(Artifacts)交换 | 后端开发工程师 | 44h | P1 | T3.2.3 |
| T3.2.5 | 多部分数据流处理 | 后端开发工程师 | 36h | P1 | T3.2.4 |
#### 3.3 Agent原子化设计
| 任务ID | 任务名称 | 负责角色 | 预计工时 | 优先级 | 依赖关系 |
|--------|----------|----------|----------|--------|----------|
| T3.3.1 | Role-Goal-Tools框架 | 架构师 | 32h | P0 | - |
| T3.3.2 | Agent注册与认证 | 后端开发工程师 | 40h | P0 | T3.3.1 |
| T3.3.3 | 工具权限管理 | 后端开发工程师 | 48h | P1 | T3.3.2 |
| T3.3.4 | Agent版本控制 | 后端开发工程师 | 32h | P2 | T3.3.3 |
---
### 4️⃣ 第四平面：MCP为核心的本地编排
#### 4.1 主流框架适配
| 任务ID | 任务名称 | 负责角色 | 预计工时 | 优先级 | 依赖关系 |
|--------|----------|----------|----------|--------|----------|
| T4.1.1 | LangChain MCP适配器 | 后端开发工程师 | 56h | P0 | T3.1.5 |
| T4.1.2 | CrewAI集成机制 | 后端开发工程师 | 48h | P0 | T3.1.5 |
| T4.1.3 | AutoGen StdioMcp适配 | 后端开发工程师 | 52h | P0 | T3.1.5 |
| T4.1.4 | MultiServerMCPClient | 后端开发工程师 | 40h | P1 | T4.1.1 |
#### 4.2 IDE与客户端支持
| 任务ID | 任务名称 | 负责角色 | 预计工时 | 优先级 | 依赖关系 |
|--------|----------|----------|----------|--------|----------|
| T4.2.1 | Cursor IDE集成 | 前端开发工程师 | 48h | P0 | T3.1.4 |
| T4.2.2 | Claude Desktop适配 | 前端开发工程师 | 40h | P1 | T3.1.4 |
| T4.2.3 | VS Code扩展开发 | 前端开发工程师 | 64h | P2 | T4.2.1 |
| T4.2.4 | Web管理界面 | 前端开发工程师 | 80h | P1 | T4.2.1 |
#### 4.3 动态发现与编排
| 任务ID | 任务名称 | 负责角色 | 预计工时 | 优先级 | 依赖关系 |
|--------|----------|----------|----------|--------|----------|
| T4.3.1 | tools/list动态发现 | 后端开发工程师 | 32h | P0 | T3.1.4 |
| T4.3.2 | 热加载机制 | 后端开发工程师 | 40h | P1 | T4.3.1 |
| T4.3.3 | 编排DSL设计 | 架构师 | 48h | P1 | T4.3.2 |
| T4.3.4 | 可视化编排界面 | 前端开发工程师 | 72h | P2 | T4.3.3 |
---
### 5️⃣ 第五平面：EU计费与治理
#### 5.1 执行单元(EU)计费
| 任务ID | 任务名称 | 负责角色 | 预计工时 | 优先级 | 依赖关系 |
|--------|----------|----------|----------|--------|----------|
| T5.1.1 | EU计算公式实现 | 后端开发工程师 | 48h | P0 | - |
| T5.1.2 | 资源使用监控 | DevOps工程师 | 56h | P0 | T5.1.1 |
| T5.1.3 | NATS事件采集 | 后端开发工程师 | 40h | P0 | T5.1.2 |
| T5.1.4 | 预付费配额管理 | 后端开发工程师 | 44h | P1 | T5.1.3 |
| T5.1.5 | 实时计费仪表盘 | 前端开发工程师 | 64h | P1 | T5.1.4 |
#### 5.2 安全隔离机制
| 任务ID | 任务名称 | 负责角色 | 预计工时 | 优先级 | 依赖关系 |
|--------|----------|----------|----------|--------|----------|
| T5.2.1 | Firecracker MicroVM集成 | DevOps工程师 | 72h | P0 | - |
| T5.2.2 | gVisor容器隔离 | DevOps工程师 | 64h | P1 | T5.2.1 |
| T5.2.3 | 多租户数据隔离(RLS) | 后端开发工程师 | 56h | P0 | T5.2.1 |
| T5.2.4 | 按租户加密机制 | 安全工程师 | 48h | P1 | T5.2.3 |
| T5.2.5 | 网络VPC隔离 | DevOps工程师 | 40h | P1 | T5.2.1 |
#### 5.3 身份认证与权限
| 任务ID | 任务名称 | 负责角色 | 预计工时 | 优先级 | 依赖关系 |
|--------|----------|----------|----------|--------|----------|
| T5.3.1 | Pomerium网关部署 | DevOps工程师 | 40h | P0 | - |
| T5.3.2 | Okta身份提供商集成 | 后端开发工程师 | 48h | P1 | T5.3.1 |
| T5.3.3 | RBAC/ABAC权限系统 | 后端开发工程师 | 64h | P0 | T5.3.2 |
| T5.3.4 | 上下文访问策略 | 安全工程师 | 36h | P1 | T5.3.3 |
#### 5.4 监控与审计
| 任务ID | 任务名称 | 负责角色 | 预计工时 | 优先级 | 依赖关系 |
|--------|----------|----------|----------|--------|----------|
| T5.4.1 | Datadog/Prometheus集成 | DevOps工程师 | 48h | P0 | - |
| T5.4.2 | Agent轨迹追踪 | 后端开发工程师 | 56h | P1 | T5.4.1 |
| T5.4.3 | 合规审计日志 | 后端开发工程师 | 44h | P1 | T5.4.2 |
| T5.4.4 | SOC2/HIPAA合规 | 合规专员 | 80h | P2 | T5.4.3 |
---
## 👥 角色职责分配
### 核心团队角色
#### 架构师 (1人)
- **主要职责**: 技术架构设计、关键技术决策、跨模块协调
- **核心任务**: T3.3.1, T4.3.3
- **技能要求**: 分布式系统、AI架构、协议设计
#### 后端开发工程师 (4-5人)
- **Team Lead**: 负责API设计与核心业务逻辑
- **AI专家**: 专注APILLAMA与模型相关功能
- **协议专家**: 负责MCP/A2A协议实现
- **业务开发**: 负责Agent管理与编排功能
- **计费专家**: 专注EU计费与权限系统
#### 前端开发工程师 (2人)
- **UI/UX专家**: 负责管理界面与可视化编排
- **集成专家**: 负责IDE插件与客户端适配
#### DevOps工程师 (2人)
- **基础设施专家**: 负责容器化、安全隔离
- **监控专家**: 负责可观测性与运维工具
#### 测试工程师 (2人)
- **自动化测试**: 单元测试、集成测试
- **性能测试**: 压力测试、安全测试
## ⏱️ 工时统计与分配
### 按技术平面统计
| 技术平面 | 总工时 | 占比 |
|----------|--------|------|
| 第一平面 (数据接入) | 464h | 22% |
| 第二平面 (模型治理) | 396h | 19% |
| 第三平面 (Agent封装) | 528h | 25% |
| 第四平面 (本地编排) | 448h | 21% |
| 第五平面 (计费治理) | 700h | 33% |
| **总计** | **2536h** | **100%** |
### 按优先级统计
| 优先级 | 任务数 | 工时 | 占比 |
|--------|--------|------|------|
| P0 (核心功能) | 32 | 1456h | 57% |
| P1 (重要功能) | 28 | 868h | 34% |
| P2 (增强功能) | 8 | 212h | 9% |
## 📊 里程碑与交付物
### 主要里程碑
1. **M1**: 数据接入层完成 (3个月)
2. **M2**: 模型治理层完成 (6个月)
3. **M3**: Agent协议完成 (10个月)
4. **M4**: 集成平台完成 (13个月)
5. **M5**: 计费治理完成 (18个月)
6. **M6**: 系统上线运行 (20个月)
### 关键交付物
- [ ] APILLAMA模型部署包
- [ ] LiteLLM统一网关
- [ ] MCP/A2A协议SDK
- [ ] 主流框架适配器
- [ ] EU计费引擎
- [ ] 多租户安全方案
- [ ] 监控与运维工具包
- [ ] 技术文档与培训材料
---
**创建时间**: 2025年12月20日
**版本**: v1.0
**负责人**: 项目组
**下次更新**: 每两周更新任务进度
Docs/工程排期计划.md
+152
View File
@@ -0,0 +1,152 @@
# taiji-AI-PAD 工程排期计划
## 📋 项目总览
**项目名称**: Agent 赋能平台 (taiji-AI-PAD)
**项目类型**: 全栈工程化平台
**技术架构**: 五层技术平面
**预计总工期**: 18-24个月
**团队规模建议**: 12-15人
## 🎯 项目目标
构建一个将AI Agents从实验性脚本演进为工业级生产力单元的全栈工程化平台，通过标准化的智力资源分发与治理体系，整合异构数据，支持多模型动态切换，并具备透明的计费与安全隔离机制。
## 📅 分阶段排期
### Phase 1: 基础设施与数据接入层 (3-4个月)
**时间**: 2025年1月 - 2025年4月
**关键里程碑**:
- 完成全域数据接入系统
- 实现APILLAMA技术栈
- 建立RapidAPI生态集成
**详细排期**:
- **Week 1-2**: 项目初始化与开发环境搭建
- **Week 3-6**: RapidAPI集成与统一API Key管理
- **Week 7-10**: APILLAMA模型部署与API文档转换
- **Week 11-14**: OpenAPI/Swagger动态加载机制
- **Week 15-16**: 第一阶段测试与优化
### Phase 2: 模型抽象与治理层 (2-3个月)
**时间**: 2025年4月 - 2025年7月
**关键里程碑**:
- LiteLLM网关部署
- 多模型路由与负载均衡
- 上下文管理与成本控制
**详细排期**:
- **Week 1-3**: LiteLLM集成与100+模型API支持
- **Week 4-6**: 高可用路由与故障转移机制
- **Week 7-9**: 上下文窗口管理与会话截断
- **Week 10-12**: 性能监控与链路追踪集成
### Phase 3: Agent协议化封装 (3-4个月)
**时间**: 2025年7月 - 2025年11月
**关键里程碑**:
- MCP协议实现
- A2A通信协议支持
- 单体Agent标准化
**详细排期**:
- **Week 1-4**: MCP Server/Client实现
- **Week 5-8**: A2A协议与Agent Card系统
- **Week 9-12**: 单体Agent封装与标准化
- **Week 13-16**: Agent注册与发现机制
### Phase 4: 本地编排与集成平台 (2-3个月)
**时间**: 2025年11月 - 2026年2月
**关键里程碑**:
- 主流框架适配器
- MCP-First集成策略
- IDE与客户端支持
**详细排期**:
- **Week 1-3**: LangChain/CrewAI/AutoGen适配器
- **Week 4-6**: Cursor/Claude Desktop集成
- **Week 7-9**: 动态发现与热加载机制
- **Week 10-12**: 本地编排工具开发
### Phase 5: 计费治理与安全平台 (4-5个月)
**时间**: 2026年2月 - 2026年7月
**关键里程碑**:
- EU计费系统
- 多租户安全隔离
- 生产环境部署
**详细排期**:
- **Week 1-4**: 执行单元(EU)计费引擎
- **Week 5-8**: Firecracker/gVisor安全隔离
- **Week 9-12**: 多租户数据与网络隔离
- **Week 13-16**: Pomerium身份认证集成
- **Week 17-20**: 监控、审计与合规系统
### Phase 6: 优化与上线 (2-3个月)
**时间**: 2026年7月 - 2026年10月
**关键里程碑**:
- 性能优化与压力测试
- 文档完善与培训
- 正式上线与运营支持
## 🔄 并行开发策略
### 可并行模块
1. **数据接入层 + 模型治理层**: 两个团队可并行开发
2. **前端界面 + 后端API**: UI/UX团队可提前开始
3. **安全隔离 + 计费系统**: 基础设施团队独立进行
4. **文档编写 + 测试用例**: 贯穿整个开发过程
### 关键依赖关系
- Phase 2 依赖 Phase 1 的API标准化
- Phase 3 依赖 Phase 2 的模型抽象层
- Phase 4 依赖 Phase 3 的Agent标准
- Phase 5 需要前四个阶段的基础支撑
## ⚠️ 风险评估与应对
### 高风险项目
1. **APILLAMA模型性能**: 可能需要额外的模型微调时间
2. **多模型兼容性**: 不同厂商API的差异化处理
3. **安全隔离复杂度**: Firecracker/gVisor的生产环境稳定性
### 应对策略
1. 提前准备备选技术方案
2. 建立每周技术评审机制
3. 关键模块预留20%缓冲时间
## 📊 资源分配建议
### 人员配置 (12-15人)
- **架构师**: 1人 (全程)
- **后端开发**: 4-5人
- **前端开发**: 2人
- **DevOps工程师**: 2人
- **测试工程师**: 2人
- **产品经理**: 1人
- **项目经理**: 1人
### 技术栈培训计划
- **Month 1**: Golang, NATS, LiteLLM基础培训
- **Month 2**: MCP协议, A2A通信深度培训
- **Month 3**: Firecracker, 容器安全培训
- **Month 4**: 监控系统, 计费引擎培训
## 🎯 成功标准
### 技术指标
- API响应时间 < 100ms (P95)
- 系统可用性 > 99.9%
- 支持1000+并发Agent
- 覆盖100+模型API
### 业务指标
- 支持主流开发框架集成
- 透明的EU计费体系
- 完整的安全隔离机制
- 企业级合规认证
---
**更新时间**: 2025年12月20日
**版本**: v1.0
**负责人**: 项目组
Docs/系统运作流程图.md
+466
View File
@@ -0,0 +1,466 @@
# taiji-AI-PAD 系统运作流程图
## 🔄 整体系统架构流程
### 核心数据流架构
```mermaid
graph TB
subgraph "用户层"
U1[开发者/企业用户]
U2[IDE: Cursor/VS Code]
U3[AI客户端: Claude Desktop]
U4[框架: LangChain/CrewAI/AutoGen]
end
subgraph "第四平面：本地编排层"
L1[MCP Client]
L2[Framework Adapters]
L3[Local Orchestrator]
end
subgraph "第三平面：Agent协议层"
A1[MCP Server]
A2[A2A Communication]
A3[Agent Registry]
A4[Agent Card System]
end
subgraph "第二平面：模型治理层"
M1[LiteLLM Gateway]
M2[Model Router]
M3[Context Manager]
M4[Cost Monitor]
end
subgraph "第一平面：数据接入层"
D1[RapidAPI Hub]
D2[APILLAMA Processor]
D3[Tool Generator]
D4[Private API Adapter]
end
subgraph "第五平面：计费治理层"
B1[EU Billing Engine]
B2[Security Isolation]
B3[Multi-tenant Manager]
B4[Audit System]
end
subgraph "外部资源"
E1[RapidAPI 16000+ APIs]
E2[OpenAI/Anthropic/等]
E3[Private APIs]
E4[Database/Storage]
end
%% 数据流连接
U1 --> L3
U2 --> L1
U3 --> L1
U4 --> L2
L1 --> A1
L2 --> A1
L3 --> A2
A1 --> M1
A2 --> A3
A3 --> M1
M1 --> M2
M2 --> E2
M3 --> M4
M1 --> D3
D1 --> D2
D2 --> D3
D4 --> D3
E1 --> D1
E3 --> D4
A1 --> B1
B1 --> B2
B2 --> B3
B3 --> B4
B4 --> E4
```
## 🚀 Agent完整生命周期流程
### 从创建到执行的端到端流程
```mermaid
sequenceDiagram
participant Dev as 开发者
participant Reg as Agent注册中心
participant MCP as MCP Server
participant Gateway as LiteLLM网关
participant Tool as 工具层
participant EU as EU计费引擎
participant Sec as 安全隔离
%% Agent创建阶段
Dev->>Reg: 1. 提交Agent定义(Role+Goal+Tools)
Reg->>Reg: 2. 验证Agent配置
Reg->>MCP: 3. 生成MCP Server实例
MCP->>Tool: 4. 绑定授权工具集
Reg->>EU: 5. 创建计费账户
%% Agent部署阶段
MCP->>Sec: 6. 申请安全容器
Sec->>Sec: 7. 创建Firecracker VM
Sec->>MCP: 8. 返回容器端点
MCP->>Reg: 9. 注册Agent服务地址
%% Agent发现与调用阶段
Dev->>Reg: 10. 查询可用Agent
Reg->>Dev: 11. 返回Agent Card列表
Dev->>MCP: 12. 通过MCP协议调用Agent
%% 执行阶段
MCP->>EU: 13. 启动计费计时器
MCP->>Gateway: 14. 请求模型推理
Gateway->>Gateway: 15. 路由到最佳模型
Gateway->>MCP: 16. 返回推理结果
MCP->>Tool: 17. 调用外部API工具
Tool->>Tool: 18. 执行API调用
Tool->>MCP: 19. 返回工具执行结果
MCP->>EU: 20. 停止计费，计算EU消耗
MCP->>Dev: 21. 返回最终结果
```
## 💡 用户使用流程详解
### 三种典型使用场景
#### 场景1: IDE集成开发流程
```mermaid
flowchart TD
A[开发者打开Cursor] --> B[配置MCP服务器端点]
B --> C[Cursor自动发现可用Agent]
C --> D[在代码中@调用Agent]
D --> E[Agent执行任务]
E --> F[返回结果到IDE]
F --> G[开发者继续编码]
subgraph "后台处理"
H[MCP协议通信]
I[模型推理]
J[工具调用]
K[EU计费]
end
E --> H
H --> I
I --> J
J --> K
```
#### 场景2: 企业级Agent编排流程
```mermaid
flowchart TD
A[业务需求分析] --> B[设计Multi-Agent架构]
B --> C[选择平台Agent]
C --> D[配置A2A通信]
D --> E[部署到生产环境]
E --> F[监控执行状态]
F --> G[成本分析优化]
subgraph "技术实现"
H[Agent Card发现]
I[任务分发]
J[结果聚合]
K[异常处理]
end
C --> H
D --> I
E --> J
F --> K
```
#### 场景3: 框架集成开发流程
```mermaid
flowchart TD
A[选择框架: LangChain/CrewAI] --> B[安装MCP适配器]
B --> C[配置平台Agent端点]
C --> D[编写业务逻辑]
D --> E[本地测试调试]
E --> F[部署到生产环境]
subgraph "适配层处理"
G[MultiServerMCPClient]
H[自动工具注入]
I[状态管理]
J[错误处理]
end
B --> G
C --> H
D --> I
E --> J
```
## 🔧 数据接入与工具化流程
### API到Agent工具的转换过程
```mermaid
flowchart TD
A[外部API] --> B{API类型判断}
B -->|RapidAPI| C[统一Key代理]
B -->|OpenAPI/Swagger| D[FastMCP解析]
B -->|私有API| E[自定义适配器]
C --> F[APILLAMA处理]
D --> F
E --> F
F --> G[结构化提取]
G --> H[Pydantic Schema生成]
H --> I[语义增强]
I --> J[MCP Tool注册]
J --> K[Agent可用工具]
subgraph "质量保障"
L[参数验证]
M[错误处理]
N[性能监控]
O[成本跟踪]
end
J --> L
L --> M
M --> N
N --> O
```
## 💰 EU计费系统运作流程
### 执行单元计费的完整链路
```mermaid
sequenceDiagram
participant User as 用户
participant Agent as Agent实例
participant Monitor as 资源监控
participant NATS as NATS消息队列
participant Billing as 计费引擎
participant Account as 账户系统
participant Dashboard as 实时仪表盘
User->>Agent: 1. 提交任务
Agent->>Monitor: 2. 申请资源配额
Monitor->>Account: 3. 检查账户余额
Account->>Agent: 4. 确认可用额度
Agent->>NATS: 5. 发送任务开始事件
NATS->>Billing: 6. 触发计费开始
Billing->>Monitor: 7. 开始资源监控
loop 任务执行期间
Monitor->>Monitor: 8. 记录CPU/内存/网络使用
Monitor->>NATS: 9. 周期性发送使用数据
NATS->>Billing: 10. 更新实时成本
Billing->>Dashboard: 11. 更新仪表盘显示
end
Agent->>NATS: 12. 发送任务完成事件
NATS->>Billing: 13. 停止计费计时
Billing->>Billing: 14. 计算最终EU消耗
Billing->>Account: 15. 扣除费用
Account->>Dashboard: 16. 更新账户余额
Dashboard->>User: 17. 显示任务成本明细
```
## 🛡️ 安全隔离与多租户流程
### 租户隔离的三层防护
```mermaid
flowchart TD
A[租户请求] --> B[身份验证]
B --> C{Pomerium网关}
C -->|认证失败| D[拒绝访问]
C -->|认证成功| E[权限检查]
E --> F{RBAC/ABAC}
F -->|无权限| D
F -->|有权限| G[计算资源分配]
G --> H[Firecracker VM创建]
H --> I[网络VPC隔离]
I --> J[数据RLS过滤]
J --> K[执行环境准备]
K --> L[Agent任务执行]
L --> M[Sidecar监控]
M --> N[审计日志记录]
N --> O[资源清理]
subgraph "三层隔离"
P[计算隔离: MicroVM]
Q[网络隔离: VPC]
R[数据隔离: RLS+加密]
end
H --> P
I --> Q
J --> R
```
## 🔄 故障恢复与高可用流程
### 系统自愈机制
```mermaid
flowchart TD
A[系统运行] --> B[健康检查]
B --> C{状态正常?}
C -->|是| A
C -->|否| D[故障检测]
D --> E{故障类型}
E -->|模型服务| F[模型切换]
E -->|Agent异常| G[容器重启]
E -->|API失效| H[降级服务]
E -->|网络异常| I[重试机制]
F --> J[LiteLLM路由切换]
G --> K[保存执行状态]
H --> L[启用备用API]
I --> M[指数退避重试]
J --> N[服务恢复]
K --> N
L --> N
M --> N
N --> O[通知运维]
O --> P[更新监控]
P --> A
```
## 📊 监控与可观测性流程
### 全链路追踪与性能监控
```mermaid
flowchart LR
A[用户请求] --> B[Trace开始]
B --> C[Agent执行]
C --> D[模型调用]
D --> E[工具执行]
E --> F[结果返回]
subgraph "监控采集"
G[Datadog Agent]
H[Prometheus Metrics]
I[LangSmith Tracing]
J[自定义Events]
end
subgraph "数据处理"
K[指标聚合]
L[异常检测]
M[性能分析]
N[成本归因]
end
subgraph "可视化展示"
O[Grafana Dashboard]
P[告警系统]
Q[成本报告]
R[性能优化建议]
end
C --> G
D --> H
E --> I
F --> J
G --> K
H --> L
I --> M
J --> N
K --> O
L --> P
M --> Q
N --> R
```
## 🚀 扩展性与未来演进
### 平台演进路径
```mermaid
flowchart TD
A[当前版本: 基础平台] --> B[v2.0: 智能调度]
B --> C[v3.0: 自治优化]
C --> D[v4.0: 生态繁荣]
subgraph "v2.0 特性"
E[元调度器]
F[成本优化AI]
G[性能预测]
end
subgraph "v3.0 特性"
H[Agent信誉系统]
I[自动化运维]
J[跨云调度]
end
subgraph "v4.0 特性"
K[Agent市场]
L[开发者生态]
M[行业标准制定]
end
B --> E
E --> F
F --> G
C --> H
H --> I
I --> J
D --> K
K --> L
L --> M
```
---
## 🎯 关键流程说明
### 1. 冷启动优化流程
- **Agent预热**: 常用Agent保持热启动状态
- **资源池管理**: 预分配计算资源，减少启动时间
- **缓存策略**: 模型响应和工具结果智能缓存
### 2. 成本控制流程
- **预算管理**: 设置租户级别的支出上限
- **实时熔断**: 超出预算自动暂停服务
- **成本优化建议**: AI驱动的资源配置优化
### 3. 安全审计流程
- **行为基线**: 建立正常行为模式
- **异常检测**: 实时监控异常访问模式
- **自动响应**: 可疑行为自动隔离和告警
### 4. 开发者体验优化
- **一键部署**: 简化Agent上线流程
- **可视化调试**: 提供Agent执行轨迹可视化
- **性能分析**: 详细的执行性能报告
---
**创建时间**: 2025年12月20日
**版本**: v1.0
**说明**: 本流程图展示了taiji-AI-PAD平台的核心运作机制，包含完整的数据流、控制流和业务流程。
Docs/项目说明.md.txt → Docs/项目说明.txt
View File
PROJECT_STATUS.md
+240
View File
@@ -0,0 +1,240 @@
# taiji-AI-PAD 项目开发状态
## 📋 项目概览
taiji-AI-PAD 是一个将AI Agents从实验性脚本演进为工业级生产力单元的全栈工程化平台。项目采用五层技术架构，通过标准化的智力资源分发与治理体系，整合异构数据，支持多模型动态切换。
## 🏗️ 当前实现状态
### ✅ 已完成
#### 1. 项目基础架构
- [x] Docker容器化环境
- [x] Docker Compose编排配置
- [x] 微服务架构设计
- [x] Nginx API网关配置
- [x] PostgreSQL数据库初始化
- [x] Redis缓存服务
- [x] NATS消息队列
#### 2. MCP Server (核心服务)
- [x] FastAPI应用框架
- [x] MCP协议处理器
- [x] Agent注册管理
- [x] 工具发现与执行
- [x] WebSocket支持
- [x] 数据库模型定义
- [x] Redis缓存集成
- [x] NATS事件发布
#### 3. 数据接入服务
- [x] FastAPI服务框架
- [x] RapidAPI集成架构
- [x] APILLAMA处理器框架
- [x] OpenAPI解析器
- [x] 工具生成器
- [x] 批处理支持
#### 4. 模型网关服务
- [x] LiteLLM代理配置
- [x] 100+模型支持配置
- [x] 路由与负载均衡
- [x] 故障转移机制
- [x] 成本跟踪
- [x] 用户权限管理
#### 5. 配置与脚本
- [x] 环境配置管理
- [x] 启动脚本 (start.sh)
- [x] 停止脚本 (stop.sh)
- [x] 测试脚本 (test.sh)
- [x] 数据库初始化脚本
#### 6. 文档
- [x] 工程排期计划
- [x] 任务拆分与分工
- [x] 系统运作流程图
- [x] API文档自动生成
### 🔄 进行中
#### 1. Agent Registry Service (Go)
- [ ] Go服务框架搭建
- [ ] Agent Card系统
- [ ] A2A通信协议
- [ ] 服务发现机制
#### 2. Billing Engine (Go)
- [ ] EU计费引擎
- [ ] 资源监控
- [ ] 成本归因分析
- [ ] 实时计费仪表盘
#### 3. 高级功能实现
- [ ] RapidAPI客户端具体实现
- [ ] APILLAMA模型集成
- [ ] 安全隔离机制(Firecracker)
- [ ] 监控与指标收集
### ⏭️ 待开始
#### 1. 框架适配器
- [ ] LangChain适配器
- [ ] CrewAI适配器
- [ ] AutoGen适配器
- [ ] IDE插件开发
#### 2. 安全与治理
- [ ] Pomerium身份认证
- [ ] 多租户数据隔离
- [ ] 审计日志系统
- [ ] SOC2/HIPAA合规
#### 3. 测试与质量保证
- [ ] 单元测试套件
- [ ] 集成测试
- [ ] 性能测试
- [ ] 安全测试
## 🚀 快速启动
```bash
# 克隆项目
git clone <repository-url>
cd taiji-AI-PAD
# 启动服务
./scripts/start.sh
# 运行测试
./scripts/test.sh
# 停止服务
./scripts/stop.sh
```
## 🌐 服务端口
| 服务 | 端口 | 描述 |
|------|------|------|
| API网关 | 80 | Nginx反向代理 |
| MCP Server | 8002 | 核心MCP协议服务 |
| 数据接入服务 | 8001 | RapidAPI与APILLAMA |
| Agent注册中心 | 8003 | Go微服务 |
| 计费引擎 | 8004 | Go微服务 |
| LiteLLM网关 | 4000 | 模型代理网关 |
| PostgreSQL | 5432 | 主数据库 |
| Redis | 6379 | 缓存服务 |
| NATS | 4222 | 消息队列 |
| Prometheus | 9090 | 指标收集 |
| Grafana | 3000 | 监控仪表板 |
## 📁 项目结构
```
taiji-AI-PAD/
├── services/ # 微服务源码
│ ├── mcp-server/ # MCP协议服务器 (Python)
│ ├── data-ingestion/ # 数据接入服务 (Python)
│ ├── model-gateway/ # 模型网关服务 (LiteLLM)
│ ├── agent-registry/ # Agent注册中心 (Go)
│ └── billing-engine/ # 计费引擎 (Go)
├── config/ # 配置文件
├── scripts/ # 管理脚本
├── Docs/ # 项目文档
├── docker-compose.yml # Docker编排文件
└── README.md # 项目说明
```
## 🛠️ 技术栈
### 后端服务
- **Python**: FastAPI, SQLAlchemy, Redis, NATS
- **Go**: Gin, GORM, 高性能微服务
- **数据库**: PostgreSQL, Redis
- **消息队列**: NATS JetStream
- **代理网关**: LiteLLM, Nginx
### AI/ML组件
- **模型管理**: LiteLLM (100+模型支持)
- **协议**: MCP (Model Context Protocol)
- **工具化**: APILLAMA技术栈
- **API集成**: RapidAPI生态
### 基础设施
- **容器化**: Docker, Docker Compose
- **监控**: Prometheus, Grafana
- **安全**: Firecracker, gVisor (计划中)
- **认证**: Pomerium (计划中)
## 🎯 核心特性
### 已实现特性
1. **MCP协议支持** - 标准化Agent通信
2. **多模型抽象** - 统一的LLM访问接口
3. **工具化治理** - API到Agent工具的自动转换
4. **容器化部署** - 一键启动完整平台
5. **监控体系** - Prometheus + Grafana
### 规划中特性
1. **EU计费模式** - 基于执行单元的透明计费
2. **A2A通信** - Agent间协作协议
3. **安全隔离** - 多租户环境支持
4. **框架集成** - 主流AI框架无缝接入
5. **自治优化** - AI驱动的成本和性能优化
## 🔧 开发指南
### 环境要求
- Docker & Docker Compose
- Python 3.11+
- Go 1.21+
- Node.js 18+ (用于前端开发)
### 开发流程
1. 修改相应服务代码
2. 使用Docker重新构建: `docker-compose build <service>`
3. 重启服务: `docker-compose restart <service>`
4. 运行测试: `./scripts/test.sh`
### API文档
- MCP Server: http://localhost:8002/docs
- 数据接入服务: http://localhost:8001/docs
## 📊 进度统计
- **总体进度**: 约60%完成
- **核心架构**: 90%完成
- **基础服务**: 75%完成
- **高级功能**: 30%完成
- **测试覆盖**: 40%完成
## 🤝 贡献指南
1. Fork项目
2. 创建功能分支: `git checkout -b feature/amazing-feature`
3. 提交变更: `git commit -m 'Add amazing feature'`
4. 推送到分支: `git push origin feature/amazing-feature`
5. 创建Pull Request
## 📝 更新日志
### v0.1.0 (2025-12-20)
- ✅ 初始项目架构搭建
- ✅ MCP Server核心实现
- ✅ 数据接入服务框架
- ✅ LiteLLM网关配置
- ✅ Docker容器化环境
- ✅ 基础监控体系
### 计划中版本
- **v0.2.0**: Agent Registry + Billing Engine
- **v0.3.0**: 安全隔离 + 权限管理
- **v0.4.0**: 框架适配器 + IDE插件
- **v1.0.0**: 生产环境就绪版本
---
**更新时间**: 2025年12月20日
**当前版本**: v0.1.0-dev
**维护状态**: 积极开发中
config/nginx.conf
+271
View File
@@ -0,0 +1,271 @@
user nginx;
worker_processes auto;
error_log /var/log/nginx/error.log notice;
pid /var/run/nginx.pid;
events {
worker_connections 1024;
use epoll;
multi_accept on;
}
http {
include /etc/nginx/mime.types;
default_type application/octet-stream;
# 日志格式
log_format main '$remote_addr - $remote_user [$time_local] "$request" '
'$status $body_bytes_sent "$http_referer" '
'"$http_user_agent" "$http_x_forwarded_for" '
'rt=$request_time ut="$upstream_response_time"';
access_log /var/log/nginx/access.log main;
# 基础配置
sendfile on;
tcp_nopush on;
tcp_nodelay on;
keepalive_timeout 65;
types_hash_max_size 2048;
client_max_body_size 50M;
# Gzip压缩
gzip on;
gzip_vary on;
gzip_min_length 1024;
gzip_proxied any;
gzip_comp_level 6;
gzip_types
text/plain
text/css
text/xml
text/javascript
application/json
application/javascript
application/xml+rss
application/atom+xml
image/svg+xml;
# 上游服务器配置
upstream mcp-server {
least_conn;
server mcp-server:8000 max_fails=3 fail_timeout=30s;
keepalive 32;
}
upstream data-ingestion {
least_conn;
server data-ingestion:8000 max_fails=3 fail_timeout=30s;
keepalive 32;
}
upstream agent-registry {
least_conn;
server agent-registry:8080 max_fails=3 fail_timeout=30s;
keepalive 32;
}
upstream billing-engine {
least_conn;
server billing-engine:8080 max_fails=3 fail_timeout=30s;
keepalive 32;
}
upstream litellm-gateway {
least_conn;
server litellm-gateway:4000 max_fails=3 fail_timeout=30s;
keepalive 32;
}
# 限流配置
limit_req_zone $binary_remote_addr zone=api:10m rate=100r/m;
limit_req_zone $binary_remote_addr zone=auth:10m rate=20r/m;
# 主服务器配置
server {
listen 80;
server_name localhost;
# 安全头
add_header X-Frame-Options DENY;
add_header X-Content-Type-Options nosniff;
add_header X-XSS-Protection "1; mode=block";
add_header Referrer-Policy "strict-origin-when-cross-origin";
# 健康检查端点
location /health {
access_log off;
return 200 "OK\n";
add_header Content-Type text/plain;
}
# MCP服务器路由
location /api/mcp/ {
limit_req zone=api burst=50 nodelay;
proxy_pass http://mcp-server/;
proxy_set_header Host $host;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Forwarded-Proto $scheme;
# WebSocket支持
proxy_http_version 1.1;
proxy_set_header Upgrade $http_upgrade;
proxy_set_header Connection "upgrade";
# 超时设置
proxy_connect_timeout 30s;
proxy_send_timeout 30s;
proxy_read_timeout 30s;
# 缓冲设置
proxy_buffering on;
proxy_buffer_size 4k;
proxy_buffers 8 4k;
}
# 数据接入服务路由
location /api/data/ {
limit_req zone=api burst=30 nodelay;
proxy_pass http://data-ingestion/;
proxy_set_header Host $host;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Forwarded-Proto $scheme;
# 长时间处理支持
proxy_connect_timeout 60s;
proxy_send_timeout 60s;
proxy_read_timeout 300s;
}
# Agent注册中心路由
location /api/agents/ {
limit_req zone=api burst=20 nodelay;
proxy_pass http://agent-registry/;
proxy_set_header Host $host;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Forwarded-Proto $scheme;
proxy_connect_timeout 30s;
proxy_send_timeout 30s;
proxy_read_timeout 30s;
}
# 计费引擎路由
location /api/billing/ {
limit_req zone=api burst=100 nodelay;
proxy_pass http://billing-engine/;
proxy_set_header Host $host;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Forwarded-Proto $scheme;
proxy_connect_timeout 30s;
proxy_send_timeout 30s;
proxy_read_timeout 30s;
}
# LiteLLM网关路由
location /api/llm/ {
limit_req zone=api burst=20 nodelay;
proxy_pass http://litellm-gateway/;
proxy_set_header Host $host;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Forwarded-Proto $scheme;
# LLM请求可能需要更长时间
proxy_connect_timeout 60s;
proxy_send_timeout 60s;
proxy_read_timeout 300s;
}
# 静态文件
location /static/ {
alias /usr/share/nginx/html/static/;
expires 1d;
add_header Cache-Control "public, immutable";
}
# API文档路由
location /docs {
proxy_pass http://mcp-server/docs;
proxy_set_header Host $host;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Forwarded-Proto $scheme;
}
# 默认路由
location / {
return 404 '{"error": "Not Found", "message": "请使用正确的API端点"}';
add_header Content-Type application/json;
}
# 错误页面
error_page 404 /404.html;
error_page 500 502 503 504 /50x.html;
location = /404.html {
return 404 '{"error": "Not Found", "message": "请求的资源不存在"}';
add_header Content-Type application/json;
}
location = /50x.html {
return 500 '{"error": "Internal Server Error", "message": "服务器内部错误"}';
add_header Content-Type application/json;
}
}
# HTTPS配置（生产环境使用）
server {
listen 443 ssl http2;
server_name localhost;
# SSL证书配置（需要实际证书文件）
# ssl_certificate /etc/nginx/ssl/cert.pem;
# ssl_certificate_key /etc/nginx/ssl/key.pem;
# SSL配置
ssl_protocols TLSv1.2 TLSv1.3;
ssl_ciphers ECDHE-RSA-AES256-GCM-SHA512:DHE-RSA-AES256-GCM-SHA512:ECDHE-RSA-AES256-GCM-SHA384;
ssl_prefer_server_ciphers off;
ssl_session_cache shared:SSL:10m;
ssl_session_timeout 10m;
# HSTS
add_header Strict-Transport-Security "max-age=63072000" always;
# 其他配置与HTTP相同...
# 临时重定向到HTTP（开发环境）
return 301 http://$server_name$request_uri;
}
# 监控和状态页面
server {
listen 8080;
server_name localhost;
location /nginx_status {
stub_status on;
access_log off;
allow 127.0.0.1;
allow 172.20.0.0/16; # Docker网络
deny all;
}
location /health {
access_log off;
return 200 "Nginx OK\n";
add_header Content-Type text/plain;
}
}
}
docker-compose.yml
+249
View File
@@ -0,0 +1,249 @@
version: '3.8'
services:
# 数据库服务
postgres:
image: postgres:15-alpine
container_name: taiji-postgres
environment:
POSTGRES_DB: taiji_db
POSTGRES_USER: taiji_user
POSTGRES_PASSWORD: taiji_pass
volumes:
- postgres_data:/var/lib/postgresql/data
- ./scripts/init.sql:/docker-entrypoint-initdb.d/init.sql
ports:
- "5432:5432"
networks:
- taiji-network
restart: unless-stopped
# Redis缓存服务
redis:
image: redis:7-alpine
container_name: taiji-redis
ports:
- "6379:6379"
volumes:
- redis_data:/data
networks:
- taiji-network
restart: unless-stopped
# NATS消息队列
nats:
image: nats:2.10-alpine
container_name: taiji-nats
ports:
- "4222:4222" # Client connections
- "6222:6222" # Routing
- "8222:8222" # Monitoring
command: ["-js", "-m", "8222"] # Enable JetStream and monitoring
volumes:
- nats_data:/data
networks:
- taiji-network
restart: unless-stopped
# LiteLLM网关服务
litellm-gateway:
build:
context: ./services/model-gateway
dockerfile: Dockerfile
container_name: taiji-litellm-gateway
ports:
- "4000:4000"
environment:
- LITELLM_MASTER_KEY=sk-taiji-master-key
- DATABASE_URL=postgresql://taiji_user:taiji_pass@postgres:5432/taiji_db
- REDIS_URL=redis://redis:6379
volumes:
- ./services/model-gateway/config:/app/config
- ./logs:/app/logs
depends_on:
- postgres
- redis
networks:
- taiji-network
restart: unless-stopped
# 数据接入服务 (Python)
data-ingestion:
build:
context: ./services/data-ingestion
dockerfile: Dockerfile
container_name: taiji-data-ingestion
ports:
- "8001:8000"
environment:
- DATABASE_URL=postgresql://taiji_user:taiji_pass@postgres:5432/taiji_db
- REDIS_URL=redis://redis:6379
- NATS_URL=nats://nats:4222
volumes:
- ./services/data-ingestion:/app
- ./logs:/app/logs
depends_on:
- postgres
- redis
- nats
networks:
- taiji-network
restart: unless-stopped
# MCP服务器 (Python)
mcp-server:
build:
context: ./services/mcp-server
dockerfile: Dockerfile
container_name: taiji-mcp-server
ports:
- "8002:8000"
environment:
- DATABASE_URL=postgresql://taiji_user:taiji_pass@postgres:5432/taiji_db
- REDIS_URL=redis://redis:6379
- NATS_URL=nats://nats:4222
- LITELLM_URL=http://litellm-gateway:4000
volumes:
- ./services/mcp-server:/app
- ./logs:/app/logs
depends_on:
- postgres
- redis
- nats
- litellm-gateway
networks:
- taiji-network
restart: unless-stopped
# Agent注册中心 (Go)
agent-registry:
build:
context: ./services/agent-registry
dockerfile: Dockerfile
container_name: taiji-agent-registry
ports:
- "8003:8080"
environment:
- DATABASE_URL=postgresql://taiji_user:taiji_pass@postgres:5432/taiji_db
- REDIS_URL=redis://redis:6379
- NATS_URL=nats://nats:4222
volumes:
- ./services/agent-registry:/app
- ./logs:/app/logs
depends_on:
- postgres
- redis
- nats
networks:
- taiji-network
restart: unless-stopped
# EU计费引擎 (Go)
billing-engine:
build:
context: ./services/billing-engine
dockerfile: Dockerfile
container_name: taiji-billing-engine
ports:
- "8004:8080"
environment:
- DATABASE_URL=postgresql://taiji_user:taiji_pass@postgres:5432/taiji_db
- REDIS_URL=redis://redis:6379
- NATS_URL=nats://nats:4222
volumes:
- ./services/billing-engine:/app
- ./logs:/app/logs
depends_on:
- postgres
- redis
- nats
networks:
- taiji-network
restart: unless-stopped
# API网关 (Nginx)
api-gateway:
image: nginx:alpine
container_name: taiji-api-gateway
ports:
- "80:80"
- "443:443"
volumes:
- ./config/nginx.conf:/etc/nginx/nginx.conf
- ./config/ssl:/etc/nginx/ssl
depends_on:
- data-ingestion
- mcp-server
- agent-registry
- billing-engine
networks:
- taiji-network
restart: unless-stopped
# 监控服务 - Prometheus
prometheus:
image: prom/prometheus:latest
container_name: taiji-prometheus
ports:
- "9090:9090"
volumes:
- ./config/prometheus.yml:/etc/prometheus/prometheus.yml
- prometheus_data:/prometheus
command:
- '--config.file=/etc/prometheus/prometheus.yml'
- '--storage.tsdb.path=/prometheus'
- '--web.console.libraries=/etc/prometheus/console_libraries'
- '--web.console.templates=/etc/prometheus/consoles'
- '--web.enable-lifecycle'
networks:
- taiji-network
restart: unless-stopped
# 监控服务 - Grafana
grafana:
image: grafana/grafana:latest
container_name: taiji-grafana
ports:
- "3000:3000"
environment:
- GF_SECURITY_ADMIN_PASSWORD=admin
volumes:
- grafana_data:/var/lib/grafana
- ./config/grafana/dashboards:/etc/grafana/provisioning/dashboards
- ./config/grafana/datasources:/etc/grafana/provisioning/datasources
depends_on:
- prometheus
networks:
- taiji-network
restart: unless-stopped
# 开发环境容器 (可选)
dev-container:
build:
context: ./dev-environment
dockerfile: Dockerfile
container_name: taiji-dev
volumes:
- .:/workspace
- /var/run/docker.sock:/var/run/docker.sock
working_dir: /workspace
tty: true
stdin_open: true
networks:
- taiji-network
profiles:
- dev
networks:
taiji-network:
driver: bridge
ipam:
config:
- subnet: 172.20.0.0/16
volumes:
postgres_data:
redis_data:
nats_data:
prometheus_data:
grafana_data:
scripts/init.sql
+106
View File
@@ -0,0 +1,106 @@
-- taiji-AI-PAD 数据库初始化脚本
-- 创建扩展
CREATE EXTENSION IF NOT EXISTS "uuid-ossp";
CREATE EXTENSION IF NOT EXISTS "pg_trgm";
-- 创建数据库（如果不存在）
-- 注意：在Docker初始化脚本中，数据库已经存在
-- 设置时区
SET timezone = 'UTC';
-- 创建一些基础索引（如果表已存在的话，模型会自动创建）
-- 这里可以添加一些额外的性能优化索引
-- 创建全文搜索配置
CREATE TEXT SEARCH CONFIGURATION IF NOT EXISTS simple_english (COPY = english);
-- 创建一些有用的函数
CREATE OR REPLACE FUNCTION update_updated_at_column()
RETURNS TRIGGER AS $$
BEGIN
NEW.updated_at = NOW();
RETURN NEW;
END;
$$ language 'plpgsql';
-- 日志表（用于审计和调试）
CREATE TABLE IF NOT EXISTS system_logs (
id UUID PRIMARY KEY DEFAULT uuid_generate_v4(),
level VARCHAR(20) NOT NULL,
service VARCHAR(50) NOT NULL,
message TEXT NOT NULL,
context JSONB,
created_at TIMESTAMP WITH TIME ZONE DEFAULT NOW()
);
CREATE INDEX IF NOT EXISTS idx_system_logs_service ON system_logs(service);
CREATE INDEX IF NOT EXISTS idx_system_logs_level ON system_logs(level);
CREATE INDEX IF NOT EXISTS idx_system_logs_created ON system_logs(created_at);
-- 配置表
CREATE TABLE IF NOT EXISTS system_config (
key VARCHAR(100) PRIMARY KEY,
value JSONB NOT NULL,
description TEXT,
created_at TIMESTAMP WITH TIME ZONE DEFAULT NOW(),
updated_at TIMESTAMP WITH TIME ZONE DEFAULT NOW()
);
-- 创建触发器
CREATE TRIGGER update_system_config_updated_at
BEFORE UPDATE ON system_config
FOR EACH ROW EXECUTE FUNCTION update_updated_at_column();
-- 插入初始配置
INSERT INTO system_config (key, value, description) VALUES
('app_version', '"1.0.0"', 'Application version'),
('maintenance_mode', 'false', 'Maintenance mode flag'),
('max_api_calls_per_minute', '1000', 'Maximum API calls per minute'),
('default_timeout', '30', 'Default timeout in seconds')
ON CONFLICT (key) DO NOTHING;
-- 性能优化设置
-- 注意：这些设置可能需要根据实际硬件调整
ALTER SYSTEM SET shared_preload_libraries = 'pg_stat_statements';
ALTER SYSTEM SET max_connections = 200;
ALTER SYSTEM SET shared_buffers = '256MB';
ALTER SYSTEM SET effective_cache_size = '1GB';
ALTER SYSTEM SET maintenance_work_mem = '64MB';
ALTER SYSTEM SET checkpoint_completion_target = 0.9;
ALTER SYSTEM SET wal_buffers = '16MB';
ALTER SYSTEM SET default_statistics_target = 100;
ALTER SYSTEM SET random_page_cost = 1.1;
ALTER SYSTEM SET effective_io_concurrency = 200;
ALTER SYSTEM SET work_mem = '4MB';
ALTER SYSTEM SET min_wal_size = '1GB';
ALTER SYSTEM SET max_wal_size = '4GB';
-- 创建监控视图
CREATE OR REPLACE VIEW system_stats AS
SELECT
schemaname,
tablename,
attname,
n_distinct,
correlation
FROM pg_stats
WHERE schemaname = 'public';
-- 创建连接监控视图
CREATE OR REPLACE VIEW connection_stats AS
SELECT
datname,
numbackends,
xact_commit,
xact_rollback,
blks_read,
blks_hit,
tup_returned,
tup_fetched,
tup_inserted,
tup_updated,
tup_deleted
FROM pg_stat_database
WHERE datname = current_database();
scripts/start.sh
Executable
+168
View File
@@ -0,0 +1,168 @@
#!/bin/bash
# taiji-AI-PAD 启动脚本
set -e
echo "🚀 启动 taiji-AI-PAD 平台..."
# 检查Docker是否运行
if ! docker info >/dev/null 2>&1; then
echo "❌ Docker 未运行，请先启动Docker"
exit 1
fi
# 检查Docker Compose是否可用
if ! command -v docker-compose >/dev/null 2>&1; then
echo "❌ Docker Compose 未找到，请安装Docker Compose"
exit 1
fi
# 创建必要的目录
echo "📁 创建必要的目录..."
mkdir -p logs
mkdir -p config/ssl
mkdir -p services/model-gateway/config
mkdir -p services/data-ingestion/models
mkdir -p services/data-ingestion/cache
# 设置环境变量（如果.env文件不存在）
if [ ! -f .env ]; then
echo "⚙️ 创建环境配置文件..."
cat > .env << EOF
# 环境设置
ENVIRONMENT=development
# 数据库设置
POSTGRES_DB=taiji_db
POSTGRES_USER=taiji_user
POSTGRES_PASSWORD=taiji_pass
DATABASE_URL=postgresql+asyncpg://taiji_user:taiji_pass@postgres:5432/taiji_db
# Redis设置
REDIS_URL=redis://redis:6379
# NATS设置
NATS_URL=nats://nats:4222
# LiteLLM设置
LITELLM_MASTER_KEY=sk-taiji-master-key
LITELLM_URL=http://litellm-gateway:4000
# RapidAPI设置（需要实际的API Key）
RAPIDAPI_KEY=your-rapidapi-key-here
RAPIDAPI_HOST=rapidapi.com
# APILLAMA模型设置
APILLAMA_MODEL_PATH=/app/models/llama-3-8b-instruct
APILLAMA_DEVICE=cpu
EOF
echo "✅ 环境配置文件已创建，请根据需要修改 .env 文件"
fi
# 检查必要的配置文件
if [ ! -f config/nginx.conf ]; then
echo "❌ Nginx配置文件未找到：config/nginx.conf"
exit 1
fi
if [ ! -f scripts/init.sql ]; then
echo "❌ 数据库初始化脚本未找到：scripts/init.sql"
exit 1
fi
# 拉取基础镜像
echo "⬇️ 拉取基础镜像..."
docker-compose pull postgres redis nats prometheus grafana nginx
# 构建服务镜像
echo "🏗️ 构建服务镜像..."
docker-compose build
# 启动基础设施服务
echo "🗄️ 启动基础设施服务..."
docker-compose up -d postgres redis nats
# 等待数据库就绪
echo "⏳ 等待数据库就绪..."
sleep 10
# 检查数据库连接
echo "🔍 检查数据库连接..."
until docker-compose exec -T postgres pg_isready -U taiji_user -d taiji_db; do
echo "等待数据库..."
sleep 2
done
# 启动应用服务
echo "🚀 启动应用服务..."
docker-compose up -d
# 等待服务启动
echo "⏳ 等待服务启动..."
sleep 15
# 检查服务状态
echo "🔍 检查服务状态..."
docker-compose ps
# 健康检查
echo "🏥 执行健康检查..."
services=("mcp-server:8002" "data-ingestion:8001" "agent-registry:8003" "billing-engine:8004")
for service in "${services[@]}"; do
service_name=$(echo $service | cut -d':' -f1)
port=$(echo $service | cut -d':' -f2)
echo "检查 $service_name..."
if curl -f -s http://localhost:$port/health > /dev/null; then
echo "✅ $service_name 健康"
else
echo "⚠️ $service_name 可能未就绪"
fi
done
# 显示访问信息
echo ""
echo "🎉 taiji-AI-PAD 启动完成！"
echo ""
echo "📊 服务访问地址："
echo " • API网关: http://localhost"
echo " • MCP服务器: http://localhost:8002"
echo " • 数据接入服务: http://localhost:8001"
echo " • Agent注册中心: http://localhost:8003"
echo " • 计费引擎: http://localhost:8004"
echo " • LiteLLM网关: http://localhost:4000"
echo ""
echo "📈 监控服务："
echo " • Grafana: http://localhost:3000 (admin/admin)"
echo " • Prometheus: http://localhost:9090"
echo ""
echo "🗄️ 数据库服务："
echo " • PostgreSQL: localhost:5432"
echo " • Redis: localhost:6379"
echo " • NATS: localhost:4222"
echo ""
echo "📚 API文档："
echo " • MCP服务器: http://localhost:8002/docs"
echo " • 数据接入服务: http://localhost:8001/docs"
echo ""
echo "🔧 管理命令："
echo " • 查看日志: docker-compose logs -f [服务名]"
echo " • 停止服务: docker-compose down"
echo " • 重启服务: docker-compose restart [服务名]"
echo ""
# 开发环境提示
if [ "$ENVIRONMENT" = "development" ]; then
echo "🔧 开发环境提示："
echo " • 代码变更会自动重载"
echo " • 日志级别设置为DEBUG"
echo " • 请确保修改.env文件中的API密钥"
echo ""
fi
echo "🎯 接下来您可以："
echo " 1. 访问 http://localhost:8002/docs 查看MCP API文档"
echo " 2. 使用 scripts/test.sh 运行测试"
echo " 3. 查看 docs/ 目录了解更多使用方法"
scripts/stop.sh
Executable
+57
View File
@@ -0,0 +1,57 @@
#!/bin/bash
# taiji-AI-PAD 停止脚本
set -e
echo "🛑 停止 taiji-AI-PAD 平台..."
# 检查Docker Compose是否可用
if ! command -v docker-compose >/dev/null 2>&1; then
echo "❌ Docker Compose 未找到"
exit 1
fi
# 显示当前运行的服务
echo "📋 当前运行的服务："
docker-compose ps
# 停止所有服务
echo "⏹️ 停止所有服务..."
docker-compose down
# 可选：清理数据卷（谨慎使用）
if [ "$1" = "--clean" ]; then
echo "🧹 清理数据卷..."
read -p "⚠️ 这将删除所有数据，是否继续？ (y/N): " -n 1 -r
echo
if [[ $REPLY =~ ^[Yy]$ ]]; then
docker-compose down -v
docker system prune -f
echo "✅ 数据卷已清理"
else
echo "❌ 已取消清理操作"
fi
fi
# 可选：清理镜像
if [ "$1" = "--clean-all" ]; then
echo "🧹 清理镜像和数据..."
read -p "⚠️ 这将删除所有镜像和数据，是否继续？ (y/N): " -n 1 -r
echo
if [[ $REPLY =~ ^[Yy]$ ]]; then
docker-compose down -v --rmi all
docker system prune -a -f
echo "✅ 镜像和数据已清理"
else
echo "❌ 已取消清理操作"
fi
fi
echo ""
echo "✅ taiji-AI-PAD 已停止"
echo ""
echo "💡 清理选项："
echo " • 清理数据卷: ./scripts/stop.sh --clean"
echo " • 清理所有数据: ./scripts/stop.sh --clean-all"
echo " • 重新启动: ./scripts/start.sh"
scripts/test.sh
Executable
+242
View File
@@ -0,0 +1,242 @@
#!/bin/bash
# taiji-AI-PAD 测试脚本
set -e
echo "🧪 开始测试 taiji-AI-PAD 平台..."
# 检查服务是否运行
check_service() {
local service_name=$1
local url=$2
local expected_status=${3:-200}
echo "🔍 检查 $service_name..."
if curl -s -o /dev/null -w "%{http_code}" "$url" | grep -q "$expected_status"; then
echo "✅ $service_name 正常运行"
return 0
else
echo "❌ $service_name 无响应"
return 1
fi
}
# 测试API端点
test_api_endpoint() {
local name=$1
local url=$2
local method=${3:-GET}
local data=${4:-""}
echo "🧪 测试 $name..."
if [ -n "$data" ]; then
response=$(curl -s -X "$method" -H "Content-Type: application/json" -d "$data" "$url" 2>/dev/null || echo "ERROR")
else
response=$(curl -s -X "$method" "$url" 2>/dev/null || echo "ERROR")
fi
if [ "$response" = "ERROR" ]; then
echo "❌ $name 测试失败"
return 1
else
echo "✅ $name 测试通过"
if command -v jq >/dev/null 2>&1; then
echo " 响应: $(echo "$response" | jq -c . 2>/dev/null || echo "$response")"
else
echo " 响应: $response"
fi
return 0
fi
}
# 等待服务启动
wait_for_services() {
echo "⏳ 等待服务启动..."
sleep 5
local max_attempts=30
local attempt=1
while [ $attempt -le $max_attempts ]; do
if curl -s http://localhost:8002/health > /dev/null 2>&1; then
echo "✅ 服务已就绪"
break
fi
echo "等待中... ($attempt/$max_attempts)"
sleep 2
((attempt++))
done
if [ $attempt -gt $max_attempts ]; then
echo "❌ 服务启动超时"
exit 1
fi
}
# 主测试流程
main() {
echo "🚀 taiji-AI-PAD 平台测试"
echo "========================"
# 等待服务启动
wait_for_services
# 基础健康检查
echo ""
echo "📋 基础健康检查"
echo "----------------"
local services=(
"MCP服务器:http://localhost:8002/health"
"数据接入服务:http://localhost:8001/health"
"API网关:http://localhost/health"
)
local failed_services=0
for service_info in "${services[@]}"; do
IFS=':' read -r name url <<< "$service_info"
if ! check_service "$name" "$url"; then
((failed_services++))
fi
done
# API功能测试
echo ""
echo "🔧 API功能测试"
echo "---------------"
local api_tests=(
"MCP服务器健康检查:http://localhost:8002/health:GET"
"数据接入服务健康检查:http://localhost:8001/health:GET"
"MCP工具列表:http://localhost:8002/tools:GET"
"数据接入统计:http://localhost:8001/stats:GET"
)
local failed_tests=0
for test_info in "${api_tests[@]}"; do
IFS=':' read -r name url method <<< "$test_info"
if ! test_api_endpoint "$name" "$url" "$method"; then
((failed_tests++))
fi
done
# Agent创建测试
echo ""
echo "🤖 Agent创建测试"
echo "----------------"
local agent_data='{
"name": "test-agent",
"description": "测试Agent",
"role": "测试助手",
"goal": "执行测试任务",
"tools": ["web_search"],
"config": {}
}'
if test_api_endpoint "创建Agent" "http://localhost:8002/agents" "POST" "$agent_data"; then
echo "🎉 Agent创建测试通过"
else
echo "❌ Agent创建测试失败"
((failed_tests++))
fi
# MCP协议测试
echo ""
echo "🔗 MCP协议测试"
echo "--------------"
local mcp_request='{
"jsonrpc": "2.0",
"id": "test-1",
"method": "tools/list",
"params": {}
}'
if test_api_endpoint "MCP工具列表" "http://localhost:8002/agents/test-agent/execute" "POST" "$mcp_request"; then
echo "🎉 MCP协议测试通过"
else
echo "❌ MCP协议测试失败"
((failed_tests++))
fi
# 性能测试
echo ""
echo "⚡ 简单性能测试"
echo "---------------"
echo "🔄 并发请求测试..."
local start_time=$(date +%s%N)
for i in {1..10}; do
curl -s http://localhost:8002/health > /dev/null &
done
wait
local end_time=$(date +%s%N)
local duration=$((($end_time - $start_time) / 1000000))
echo "✅ 10个并发请求耗时: ${duration}ms"
# 负载测试（如果安装了ab）
if command -v ab >/dev/null 2>&1; then
echo "🚀 负载测试 (100个请求，并发10)..."
ab -n 100 -c 10 -q http://localhost:8002/health | grep -E "(Requests per second|Time per request)"
else
echo "💡 提示: 安装 apache2-utils 可进行更详细的性能测试"
fi
# 测试报告
echo ""
echo "📊 测试报告"
echo "==========="
local total_services=${#services[@]}
local total_tests=$((${#api_tests[@]} + 2)) # API测试 + Agent创建 + MCP协议
echo "服务检查: $((total_services - failed_services))/$total_services 通过"
echo "功能测试: $((total_tests - failed_tests))/$total_tests 通过"
if [ $failed_services -eq 0 ] && [ $failed_tests -eq 0 ]; then
echo ""
echo "🎉 所有测试通过！taiji-AI-PAD 运行正常"
echo ""
echo "🔗 快速访问链接："
echo " • MCP API文档: http://localhost:8002/docs"
echo " • 数据接入API: http://localhost:8001/docs"
echo " • Grafana监控: http://localhost:3000"
echo ""
return 0
else
echo ""
echo "❌ 部分测试失败，请检查服务状态"
echo ""
echo "🔧 故障排除："
echo " • 查看日志: docker-compose logs"
echo " • 检查服务状态: docker-compose ps"
echo " • 重启服务: docker-compose restart"
echo ""
return 1
fi
}
# 清理函数
cleanup() {
echo ""
echo "🧹 测试清理..."
# 删除测试创建的Agent（如果存在）
curl -s -X DELETE http://localhost:8002/agents/test-agent > /dev/null 2>&1 || true
echo "✅ 清理完成"
}
# 设置清理陷阱
trap cleanup EXIT
# 运行测试
main "$@"
services/data-ingestion/Dockerfile
+39
View File
@@ -0,0 +1,39 @@
FROM python:3.11-slim
# 设置工作目录
WORKDIR /app
# 安装系统依赖
RUN apt-get update && apt-get install -y \
gcc \
g++ \
make \
curl \
git \
&& rm -rf /var/lib/apt/lists/*
# 复制requirements文件
COPY requirements.txt .
# 安装Python依赖
RUN pip install --no-cache-dir -r requirements.txt
# 复制源代码
COPY . .
# 创建必要目录
RUN mkdir -p logs models cache
# 设置环境变量
ENV PYTHONPATH=/app
ENV PYTHONUNBUFFERED=1
# 暴露端口
EXPOSE 8000
# 健康检查
HEALTHCHECK --interval=30s --timeout=30s --start-period=5s --retries=3 \
CMD curl -f http://localhost:8000/health || exit 1
# 启动应用
CMD ["python", "-m", "uvicorn", "main:app", "--host", "0.0.0.0", "--port", "8000", "--reload"]
services/data-ingestion/config.py
+146
View File
@@ -0,0 +1,146 @@
"""
数据接入服务配置管理
"""
import os
from typing import List, Optional
from pydantic_settings import BaseSettings
class Settings(BaseSettings):
"""数据接入服务配置"""
# 应用设置
app_name: str = "taiji-AI-PAD 数据接入服务"
debug: bool = False
# Redis设置
redis_url: str = os.getenv("REDIS_URL", "redis://redis:6379")
redis_max_connections: int = 20
# NATS设置
nats_url: str = os.getenv("NATS_URL", "nats://nats:4222")
# RapidAPI设置
rapidapi_key: str = os.getenv("RAPIDAPI_KEY", "")
rapidapi_host: str = os.getenv("RAPIDAPI_HOST", "rapidapi.com")
rapidapi_base_url: str = "https://rapidapi.com"
rapidapi_timeout: int = 30
rapidapi_rate_limit: int = 1000 # 每分钟请求数
# APILLAMA模型设置
apillama_model_path: str = os.getenv(
"APILLAMA_MODEL_PATH",
"/app/models/llama-3-8b-instruct"
)
apillama_device: str = os.getenv("APILLAMA_DEVICE", "cpu")
apillama_max_length: int = 2048
apillama_temperature: float = 0.3
apillama_top_p: float = 0.9
# 缓存设置
cache_dir: str = "/app/cache"
cache_ttl: int = 3600 # 秒
max_cache_size: int = 1000 # MB
# OpenAPI解析设置
openapi_timeout: int = 60
openapi_max_size: int = 10 * 1024 * 1024 # 10MB
supported_openapi_versions: List[str] = ["2.0", "3.0", "3.1"]
# 工具生成设置
max_tools_per_api: int = 50
tool_name_max_length: int = 100
tool_description_max_length: int = 500
# API处理设置
max_concurrent_requests: int = 10
request_timeout: int = 30
retry_attempts: int = 3
retry_delay: float = 1.0
# 安全设置
allowed_domains: List[str] = [
"rapidapi.com",
"github.com",
"swagger.io",
"openapis.org"
]
blocked_domains: List[str] = []
# 监控设置
enable_metrics: bool = True
metrics_port: int = 8001
# 日志设置
log_level: str = "INFO"
log_format: str = "json"
log_file: Optional[str] = "/app/logs/data-ingestion.log"
# 并发设置
max_workers: int = 4
max_queue_size: int = 1000
# API限制设置
max_endpoints_per_spec: int = 200
max_parameters_per_endpoint: int = 50
max_response_schemas: int = 100
# 文件处理设置
temp_dir: str = "/tmp/taiji-data-ingestion"
max_file_size: int = 50 * 1024 * 1024 # 50MB
allowed_file_types: List[str] = [
"application/json",
"text/yaml",
"text/plain",
"application/yaml"
]
# 数据库设置（如果需要持久化）
database_url: Optional[str] = os.getenv("DATABASE_URL")
class Config:
env_file = ".env"
env_file_encoding = "utf-8"
case_sensitive = False
class DevelopmentSettings(Settings):
"""开发环境配置"""
debug: bool = True
log_level: str = "DEBUG"
apillama_device: str = "cpu"
cache_ttl: int = 600 # 10分钟
class ProductionSettings(Settings):
"""生产环境配置"""
debug: bool = False
log_level: str = "INFO"
apillama_device: str = "cuda" # 如果有GPU
max_concurrent_requests: int = 50
max_workers: int = 8
class TestingSettings(Settings):
"""测试环境配置"""
debug: bool = True
redis_url: str = "redis://localhost:6379/1" # 使用不同的数据库
cache_ttl: int = 60 # 1分钟
rapidapi_key: str = "test-key"
def get_settings() -> Settings:
"""根据环境变量获取相应的配置"""
environment = os.getenv("ENVIRONMENT", "development").lower()
if environment == "production":
return ProductionSettings()
elif environment == "testing":
return TestingSettings()
else:
return DevelopmentSettings()
# 全局配置实例
settings = get_settings()
services/data-ingestion/main.py
+547
View File
@@ -0,0 +1,547 @@
"""
taiji-AI-PAD 数据接入服务
负责全域数据接入与工具化治理，包括RapidAPI集成和APILLAMA技术实现
"""
import asyncio
import json
import logging
import os
from datetime import datetime
from typing import Any, Dict, List, Optional, Union
import structlog
from fastapi import FastAPI, HTTPException, BackgroundTasks, Depends
from fastapi.middleware.cors import CORSMiddleware
from fastapi.responses import JSONResponse
from pydantic import BaseModel
import redis.asyncio as redis
import nats
import httpx
from .config import Settings
from .schemas import (
APIEndpoint, ToolDefinition,
RapidAPIRequest, APIParsedResponse,
APILLAMARequest, APILLAMAResponse
)
from .rapidapi_client import RapidAPIClient
from .apillama_processor import APILLAMAProcessor
from .openapi_parser import OpenAPIParser
from .tool_generator import ToolGenerator
# 配置日志
structlog.configure(
processors=[
structlog.stdlib.filter_by_level,
structlog.stdlib.add_logger_name,
structlog.stdlib.add_log_level,
structlog.stdlib.PositionalArgumentsFormatter(),
structlog.processors.TimeStamper(fmt="iso"),
structlog.processors.StackInfoRenderer(),
structlog.processors.format_exc_info,
structlog.processors.UnicodeDecoder(),
structlog.processors.JSONRenderer()
],
context_class=dict,
logger_factory=structlog.stdlib.LoggerFactory(),
cache_logger_on_first_use=True,
)
logger = structlog.get_logger()
# 应用设置
settings = Settings()
app = FastAPI(
title="taiji-AI-PAD 数据接入服务",
description="全域数据接入与工具化治理服务，支持RapidAPI集成和APILLAMA技术",
version="1.0.0",
docs_url="/docs",
redoc_url="/redoc"
)
# CORS配置
app.add_middleware(
CORSMiddleware,
allow_origins=["*"],
allow_credentials=True,
allow_methods=["*"],
allow_headers=["*"],
)
# 全局变量
redis_client: Optional[redis.Redis] = None
nats_client: Optional[nats.NATS] = None
rapidapi_client: Optional[RapidAPIClient] = None
apillama_processor: Optional[APILLAMAProcessor] = None
openapi_parser: Optional[OpenAPIParser] = None
tool_generator: Optional[ToolGenerator] = None
class HealthResponse(BaseModel):
status: str
timestamp: str
services: Dict[str, str]
stats: Dict[str, int]
@app.on_event("startup")
async def startup_event():
"""应用启动初始化"""
global redis_client, nats_client, rapidapi_client
global apillama_processor, openapi_parser, tool_generator
try:
# 连接Redis
redis_client = redis.from_url(
settings.redis_url,
encoding="utf-8",
decode_responses=True
)
await redis_client.ping()
logger.info("Redis连接成功")
# 连接NATS
nats_client = await nats.connect(settings.nats_url)
logger.info("NATS连接成功")
# 初始化RapidAPI客户端
rapidapi_client = RapidAPIClient(
api_key=settings.rapidapi_key,
host=settings.rapidapi_host,
redis_client=redis_client
)
logger.info("RapidAPI客户端初始化完成")
# 初始化APILLAMA处理器
apillama_processor = APILLAMAProcessor(
model_path=settings.apillama_model_path,
cache_dir=settings.cache_dir,
redis_client=redis_client
)
await apillama_processor.initialize()
logger.info("APILLAMA处理器初始化完成")
# 初始化OpenAPI解析器
openapi_parser = OpenAPIParser(
cache_dir=settings.cache_dir,
redis_client=redis_client
)
logger.info("OpenAPI解析器初始化完成")
# 初始化工具生成器
tool_generator = ToolGenerator(
redis_client=redis_client,
nats_client=nats_client,
apillama_processor=apillama_processor
)
logger.info("工具生成器初始化完成")
# 启动后台任务
asyncio.create_task(background_api_sync())
logger.info("数据接入服务启动完成")
except Exception as e:
logger.error(f"服务启动失败: {e}")
raise
@app.on_event("shutdown")
async def shutdown_event():
"""应用关闭清理"""
global redis_client, nats_client, apillama_processor
try:
# 关闭NATS连接
if nats_client:
await nats_client.close()
# 关闭Redis连接
if redis_client:
await redis_client.close()
# 清理APILLAMA处理器
if apillama_processor:
await apillama_processor.cleanup()
logger.info("资源清理完成")
except Exception as e:
logger.error(f"资源清理失败: {e}")
@app.get("/health", response_model=HealthResponse)
async def health_check():
"""健康检查端点"""
services = {
"data_ingestion": "healthy",
"redis": "unknown",
"nats": "unknown",
"rapidapi": "unknown",
"apillama": "unknown"
}
stats = {
"total_apis": 0,
"processed_apis": 0,
"generated_tools": 0,
"cache_size": 0
}
try:
# 检查Redis
if redis_client:
await redis_client.ping()
services["redis"] = "healthy"
# 获取统计信息
stats["cache_size"] = await redis_client.dbsize()
stats["total_apis"] = await redis_client.scard("rapidapi:endpoints") or 0
stats["processed_apis"] = await redis_client.scard("processed:apis") or 0
stats["generated_tools"] = await redis_client.scard("tools:registry") or 0
except Exception:
services["redis"] = "unhealthy"
try:
# 检查NATS
if nats_client and nats_client.is_connected:
services["nats"] = "healthy"
except Exception:
services["nats"] = "unhealthy"
try:
# 检查RapidAPI
if rapidapi_client:
await rapidapi_client.test_connection()
services["rapidapi"] = "healthy"
except Exception:
services["rapidapi"] = "unhealthy"
try:
# 检查APILLAMA
if apillama_processor and apillama_processor.is_ready():
services["apillama"] = "healthy"
except Exception:
services["apillama"] = "unhealthy"
return HealthResponse(
status="healthy" if all(s == "healthy" for s in services.values()) else "degraded",
timestamp=datetime.utcnow().isoformat(),
services=services,
stats=stats
)
@app.post("/rapidapi/sync")
async def sync_rapidapi_endpoints(
background_tasks: BackgroundTasks,
category: Optional[str] = None,
limit: int = 100
):
"""同步RapidAPI端点"""
try:
if not rapidapi_client:
raise HTTPException(status_code=500, detail="RapidAPI客户端未初始化")
# 启动后台同步任务
background_tasks.add_task(
rapidapi_client.sync_endpoints,
category=category,
limit=limit
)
return {
"message": "RapidAPI端点同步已启动",
"category": category,
"limit": limit
}
except Exception as e:
logger.error(f"同步RapidAPI端点失败: {e}")
raise HTTPException(status_code=500, detail=str(e))
@app.post("/rapidapi/test")
async def test_rapidapi_endpoint(request: RapidAPIRequest):
"""测试RapidAPI端点"""
try:
if not rapidapi_client:
raise HTTPException(status_code=500, detail="RapidAPI客户端未初始化")
result = await rapidapi_client.test_endpoint(
endpoint=request.endpoint,
method=request.method,
params=request.params,
headers=request.headers
)
return result
except Exception as e:
logger.error(f"测试RapidAPI端点失败: {e}")
raise HTTPException(status_code=500, detail=str(e))
@app.post("/openapi/parse", response_model=APIParsedResponse)
async def parse_openapi_spec(
url: str,
background_tasks: BackgroundTasks
):
"""解析OpenAPI规范文档"""
try:
if not openapi_parser:
raise HTTPException(status_code=500, detail="OpenAPI解析器未初始化")
# 解析OpenAPI文档
parsed_result = await openapi_parser.parse_spec(url)
# 启动后台工具生成任务
background_tasks.add_task(
generate_tools_from_spec,
parsed_result
)
return APIParsedResponse(
url=url,
title=parsed_result.get("info", {}).get("title", ""),
version=parsed_result.get("info", {}).get("version", ""),
endpoints_count=len(parsed_result.get("paths", {})),
schemas_count=len(parsed_result.get("components", {}).get("schemas", {})),
parsed_data=parsed_result
)
except Exception as e:
logger.error(f"解析OpenAPI规范失败: {e}")
raise HTTPException(status_code=500, detail=str(e))
@app.post("/apillama/process", response_model=APILLAMAResponse)
async def process_api_with_apillama(request: APILLAMARequest):
"""使用APILLAMA处理API文档"""
try:
if not apillama_processor:
raise HTTPException(status_code=500, detail="APILLAMA处理器未初始化")
result = await apillama_processor.process_api_doc(
api_doc=request.api_doc,
context=request.context,
output_format=request.output_format
)
return APILLAMAResponse(
processed=True,
output_format=request.output_format,
schema=result.get("schema"),
description=result.get("description"),
parameters=result.get("parameters", []),
examples=result.get("examples", []),
processing_time=result.get("processing_time", 0)
)
except Exception as e:
logger.error(f"APILLAMA处理失败: {e}")
raise HTTPException(status_code=500, detail=str(e))
@app.post("/tools/generate")
async def generate_tool_from_endpoint(
endpoint: APIEndpoint,
background_tasks: BackgroundTasks
):
"""从API端点生成工具定义"""
try:
if not tool_generator:
raise HTTPException(status_code=500, detail="工具生成器未初始化")
# 启动后台工具生成任务
background_tasks.add_task(
tool_generator.generate_tool,
endpoint
)
return {
"message": "工具生成任务已启动",
"endpoint": endpoint.url,
"method": endpoint.method
}
except Exception as e:
logger.error(f"生成工具失败: {e}")
raise HTTPException(status_code=500, detail=str(e))
@app.get("/tools", response_model=List[ToolDefinition])
async def list_generated_tools(
category: Optional[str] = None,
limit: int = 100,
offset: int = 0
):
"""获取生成的工具列表"""
try:
if not redis_client:
raise HTTPException(status_code=500, detail="Redis客户端未初始化")
tools = []
tool_keys = await redis_client.smembers("tools:registry")
for tool_key in list(tool_keys)[offset:offset+limit]:
tool_data = await redis_client.get(f"tool:{tool_key}")
if tool_data:
tool = json.loads(tool_data)
if not category or tool.get("category") == category:
tools.append(ToolDefinition(**tool))
return tools
except Exception as e:
logger.error(f"获取工具列表失败: {e}")
raise HTTPException(status_code=500, detail=str(e))
@app.get("/tools/{tool_name}", response_model=ToolDefinition)
async def get_tool_definition(tool_name: str):
"""获取特定工具定义"""
try:
if not redis_client:
raise HTTPException(status_code=500, detail="Redis客户端未初始化")
tool_data = await redis_client.get(f"tool:{tool_name}")
if not tool_data:
raise HTTPException(status_code=404, detail="工具不存在")
tool = json.loads(tool_data)
return ToolDefinition(**tool)
except HTTPException:
raise
except Exception as e:
logger.error(f"获取工具定义失败: {e}")
raise HTTPException(status_code=500, detail=str(e))
@app.delete("/tools/{tool_name}")
async def delete_tool(tool_name: str):
"""删除工具定义"""
try:
if not redis_client:
raise HTTPException(status_code=500, detail="Redis客户端未初始化")
# 删除工具数据
deleted = await redis_client.delete(f"tool:{tool_name}")
if not deleted:
raise HTTPException(status_code=404, detail="工具不存在")
# 从注册表中移除
await redis_client.srem("tools:registry", tool_name)
return {"message": f"工具 {tool_name} 已删除"}
except HTTPException:
raise
except Exception as e:
logger.error(f"删除工具失败: {e}")
raise HTTPException(status_code=500, detail=str(e))
@app.get("/stats")
async def get_statistics():
"""获取统计信息"""
try:
if not redis_client:
raise HTTPException(status_code=500, detail="Redis客户端未初始化")
stats = {
"total_apis": await redis_client.scard("rapidapi:endpoints") or 0,
"processed_apis": await redis_client.scard("processed:apis") or 0,
"generated_tools": await redis_client.scard("tools:registry") or 0,
"failed_processes": await redis_client.scard("failed:processes") or 0,
"cache_size": await redis_client.dbsize(),
"last_sync": await redis_client.get("last_sync_time") or "从未同步"
}
# 获取分类统计
categories = {}
tool_keys = await redis_client.smembers("tools:registry")
for tool_key in tool_keys:
tool_data = await redis_client.get(f"tool:{tool_key}")
if tool_data:
tool = json.loads(tool_data)
category = tool.get("category", "unknown")
categories[category] = categories.get(category, 0) + 1
stats["categories"] = categories
return stats
except Exception as e:
logger.error(f"获取统计信息失败: {e}")
raise HTTPException(status_code=500, detail=str(e))
@app.post("/cache/clear")
async def clear_cache():
"""清理缓存"""
try:
if not redis_client:
raise HTTPException(status_code=500, detail="Redis客户端未初始化")
# 清理处理缓存
await redis_client.delete("processed:apis")
await redis_client.delete("failed:processes")
# 清理工具缓存（保留工具注册表）
tool_keys = await redis_client.smembers("tools:registry")
if tool_keys:
cache_keys = [f"tool_cache:{key}" for key in tool_keys]
await redis_client.delete(*cache_keys)
return {"message": "缓存已清理"}
except Exception as e:
logger.error(f"清理缓存失败: {e}")
raise HTTPException(status_code=500, detail=str(e))
async def generate_tools_from_spec(parsed_spec: Dict[str, Any]):
"""从解析的OpenAPI规范生成工具"""
try:
if not tool_generator:
logger.error("工具生成器未初始化")
return
paths = parsed_spec.get("paths", {})
for path, methods in paths.items():
for method, spec in methods.items():
if method.upper() in ["GET", "POST", "PUT", "DELETE", "PATCH"]:
endpoint = APIEndpoint(
url=f"{parsed_spec.get('servers', [{}])[0].get('url', '')}{path}",
method=method.upper(),
name=spec.get("operationId", f"{method}_{path}".replace("/", "_")),
description=spec.get("summary", spec.get("description", "")),
parameters=spec.get("parameters", []),
request_body=spec.get("requestBody"),
responses=spec.get("responses", {})
)
await tool_generator.generate_tool(endpoint)
logger.info(f"从OpenAPI规范生成了 {len(paths)} 个工具")
except Exception as e:
logger.error(f"从规范生成工具失败: {e}")
async def background_api_sync():
"""后台API同步任务"""
while True:
try:
await asyncio.sleep(3600) # 每小时同步一次
if rapidapi_client:
await rapidapi_client.sync_popular_apis()
logger.info("后台API同步完成")
except Exception as e:
logger.error(f"后台API同步失败: {e}")
@app.get("/metrics")
async def get_metrics():
"""Prometheus metrics端点"""
# TODO: 实现Prometheus metrics
return JSONResponse({"message": "Metrics endpoint - TODO: implement"})
if __name__ == "__main__":
import uvicorn
uvicorn.run(
"main:app",
host="0.0.0.0",
port=8000,
reload=True,
log_level="info"
)
services/data-ingestion/requirements.txt
+83
View File
@@ -0,0 +1,83 @@
# Web框架
fastapi==0.104.1
uvicorn[standard]==0.24.0
pydantic==2.5.0
pydantic-settings==2.1.0
# 数据库
sqlalchemy==2.0.23
asyncpg==0.29.0
# Redis和缓存
redis==5.0.1
aioredis==2.0.1
# NATS消息队列
nats-py==2.6.0
# HTTP客户端
httpx==0.25.2
aiohttp==3.9.1
requests==2.31.0
# RapidAPI集成
rapidapi-python==1.2.0
# APILLAMA相关依赖
transformers==4.36.0
torch==2.1.0
tokenizers==0.15.0
accelerate==0.24.0
# OpenAPI处理
openapi-spec-validator==0.7.1
openapi-parser==1.1.0
pydantic-openapi==1.4.0
apispec==6.3.0
# JSON处理和验证
jsonschema==4.20.0
json-repair==0.7.0
# 文本处理
nltk==3.8.1
spacy==3.7.0
beautifulsoup4==4.12.2
# 机器学习工具
scikit-learn==1.3.2
numpy==1.24.3
pandas==2.1.4
# 异步处理
asyncio-throttle==1.0.2
aiofiles==23.2.1
# 配置管理
python-dotenv==1.0.0
pyyaml==6.0.1
# 监控和日志
prometheus-client==0.19.0
structlog==23.2.0
rich==13.7.0
# 缓存和存储
diskcache==5.6.3
joblib==1.3.2
# 工具和实用程序
python-multipart==0.0.6
email-validator==2.1.0
validators==0.22.0
# 开发和测试工具
pytest==7.4.3
pytest-asyncio==0.21.1
black==23.11.0
flake8==6.1.0
mypy==1.7.1
# API文档生成
swagger-ui-bundle==0.0.9
redoc==2.0.0
services/data-ingestion/schemas.py
+316
View File
@@ -0,0 +1,316 @@
"""
数据接入服务的Pydantic schemas
"""
from datetime import datetime
from typing import Any, Dict, List, Optional, Union
from pydantic import BaseModel, Field, validator
import uuid
class BaseSchema(BaseModel):
"""基础schema类"""
class Config:
from_attributes = True
json_encoders = {
datetime: lambda v: v.isoformat(),
uuid.UUID: lambda v: str(v),
}
# ========== API端点相关 ==========
class APIEndpoint(BaseSchema):
"""API端点定义"""
url: str = Field(..., description="API端点URL")
method: str = Field(..., description="HTTP方法")
name: Optional[str] = Field(None, description="端点名称")
description: Optional[str] = Field(None, description="端点描述")
# OpenAPI相关字段
parameters: List[Dict[str, Any]] = Field(default=[], description="参数定义")
request_body: Optional[Dict[str, Any]] = Field(None, description="请求体定义")
responses: Dict[str, Any] = Field(default={}, description="响应定义")
# 认证和安全
security: List[Dict[str, Any]] = Field(default=[], description="安全要求")
# 元数据
tags: List[str] = Field(default=[], description="标签")
deprecated: bool = Field(False, description="是否已弃用")
@validator('method')
def validate_method(cls, v):
"""验证HTTP方法"""
allowed_methods = ["GET", "POST", "PUT", "DELETE", "PATCH", "HEAD", "OPTIONS"]
if v.upper() not in allowed_methods:
raise ValueError(f'不支持的HTTP方法: {v}')
return v.upper()
class APIParameter(BaseSchema):
"""API参数定义"""
name: str = Field(..., description="参数名称")
type: str = Field(..., description="参数类型")
location: str = Field(..., description="参数位置 (query, path, header, body)")
description: Optional[str] = Field(None, description="参数描述")
required: bool = Field(False, description="是否必需")
default: Optional[Any] = Field(None, description="默认值")
example: Optional[Any] = Field(None, description="示例值")
enum: Optional[List[Any]] = Field(None, description="枚举值")
format: Optional[str] = Field(None, description="格式")
pattern: Optional[str] = Field(None, description="正则模式")
minimum: Optional[float] = Field(None, description="最小值")
maximum: Optional[float] = Field(None, description="最大值")
# ========== RapidAPI相关 ==========
class RapidAPIRequest(BaseSchema):
"""RapidAPI请求"""
endpoint: str = Field(..., description="API端点")
method: str = Field(..., description="HTTP方法")
params: Optional[Dict[str, Any]] = Field(None, description="请求参数")
headers: Optional[Dict[str, str]] = Field(None, description="请求头")
timeout: Optional[int] = Field(30, description="超时时间(秒)")
class RapidAPIResponse(BaseSchema):
"""RapidAPI响应"""
success: bool = Field(..., description="是否成功")
status_code: int = Field(..., description="HTTP状态码")
data: Optional[Any] = Field(None, description="响应数据")
error: Optional[str] = Field(None, description="错误信息")
response_time: float = Field(..., description="响应时间(毫秒)")
headers: Dict[str, str] = Field(default={}, description="响应头")
class RapidAPIEndpointInfo(BaseSchema):
"""RapidAPI端点信息"""
id: str = Field(..., description="端点ID")
name: str = Field(..., description="端点名称")
url: str = Field(..., description="端点URL")
method: str = Field(..., description="HTTP方法")
description: Optional[str] = Field(None, description="描述")
category: Optional[str] = Field(None, description="分类")
provider: Optional[str] = Field(None, description="提供商")
pricing: Optional[Dict[str, Any]] = Field(None, description="定价信息")
rate_limit: Optional[Dict[str, int]] = Field(None, description="速率限制")
popularity_score: Optional[float] = Field(None, description="流行度分数")
# ========== APILLAMA相关 ==========
class APILLAMARequest(BaseSchema):
"""APILLAMA处理请求"""
api_doc: str = Field(..., description="API文档内容")
context: Optional[Dict[str, Any]] = Field(None, description="上下文信息")
output_format: str = Field("pydantic", description="输出格式 (pydantic, json_schema, openapi)")
# 处理选项
include_examples: bool = Field(True, description="是否包含示例")
enhance_descriptions: bool = Field(True, description="是否增强描述")
validate_schema: bool = Field(True, description="是否验证schema")
@validator('output_format')
def validate_output_format(cls, v):
"""验证输出格式"""
allowed_formats = ["pydantic", "json_schema", "openapi"]
if v not in allowed_formats:
raise ValueError(f'不支持的输出格式: {v}')
return v
class APILLAMAResponse(BaseSchema):
"""APILLAMA处理响应"""
processed: bool = Field(..., description="是否处理成功")
output_format: str = Field(..., description="输出格式")
schema: Optional[Dict[str, Any]] = Field(None, description="生成的schema")
description: Optional[str] = Field(None, description="增强的描述")
parameters: List[APIParameter] = Field(default=[], description="参数定义")
examples: List[Dict[str, Any]] = Field(default=[], description="示例数据")
processing_time: float = Field(..., description="处理时间(秒)")
error: Optional[str] = Field(None, description="错误信息")
# 质量评估
confidence_score: Optional[float] = Field(None, description="置信度分数")
completeness_score: Optional[float] = Field(None, description="完整性分数")
# ========== OpenAPI解析相关 ==========
class OpenAPISpec(BaseSchema):
"""OpenAPI规范"""
openapi: Optional[str] = Field(None, description="OpenAPI版本")
swagger: Optional[str] = Field(None, description="Swagger版本")
info: Dict[str, Any] = Field(..., description="API信息")
servers: List[Dict[str, Any]] = Field(default=[], description="服务器列表")
paths: Dict[str, Any] = Field(default={}, description="路径定义")
components: Optional[Dict[str, Any]] = Field(None, description="组件定义")
security: Optional[List[Dict[str, Any]]] = Field(None, description="安全定义")
tags: List[Dict[str, Any]] = Field(default=[], description="标签定义")
class APIParsedResponse(BaseSchema):
"""API解析响应"""
url: str = Field(..., description="原始URL")
title: str = Field(..., description="API标题")
version: str = Field(..., description="API版本")
endpoints_count: int = Field(..., description="端点数量")
schemas_count: int = Field(..., description="模式数量")
parsed_data: OpenAPISpec = Field(..., description="解析后的数据")
parsing_time: Optional[float] = Field(None, description="解析时间(秒)")
errors: List[str] = Field(default=[], description="解析错误")
warnings: List[str] = Field(default=[], description="解析警告")
# ========== 工具生成相关 ==========
class ToolDefinition(BaseSchema):
"""工具定义"""
name: str = Field(..., description="工具名称")
description: str = Field(..., description="工具描述")
category: str = Field(..., description="工具分类")
version: str = Field("1.0.0", description="工具版本")
# 功能定义
schema: Dict[str, Any] = Field(..., description="工具schema")
parameters: List[APIParameter] = Field(default=[], description="参数定义")
returns: Optional[Dict[str, Any]] = Field(None, description="返回值定义")
# API相关
endpoint: Optional[str] = Field(None, description="API端点")
method: str = Field("POST", description="HTTP方法")
headers: Dict[str, str] = Field(default={}, description="请求头")
auth_type: Optional[str] = Field(None, description="认证类型")
# 限制和配置
rate_limit: int = Field(100, description="速率限制(每分钟)")
timeout: int = Field(30, description="超时时间(秒)")
cost_per_call: float = Field(0.0, description="每次调用成本")
max_retries: int = Field(3, description="最大重试次数")
# 状态和质量
status: str = Field("active", description="工具状态")
quality_score: Optional[float] = Field(None, description="质量分数")
usage_count: int = Field(0, description="使用次数")
success_rate: float = Field(0.0, description="成功率")
# 元数据
tags: List[str] = Field(default=[], description="标签")
author: Optional[str] = Field(None, description="作者")
license: Optional[str] = Field(None, description="许可证")
documentation_url: Optional[str] = Field(None, description="文档URL")
# 时间戳
created_at: datetime = Field(default_factory=datetime.utcnow, description="创建时间")
updated_at: datetime = Field(default_factory=datetime.utcnow, description="更新时间")
class ToolGenerationRequest(BaseSchema):
"""工具生成请求"""
endpoint: APIEndpoint = Field(..., description="API端点")
tool_name: Optional[str] = Field(None, description="自定义工具名称")
category: Optional[str] = Field(None, description="自定义分类")
# 生成选项
include_examples: bool = Field(True, description="是否包含示例")
optimize_for_llm: bool = Field(True, description="是否为LLM优化")
add_validation: bool = Field(True, description="是否添加验证")
generate_tests: bool = Field(False, description="是否生成测试")
class ToolGenerationResponse(BaseSchema):
"""工具生成响应"""
success: bool = Field(..., description="是否成功")
tool_name: str = Field(..., description="生成的工具名称")
tool_definition: Optional[ToolDefinition] = Field(None, description="工具定义")
generation_time: float = Field(..., description="生成时间(秒)")
error: Optional[str] = Field(None, description="错误信息")
warnings: List[str] = Field(default=[], description="警告信息")
# ========== 批处理相关 ==========
class BatchProcessRequest(BaseSchema):
"""批处理请求"""
items: List[Union[str, APIEndpoint]] = Field(..., description="处理项目列表")
process_type: str = Field(..., description="处理类型")
options: Dict[str, Any] = Field(default={}, description="处理选项")
@validator('process_type')
def validate_process_type(cls, v):
"""验证处理类型"""
allowed_types = ["parse_openapi", "generate_tools", "test_endpoints"]
if v not in allowed_types:
raise ValueError(f'不支持的处理类型: {v}')
return v
class BatchProcessResponse(BaseSchema):
"""批处理响应"""
total_items: int = Field(..., description="总项目数")
processed_items: int = Field(..., description="已处理项目数")
successful_items: int = Field(..., description="成功项目数")
failed_items: int = Field(..., description="失败项目数")
results: List[Dict[str, Any]] = Field(default=[], description="处理结果")
errors: List[Dict[str, Any]] = Field(default=[], description="错误列表")
start_time: datetime = Field(..., description="开始时间")
end_time: Optional[datetime] = Field(None, description="结束时间")
total_time: Optional[float] = Field(None, description="总时间(秒)")
# ========== 统计和监控相关 ==========
class ProcessingStats(BaseSchema):
"""处理统计"""
total_apis: int = Field(..., description="API总数")
processed_apis: int = Field(..., description="已处理API数")
generated_tools: int = Field(..., description="生成的工具数")
failed_processes: int = Field(..., description="失败的处理数")
success_rate: float = Field(..., description="成功率")
avg_processing_time: float = Field(..., description="平均处理时间")
categories: Dict[str, int] = Field(default={}, description="分类统计")
daily_stats: List[Dict[str, Any]] = Field(default=[], description="每日统计")
class SystemHealth(BaseSchema):
"""系统健康状态"""
status: str = Field(..., description="系统状态")
timestamp: datetime = Field(..., description="检查时间")
services: Dict[str, str] = Field(..., description="服务状态")
resources: Dict[str, Any] = Field(..., description="资源使用")
performance: Dict[str, float] = Field(..., description="性能指标")
errors: List[str] = Field(default=[], description="错误列表")
warnings: List[str] = Field(default=[], description="警告列表")
# ========== 配置和设置相关 ==========
class ServiceConfig(BaseSchema):
"""服务配置"""
rapidapi_enabled: bool = Field(True, description="是否启用RapidAPI")
apillama_enabled: bool = Field(True, description="是否启用APILLAMA")
auto_sync: bool = Field(True, description="是否自动同步")
sync_interval: int = Field(3600, description="同步间隔(秒)")
max_concurrent: int = Field(10, description="最大并发数")
cache_enabled: bool = Field(True, description="是否启用缓存")
quality_threshold: float = Field(0.7, description="质量阈值")
auto_cleanup: bool = Field(True, description="是否自动清理")
class APIResponse(BaseSchema):
"""标准API响应"""
success: bool = Field(..., description="是否成功")
message: str = Field("", description="响应消息")
data: Optional[Any] = Field(None, description="响应数据")
timestamp: datetime = Field(default_factory=datetime.utcnow, description="时间戳")
request_id: Optional[str] = Field(None, description="请求ID")
services/mcp-server/Dockerfile
+38
View File
@@ -0,0 +1,38 @@
FROM python:3.11-slim
# 设置工作目录
WORKDIR /app
# 安装系统依赖
RUN apt-get update && apt-get install -y \
gcc \
g++ \
make \
curl \
&& rm -rf /var/lib/apt/lists/*
# 复制requirements文件
COPY requirements.txt .
# 安装Python依赖
RUN pip install --no-cache-dir -r requirements.txt
# 复制源代码
COPY . .
# 创建logs目录
RUN mkdir -p logs
# 设置环境变量
ENV PYTHONPATH=/app
ENV PYTHONUNBUFFERED=1
# 暴露端口
EXPOSE 8000
# 健康检查
HEALTHCHECK --interval=30s --timeout=30s --start-period=5s --retries=3 \
CMD curl -f http://localhost:8000/health || exit 1
# 启动应用
CMD ["python", "-m", "uvicorn", "main:app", "--host", "0.0.0.0", "--port", "8000", "--reload"]
services/mcp-server/config.py
+121
View File
@@ -0,0 +1,121 @@
"""
配置管理
"""
import os
from typing import Optional
from pydantic_settings import BaseSettings
class Settings(BaseSettings):
"""应用配置"""
# 应用设置
app_name: str = "taiji-AI-PAD MCP Server"
debug: bool = False
secret_key: str = "your-secret-key-change-in-production"
# 数据库设置
database_url: str = os.getenv(
"DATABASE_URL",
"postgresql+asyncpg://taiji_user:taiji_pass@postgres:5432/taiji_db"
)
# Redis设置
redis_url: str = os.getenv("REDIS_URL", "redis://redis:6379")
redis_max_connections: int = 20
redis_retry_on_timeout: bool = True
# NATS设置
nats_url: str = os.getenv("NATS_URL", "nats://nats:4222")
nats_max_reconnect_attempts: int = 10
# LiteLLM网关设置
litellm_url: str = os.getenv("LITELLM_URL", "http://litellm-gateway:4000")
litellm_api_key: str = os.getenv("LITELLM_API_KEY", "sk-taiji-master-key")
# MCP协议设置
mcp_timeout: int = 30 # 秒
mcp_max_retries: int = 3
mcp_retry_delay: float = 1.0 # 秒
# Agent设置
max_agents_per_user: int = 100
agent_execution_timeout: int = 300 # 秒
agent_memory_limit: str = "512MB"
agent_cpu_limit: float = 1.0 # CPU核数
# 工具设置
max_tools_per_agent: int = 50
tool_execution_timeout: int = 60 # 秒
allowed_tool_domains: list = [
"rapidapi.com",
"api.openai.com",
"api.anthropic.com"
]
# 缓存设置
cache_ttl: int = 3600 # 秒
cache_max_size: int = 1000
# 日志设置
log_level: str = "INFO"
log_format: str = "json"
log_file: Optional[str] = "/app/logs/mcp-server.log"
# 安全设置
cors_origins: list = ["*"]
jwt_algorithm: str = "HS256"
jwt_expire_minutes: int = 60
# 监控设置
enable_metrics: bool = True
metrics_port: int = 8001
health_check_interval: int = 30 # 秒
# 开发设置
reload: bool = False
workers: int = 1
class Config:
env_file = ".env"
env_file_encoding = "utf-8"
case_sensitive = False
class DevelopmentSettings(Settings):
"""开发环境配置"""
debug: bool = True
reload: bool = True
log_level: str = "DEBUG"
class ProductionSettings(Settings):
"""生产环境配置"""
debug: bool = False
reload: bool = False
workers: int = 4
log_level: str = "INFO"
class TestingSettings(Settings):
"""测试环境配置"""
debug: bool = True
database_url: str = "sqlite+aiosqlite:///./test.db"
redis_url: str = "redis://localhost:6379/1" # 使用不同的数据库
def get_settings() -> Settings:
"""根据环境变量获取相应的配置"""
environment = os.getenv("ENVIRONMENT", "development").lower()
if environment == "production":
return ProductionSettings()
elif environment == "testing":
return TestingSettings()
else:
return DevelopmentSettings()
# 全局配置实例
settings = get_settings()
services/mcp-server/database.py
+404
View File
@@ -0,0 +1,404 @@
"""
数据库配置和连接管理
"""
import asyncio
from typing import AsyncGenerator
from sqlalchemy.ext.asyncio import AsyncSession, create_async_engine, async_sessionmaker
from sqlalchemy.orm import sessionmaker
from sqlalchemy import text
import logging
from .config import settings
from .models import Base
logger = logging.getLogger(__name__)
# 创建异步数据库引擎
engine = create_async_engine(
settings.database_url,
echo=settings.debug, # 在调试模式下显示SQL语句
pool_size=20,
max_overflow=0,
pool_pre_ping=True, # 连接池预检查
pool_recycle=3600, # 1小时后回收连接
)
# 创建异步会话工厂
AsyncSessionLocal = async_sessionmaker(
engine,
class_=AsyncSession,
expire_on_commit=False
)
async def get_db() -> AsyncGenerator[AsyncSession, None]:
"""获取数据库会话的依赖注入函数"""
async with AsyncSessionLocal() as session:
try:
yield session
except Exception as e:
logger.error(f"数据库会话错误: {e}")
await session.rollback()
raise
finally:
await session.close()
async def init_db():
"""初始化数据库"""
try:
# 创建所有表
async with engine.begin() as conn:
await conn.run_sync(Base.metadata.create_all)
logger.info("数据库初始化成功")
# 创建初始数据
await create_initial_data()
except Exception as e:
logger.error(f"数据库初始化失败: {e}")
raise
async def create_initial_data():
"""创建初始数据"""
try:
async with AsyncSessionLocal() as session:
# 检查是否已有数据
result = await session.execute(text("SELECT COUNT(*) FROM users"))
user_count = result.scalar()
if user_count == 0:
# 创建默认管理员用户
from .models import User
from passlib.context import CryptContext
pwd_context = CryptContext(schemes=["bcrypt"], deprecated="auto")
admin_user = User(
username="admin",
email="admin@taiji-ai.com",
hashed_password=pwd_context.hash("admin123"),
full_name="系统管理员",
is_active=True,
is_admin=True
)
session.add(admin_user)
await session.commit()
logger.info("默认管理员用户创建成功")
# 创建示例工具
await create_sample_tools(session)
except Exception as e:
logger.error(f"创建初始数据失败: {e}")
raise
async def create_sample_tools(session: AsyncSession):
"""创建示例工具"""
try:
from .models import Tool
# 检查是否已有工具
result = await session.execute(text("SELECT COUNT(*) FROM tools"))
tool_count = result.scalar()
if tool_count == 0:
# 创建示例工具
sample_tools = [
{
"name": "web_search",
"description": "网络搜索工具",
"category": "api",
"schema": {
"type": "object",
"properties": {
"query": {
"type": "string",
"description": "搜索查询"
},
"limit": {
"type": "integer",
"description": "结果数量限制",
"default": 10
}
},
"required": ["query"]
},
"endpoint": "https://api.example.com/search",
"method": "POST",
"auth_type": "api_key",
"rate_limit": 100,
"cost_per_call": 0.01,
"is_public": True
},
{
"name": "text_completion",
"description": "文本补全工具",
"category": "llm",
"schema": {
"type": "object",
"properties": {
"prompt": {
"type": "string",
"description": "输入提示"
},
"max_tokens": {
"type": "integer",
"description": "最大token数",
"default": 150
},
"temperature": {
"type": "number",
"description": "温度参数",
"default": 0.7
}
},
"required": ["prompt"]
},
"rate_limit": 60,
"cost_per_call": 0.05,
"is_public": True
},
{
"name": "weather_api",
"description": "天气查询API",
"category": "api",
"schema": {
"type": "object",
"properties": {
"city": {
"type": "string",
"description": "城市名称"
},
"unit": {
"type": "string",
"enum": ["celsius", "fahrenheit"],
"default": "celsius"
}
},
"required": ["city"]
},
"endpoint": "https://api.openweathermap.org/data/2.5/weather",
"method": "GET",
"auth_type": "api_key",
"rate_limit": 1000,
"cost_per_call": 0.001,
"is_public": True
}
]
for tool_data in sample_tools:
tool = Tool(**tool_data)
session.add(tool)
await session.commit()
logger.info(f"创建了 {len(sample_tools)} 个示例工具")
except Exception as e:
logger.error(f"创建示例工具失败: {e}")
raise
async def check_db_connection():
"""检查数据库连接"""
try:
async with AsyncSessionLocal() as session:
await session.execute(text("SELECT 1"))
return True
except Exception as e:
logger.error(f"数据库连接检查失败: {e}")
return False
async def get_db_stats():
"""获取数据库统计信息"""
try:
async with AsyncSessionLocal() as session:
stats = {}
# 获取各表的记录数
tables = ["users", "agents", "tools", "sessions", "executions", "billing"]
for table in tables:
result = await session.execute(text(f"SELECT COUNT(*) FROM {table}"))
stats[table] = result.scalar()
return stats
except Exception as e:
logger.error(f"获取数据库统计失败: {e}")
return {}
async def cleanup_old_records():
"""清理旧记录"""
try:
async with AsyncSessionLocal() as session:
# 清理超过30天的执行记录
result = await session.execute(text("""
DELETE FROM executions
WHERE created_at < NOW() - INTERVAL '30 days'
"""))
deleted_executions = result.rowcount
# 清理超过7天的会话记录
result = await session.execute(text("""
DELETE FROM sessions
WHERE created_at < NOW() - INTERVAL '7 days'
AND status != 'active'
"""))
deleted_sessions = result.rowcount
await session.commit()
logger.info(f"清理完成: 删除了 {deleted_executions} 条执行记录, {deleted_sessions} 条会话记录")
return {
"deleted_executions": deleted_executions,
"deleted_sessions": deleted_sessions
}
except Exception as e:
logger.error(f"清理旧记录失败: {e}")
return {}
async def backup_db():
"""数据库备份"""
try:
import subprocess
from datetime import datetime
import os
# 生成备份文件名
timestamp = datetime.now().strftime("%Y%m%d_%H%M%S")
backup_file = f"/app/backups/taiji_db_backup_{timestamp}.sql"
# 创建备份目录
os.makedirs("/app/backups", exist_ok=True)
# 执行pg_dump命令
cmd = [
"pg_dump",
settings.database_url.replace("postgresql+asyncpg://", "postgresql://"),
"-f", backup_file
]
result = subprocess.run(cmd, capture_output=True, text=True)
if result.returncode == 0:
logger.info(f"数据库备份成功: {backup_file}")
return backup_file
else:
logger.error(f"数据库备份失败: {result.stderr}")
return None
except Exception as e:
logger.error(f"数据库备份异常: {e}")
return None
async def close_db():
"""关闭数据库连接"""
try:
await engine.dispose()
logger.info("数据库连接已关闭")
except Exception as e:
logger.error(f"关闭数据库连接失败: {e}")
# 数据库事件处理
async def on_startup():
"""应用启动时的数据库操作"""
await init_db()
async def on_shutdown():
"""应用关闭时的数据库操作"""
await close_db()
# 定期清理任务
async def periodic_cleanup():
"""定期清理任务"""
while True:
try:
await asyncio.sleep(3600) # 每小时执行一次
await cleanup_old_records()
except Exception as e:
logger.error(f"定期清理任务异常: {e}")
# 数据库迁移辅助函数
async def migrate_db():
"""数据库迁移（简化版本）"""
try:
# 这里可以添加数据迁移逻辑
# 在生产环境中应该使用Alembic进行数据库版本管理
logger.info("数据库迁移检查完成")
except Exception as e:
logger.error(f"数据库迁移失败: {e}")
raise
# 性能优化
async def optimize_db():
"""数据库性能优化"""
try:
async with AsyncSessionLocal() as session:
# 更新表统计信息
await session.execute(text("ANALYZE;"))
# 重建索引（如果需要）
# await session.execute(text("REINDEX DATABASE taiji_db;"))
await session.commit()
logger.info("数据库优化完成")
except Exception as e:
logger.error(f"数据库优化失败: {e}")
# 健康检查
async def health_check() -> dict:
"""数据库健康检查"""
health_info = {
"database": "unknown",
"connection_pool": "unknown",
"stats": {}
}
try:
# 检查连接
if await check_db_connection():
health_info["database"] = "healthy"
else:
health_info["database"] = "unhealthy"
# 检查连接池状态
pool = engine.pool
health_info["connection_pool"] = {
"size": pool.size(),
"checked_in": pool.checkedin(),
"checked_out": pool.checkedout()
}
# 获取统计信息
health_info["stats"] = await get_db_stats()
except Exception as e:
logger.error(f"数据库健康检查失败: {e}")
health_info["database"] = "error"
health_info["error"] = str(e)
return health_info
services/mcp-server/main.py
+427
View File
@@ -0,0 +1,427 @@
"""
taiji-AI-PAD MCP Server
核心MCP协议服务器，负责Agent注册、工具管理和协议通信
"""
import asyncio
import json
import logging
import os
from datetime import datetime
from typing import Any, Dict, List, Optional
import structlog
from fastapi import FastAPI, HTTPException, Depends, WebSocket, WebSocketDisconnect
from fastapi.middleware.cors import CORSMiddleware
from fastapi.responses import JSONResponse
from pydantic import BaseModel
import redis.asyncio as redis
import nats
from sqlalchemy.ext.asyncio import AsyncSession, create_async_engine
from sqlalchemy.orm import sessionmaker
from .models import Agent, Tool, Session as DBSession
from .schemas import (
AgentCard,
AgentCreateRequest,
ToolDefinition,
MCPRequest,
MCPResponse,
ExecutionResult
)
from .mcp_protocol import MCPProtocolHandler
from .database import get_db, init_db
from .config import Settings
# 配置日志
structlog.configure(
processors=[
structlog.stdlib.filter_by_level,
structlog.stdlib.add_logger_name,
structlog.stdlib.add_log_level,
structlog.stdlib.PositionalArgumentsFormatter(),
structlog.processors.TimeStamper(fmt="iso"),
structlog.processors.StackInfoRenderer(),
structlog.processors.format_exc_info,
structlog.processors.UnicodeDecoder(),
structlog.processors.JSONRenderer()
],
context_class=dict,
logger_factory=structlog.stdlib.LoggerFactory(),
cache_logger_on_first_use=True,
)
logger = structlog.get_logger()
# 应用设置
settings = Settings()
app = FastAPI(
title="taiji-AI-PAD MCP Server",
description="Model Context Protocol Server for Agent Management",
version="1.0.0",
docs_url="/docs",
redoc_url="/redoc"
)
# CORS配置
app.add_middleware(
CORSMiddleware,
allow_origins=["*"],
allow_credentials=True,
allow_methods=["*"],
allow_headers=["*"],
)
# 全局变量
redis_client: Optional[redis.Redis] = None
nats_client: Optional[nats.NATS] = None
mcp_handler: Optional[MCPProtocolHandler] = None
active_websockets: Dict[str, WebSocket] = {}
class HealthResponse(BaseModel):
status: str
timestamp: str
services: Dict[str, str]
@app.on_event("startup")
async def startup_event():
"""应用启动时初始化服务"""
global redis_client, nats_client, mcp_handler
try:
# 初始化数据库
await init_db()
logger.info("数据库初始化完成")
# 连接Redis
redis_client = redis.from_url(
settings.redis_url,
encoding="utf-8",
decode_responses=True
)
await redis_client.ping()
logger.info("Redis连接成功")
# 连接NATS
nats_client = await nats.connect(settings.nats_url)
logger.info("NATS连接成功")
# 初始化MCP协议处理器
mcp_handler = MCPProtocolHandler(redis_client, nats_client)
logger.info("MCP协议处理器初始化完成")
# 注册NATS事件处理器
await setup_nats_handlers()
logger.info("MCP Server启动完成")
except Exception as e:
logger.error(f"服务启动失败: {e}")
raise
@app.on_event("shutdown")
async def shutdown_event():
"""应用关闭时清理资源"""
global redis_client, nats_client
try:
# 关闭所有WebSocket连接
for ws in active_websockets.values():
await ws.close()
# 关闭NATS连接
if nats_client:
await nats_client.close()
# 关闭Redis连接
if redis_client:
await redis_client.close()
logger.info("资源清理完成")
except Exception as e:
logger.error(f"资源清理失败: {e}")
async def setup_nats_handlers():
"""设置NATS事件处理器"""
if not nats_client:
return
# Agent执行事件
await nats_client.subscribe("agent.execution.*", cb=handle_agent_execution)
# 计费事件
await nats_client.subscribe("billing.*", cb=handle_billing_event)
# 系统事件
await nats_client.subscribe("system.*", cb=handle_system_event)
async def handle_agent_execution(msg):
"""处理Agent执行事件"""
try:
data = json.loads(msg.data.decode())
logger.info(f"收到Agent执行事件: {data}")
# 广播给相关的WebSocket连接
for ws in active_websockets.values():
await ws.send_json({
"type": "agent_execution",
"data": data
})
except Exception as e:
logger.error(f"处理Agent执行事件失败: {e}")
async def handle_billing_event(msg):
"""处理计费事件"""
try:
data = json.loads(msg.data.decode())
logger.info(f"收到计费事件: {data}")
except Exception as e:
logger.error(f"处理计费事件失败: {e}")
async def handle_system_event(msg):
"""处理系统事件"""
try:
data = json.loads(msg.data.decode())
logger.info(f"收到系统事件: {data}")
except Exception as e:
logger.error(f"处理系统事件失败: {e}")
@app.get("/health", response_model=HealthResponse)
async def health_check():
"""健康检查端点"""
services = {
"mcp_server": "healthy",
"redis": "unknown",
"nats": "unknown",
"database": "unknown"
}
# 检查Redis
try:
if redis_client:
await redis_client.ping()
services["redis"] = "healthy"
except Exception:
services["redis"] = "unhealthy"
# 检查NATS
try:
if nats_client and nats_client.is_connected:
services["nats"] = "healthy"
except Exception:
services["nats"] = "unhealthy"
# 检查数据库连接
try:
# 这里应该有数据库连接检查
services["database"] = "healthy"
except Exception:
services["database"] = "unhealthy"
return HealthResponse(
status="healthy",
timestamp=datetime.utcnow().isoformat(),
services=services
)
@app.post("/agents", response_model=AgentCard)
async def create_agent(
request: AgentCreateRequest,
db: AsyncSession = Depends(get_db)
):
"""创建新的Agent"""
try:
# 创建Agent记录
agent = Agent(
name=request.name,
description=request.description,
role=request.role,
goal=request.goal,
tools=request.tools,
config=request.config,
owner_id=request.owner_id
)
db.add(agent)
await db.commit()
await db.refresh(agent)
# 生成Agent Card
agent_card = AgentCard(
id=agent.id,
name=agent.name,
description=agent.description,
role=agent.role,
goal=agent.goal,
tools=agent.tools,
endpoints={
"mcp": f"mcp://localhost:8002/agents/{agent.id}",
"http": f"http://localhost:8002/agents/{agent.id}",
"websocket": f"ws://localhost:8002/agents/{agent.id}/ws"
},
created_at=agent.created_at,
updated_at=agent.updated_at
)
# 缓存到Redis
if redis_client:
await redis_client.setex(
f"agent:{agent.id}",
3600, # 1小时过期
agent_card.json()
)
# 发布Agent创建事件
if nats_client:
await nats_client.publish(
"agent.created",
json.dumps({
"agent_id": agent.id,
"name": agent.name,
"timestamp": datetime.utcnow().isoformat()
}).encode()
)
logger.info(f"Agent创建成功: {agent.id}")
return agent_card
except Exception as e:
logger.error(f"创建Agent失败: {e}")
raise HTTPException(status_code=500, detail=str(e))
@app.get("/agents", response_model=List[AgentCard])
async def list_agents(
skip: int = 0,
limit: int = 100,
db: AsyncSession = Depends(get_db)
):
"""获取Agent列表"""
try:
# 从数据库获取Agent列表
# 这里应该有实际的数据库查询逻辑
agents = [] # 临时空列表
return agents
except Exception as e:
logger.error(f"获取Agent列表失败: {e}")
raise HTTPException(status_code=500, detail=str(e))
@app.get("/agents/{agent_id}", response_model=AgentCard)
async def get_agent(agent_id: str, db: AsyncSession = Depends(get_db)):
"""获取特定Agent信息"""
try:
# 先从Redis缓存查找
if redis_client:
cached = await redis_client.get(f"agent:{agent_id}")
if cached:
return AgentCard.parse_raw(cached)
# 从数据库查找
# 这里应该有实际的数据库查询逻辑
raise HTTPException(status_code=404, detail="Agent not found")
except HTTPException:
raise
except Exception as e:
logger.error(f"获取Agent失败: {e}")
raise HTTPException(status_code=500, detail=str(e))
@app.post("/agents/{agent_id}/execute", response_model=ExecutionResult)
async def execute_agent(
agent_id: str,
request: MCPRequest,
db: AsyncSession = Depends(get_db)
):
"""执行Agent任务"""
try:
if not mcp_handler:
raise HTTPException(status_code=500, detail="MCP handler not initialized")
# 执行MCP请求
result = await mcp_handler.execute_request(agent_id, request)
# 发布执行事件
if nats_client:
await nats_client.publish(
f"agent.execution.{agent_id}",
json.dumps({
"agent_id": agent_id,
"request_id": request.id,
"method": request.method,
"timestamp": datetime.utcnow().isoformat(),
"success": result.success
}).encode()
)
return result
except Exception as e:
logger.error(f"执行Agent任务失败: {e}")
raise HTTPException(status_code=500, detail=str(e))
@app.get("/tools", response_model=List[ToolDefinition])
async def list_tools(db: AsyncSession = Depends(get_db)):
"""获取可用工具列表"""
try:
# 从数据库获取工具列表
# 这里应该有实际的工具查询逻辑
tools = []
return tools
except Exception as e:
logger.error(f"获取工具列表失败: {e}")
raise HTTPException(status_code=500, detail=str(e))
@app.websocket("/agents/{agent_id}/ws")
async def websocket_endpoint(websocket: WebSocket, agent_id: str):
"""Agent WebSocket连接端点"""
await websocket.accept()
active_websockets[agent_id] = websocket
try:
logger.info(f"WebSocket连接建立: {agent_id}")
while True:
# 等待客户端消息
data = await websocket.receive_json()
# 处理MCP消息
if mcp_handler and data.get("type") == "mcp_request":
request = MCPRequest(**data["payload"])
result = await mcp_handler.execute_request(agent_id, request)
await websocket.send_json({
"type": "mcp_response",
"payload": result.dict()
})
except WebSocketDisconnect:
logger.info(f"WebSocket连接断开: {agent_id}")
except Exception as e:
logger.error(f"WebSocket错误: {e}")
finally:
if agent_id in active_websockets:
del active_websockets[agent_id]
@app.get("/metrics")
async def get_metrics():
"""Prometheus metrics端点"""
# 这里应该返回Prometheus格式的metrics
return JSONResponse({"message": "Metrics endpoint - TODO: implement Prometheus metrics"})
if __name__ == "__main__":
import uvicorn
uvicorn.run(
"main:app",
host="0.0.0.0",
port=8000,
reload=True,
log_level="info"
)
services/mcp-server/mcp_protocol.py
+595
View File
@@ -0,0 +1,595 @@
"""
MCP (Model Context Protocol) 协议处理器
实现MCP协议的核心功能，包括工具管理、资源管理和代理通信
"""
import json
import uuid
from datetime import datetime
from typing import Any, Dict, List, Optional, Union
import asyncio
import logging
import traceback
import redis.asyncio as redis
import nats
import httpx
from .schemas import (
MCPRequest, MCPResponse, MCPError,
ToolDefinition, ToolResult, ExecutionResult
)
logger = logging.getLogger(__name__)
class MCPProtocolHandler:
"""MCP协议处理器"""
def __init__(
self,
redis_client: redis.Redis,
nats_client: nats.NATS,
litellm_url: str = "http://litellm-gateway:4000"
):
self.redis = redis_client
self.nats = nats_client
self.litellm_url = litellm_url
self.http_client = httpx.AsyncClient(timeout=30.0)
# MCP协议版本
self.protocol_version = "2024-11-05"
# 支持的MCP方法
self.supported_methods = {
"initialize",
"tools/list",
"tools/call",
"resources/list",
"resources/read",
"prompts/list",
"prompts/get",
"completion/complete",
"logging/setLevel"
}
# 工具注册表
self._tools_registry: Dict[str, ToolDefinition] = {}
# 资源注册表
self._resources_registry: Dict[str, Dict[str, Any]] = {}
# 会话管理
self._sessions: Dict[str, Dict[str, Any]] = {}
async def execute_request(self, agent_id: str, request: MCPRequest) -> ExecutionResult:
"""执行MCP请求"""
execution_id = str(uuid.uuid4())
started_at = datetime.utcnow()
logger.info(f"开始执行MCP请求: {execution_id}, method: {request.method}")
try:
# 验证方法是否支持
if request.method not in self.supported_methods:
raise ValueError(f"不支持的MCP方法: {request.method}")
# 发布执行开始事件
await self._publish_execution_event(
"execution.started",
{
"execution_id": execution_id,
"agent_id": agent_id,
"method": request.method,
"timestamp": started_at.isoformat()
}
)
# 执行具体方法
result = await self._dispatch_method(agent_id, request)
completed_at = datetime.utcnow()
execution_time = (completed_at - started_at).total_seconds() * 1000
# 发布执行完成事件
await self._publish_execution_event(
"execution.completed",
{
"execution_id": execution_id,
"agent_id": agent_id,
"method": request.method,
"execution_time": execution_time,
"success": True,
"timestamp": completed_at.isoformat()
}
)
return ExecutionResult(
execution_id=execution_id,
success=True,
result=result,
execution_time=execution_time,
started_at=started_at,
completed_at=completed_at
)
except Exception as e:
completed_at = datetime.utcnow()
execution_time = (completed_at - started_at).total_seconds() * 1000
error_msg = str(e)
logger.error(f"MCP请求执行失败: {execution_id}, error: {error_msg}")
logger.error(traceback.format_exc())
# 发布执行失败事件
await self._publish_execution_event(
"execution.failed",
{
"execution_id": execution_id,
"agent_id": agent_id,
"method": request.method,
"execution_time": execution_time,
"error": error_msg,
"timestamp": completed_at.isoformat()
}
)
return ExecutionResult(
execution_id=execution_id,
success=False,
error=error_msg,
execution_time=execution_time,
started_at=started_at,
completed_at=completed_at
)
async def _dispatch_method(self, agent_id: str, request: MCPRequest) -> Any:
"""分发MCP方法调用"""
method = request.method
params = request.params or {}
if method == "initialize":
return await self._handle_initialize(params)
elif method == "tools/list":
return await self._handle_tools_list(agent_id, params)
elif method == "tools/call":
return await self._handle_tools_call(agent_id, params)
elif method == "resources/list":
return await self._handle_resources_list(agent_id, params)
elif method == "resources/read":
return await self._handle_resources_read(agent_id, params)
elif method == "prompts/list":
return await self._handle_prompts_list(agent_id, params)
elif method == "prompts/get":
return await self._handle_prompts_get(agent_id, params)
elif method == "completion/complete":
return await self._handle_completion_complete(agent_id, params)
elif method == "logging/setLevel":
return await self._handle_logging_set_level(params)
else:
raise ValueError(f"未实现的方法: {method}")
async def _handle_initialize(self, params: Dict[str, Any]) -> Dict[str, Any]:
"""处理初始化请求"""
client_info = params.get("clientInfo", {})
protocol_version = params.get("protocolVersion")
logger.info(f"MCP客户端初始化: {client_info}")
return {
"protocolVersion": self.protocol_version,
"capabilities": {
"tools": {
"listChanged": True
},
"resources": {
"subscribe": True,
"listChanged": True
},
"prompts": {
"listChanged": True
},
"completion": {
"argument": True
},
"logging": {}
},
"serverInfo": {
"name": "taiji-AI-PAD MCP Server",
"version": "1.0.0"
}
}
async def _handle_tools_list(self, agent_id: str, params: Dict[str, Any]) -> Dict[str, Any]:
"""处理工具列表请求"""
try:
# 从Redis获取Agent的工具列表
agent_tools_key = f"agent:{agent_id}:tools"
tool_names = await self.redis.smembers(agent_tools_key)
tools = []
for tool_name in tool_names:
tool_info = await self._get_tool_info(tool_name)
if tool_info:
tools.append({
"name": tool_info["name"],
"description": tool_info["description"],
"inputSchema": tool_info.get("schema", {})
})
return {"tools": tools}
except Exception as e:
logger.error(f"获取工具列表失败: {e}")
return {"tools": []}
async def _handle_tools_call(self, agent_id: str, params: Dict[str, Any]) -> Dict[str, Any]:
"""处理工具调用请求"""
tool_name = params.get("name")
arguments = params.get("arguments", {})
if not tool_name:
raise ValueError("工具名称不能为空")
logger.info(f"调用工具: {tool_name}, arguments: {arguments}")
try:
# 验证Agent是否有权限使用该工具
agent_tools_key = f"agent:{agent_id}:tools"
if not await self.redis.sismember(agent_tools_key, tool_name):
raise ValueError(f"Agent {agent_id} 无权限使用工具 {tool_name}")
# 获取工具信息
tool_info = await self._get_tool_info(tool_name)
if not tool_info:
raise ValueError(f"工具 {tool_name} 不存在")
# 执行工具调用
result = await self._execute_tool(tool_name, tool_info, arguments)
return {
"content": [
{
"type": "text",
"text": json.dumps(result.result) if result.success else f"错误: {result.error}"
}
],
"isError": not result.success
}
except Exception as e:
logger.error(f"工具调用失败: {e}")
return {
"content": [
{
"type": "text",
"text": f"工具调用失败: {str(e)}"
}
],
"isError": True
}
async def _handle_resources_list(self, agent_id: str, params: Dict[str, Any]) -> Dict[str, Any]:
"""处理资源列表请求"""
try:
# 从Redis获取Agent的资源列表
agent_resources_key = f"agent:{agent_id}:resources"
resource_names = await self.redis.smembers(agent_resources_key)
resources = []
for resource_name in resource_names:
resource_info = await self._get_resource_info(resource_name)
if resource_info:
resources.append({
"uri": resource_info["uri"],
"name": resource_info["name"],
"description": resource_info.get("description"),
"mimeType": resource_info.get("mimeType", "application/json")
})
return {"resources": resources}
except Exception as e:
logger.error(f"获取资源列表失败: {e}")
return {"resources": []}
async def _handle_resources_read(self, agent_id: str, params: Dict[str, Any]) -> Dict[str, Any]:
"""处理资源读取请求"""
uri = params.get("uri")
if not uri:
raise ValueError("资源URI不能为空")
try:
# 验证权限
agent_resources_key = f"agent:{agent_id}:resources"
# 这里应该根据URI找到资源名称
resource_name = uri.split("/")[-1] # 简化处理
if not await self.redis.sismember(agent_resources_key, resource_name):
raise ValueError(f"Agent {agent_id} 无权限访问资源 {uri}")
# 读取资源内容
content = await self._read_resource_content(uri)
return {
"contents": [
{
"uri": uri,
"mimeType": "application/json",
"text": json.dumps(content)
}
]
}
except Exception as e:
logger.error(f"读取资源失败: {e}")
raise ValueError(f"无法读取资源 {uri}: {str(e)}")
async def _handle_prompts_list(self, agent_id: str, params: Dict[str, Any]) -> Dict[str, Any]:
"""处理提示词列表请求"""
# 获取Agent相关的提示词
prompts = [
{
"name": "system_prompt",
"description": "系统提示词",
"arguments": [
{
"name": "context",
"description": "上下文信息",
"required": False
}
]
}
]
return {"prompts": prompts}
async def _handle_prompts_get(self, agent_id: str, params: Dict[str, Any]) -> Dict[str, Any]:
"""处理获取提示词请求"""
name = params.get("name")
arguments = params.get("arguments", {})
if name == "system_prompt":
# 构建系统提示词
agent_info = await self._get_agent_info(agent_id)
prompt = f"""你是 {agent_info.get('name', 'AI助手')}。
角色定义: {agent_info.get('role', '通用助手')}
目标: {agent_info.get('goal', '帮助用户完成任务')}
可用工具: {', '.join(agent_info.get('tools', []))}
请根据用户的请求，选择合适的工具来完成任务。"""
return {
"description": "Agent系统提示词",
"messages": [
{
"role": "system",
"content": {
"type": "text",
"text": prompt
}
}
]
}
raise ValueError(f"未知的提示词: {name}")
async def _handle_completion_complete(self, agent_id: str, params: Dict[str, Any]) -> Dict[str, Any]:
"""处理补全请求"""
ref = params.get("ref", {})
argument = params.get("argument", {})
# 根据参考信息生成补全建议
completions = []
if ref.get("type") == "tool":
tool_name = ref.get("name")
if tool_name:
tool_info = await self._get_tool_info(tool_name)
if tool_info and "schema" in tool_info:
# 基于工具schema生成参数建议
schema = tool_info["schema"]
properties = schema.get("properties", {})
for prop_name, prop_info in properties.items():
completions.append({
"type": "text",
"text": prop_name,
"insertText": f'"{prop_name}": ""'
})
return {"completion": {"values": completions}}
async def _handle_logging_set_level(self, params: Dict[str, Any]) -> Dict[str, Any]:
"""处理设置日志级别请求"""
level = params.get("level", "info")
# 设置日志级别
numeric_level = getattr(logging, level.upper(), logging.INFO)
logging.getLogger().setLevel(numeric_level)
logger.info(f"日志级别已设置为: {level}")
return {"success": True}
async def _execute_tool(
self,
tool_name: str,
tool_info: Dict[str, Any],
arguments: Dict[str, Any]
) -> ToolResult:
"""执行工具调用"""
start_time = datetime.utcnow()
try:
# 根据工具类型执行不同的逻辑
category = tool_info.get("category", "api")
if category == "api":
result = await self._execute_api_tool(tool_info, arguments)
elif category == "function":
result = await self._execute_function_tool(tool_info, arguments)
elif category == "llm":
result = await self._execute_llm_tool(tool_info, arguments)
else:
raise ValueError(f"不支持的工具类型: {category}")
execution_time = (datetime.utcnow() - start_time).total_seconds() * 1000
return ToolResult(
success=True,
result=result,
execution_time=execution_time,
cost=tool_info.get("cost_per_call", 0.0)
)
except Exception as e:
execution_time = (datetime.utcnow() - start_time).total_seconds() * 1000
return ToolResult(
success=False,
error=str(e),
execution_time=execution_time,
cost=tool_info.get("cost_per_call", 0.0)
)
async def _execute_api_tool(self, tool_info: Dict[str, Any], arguments: Dict[str, Any]) -> Any:
"""执行API工具调用"""
endpoint = tool_info.get("endpoint")
method = tool_info.get("method", "POST")
headers = tool_info.get("headers", {})
timeout = tool_info.get("timeout", 30)
if not endpoint:
raise ValueError("API端点不能为空")
# 发送HTTP请求
response = await self.http_client.request(
method=method,
url=endpoint,
json=arguments,
headers=headers,
timeout=timeout
)
response.raise_for_status()
return response.json()
async def _execute_function_tool(self, tool_info: Dict[str, Any], arguments: Dict[str, Any]) -> Any:
"""执行函数工具调用"""
# 这里可以调用本地Python函数
# 为了安全起见，需要严格的沙箱机制
raise NotImplementedError("函数工具调用暂未实现")
async def _execute_llm_tool(self, tool_info: Dict[str, Any], arguments: Dict[str, Any]) -> Any:
"""执行LLM工具调用"""
# 调用LiteLLM网关
payload = {
"model": arguments.get("model", "gpt-3.5-turbo"),
"messages": arguments.get("messages", []),
"temperature": arguments.get("temperature", 0.7),
"max_tokens": arguments.get("max_tokens", 150)
}
response = await self.http_client.post(
f"{self.litellm_url}/chat/completions",
json=payload,
headers={"Authorization": "Bearer sk-taiji-master-key"}
)
response.raise_for_status()
return response.json()
async def _get_tool_info(self, tool_name: str) -> Optional[Dict[str, Any]]:
"""获取工具信息"""
tool_key = f"tool:{tool_name}"
tool_data = await self.redis.get(tool_key)
if tool_data:
return json.loads(tool_data)
return None
async def _get_resource_info(self, resource_name: str) -> Optional[Dict[str, Any]]:
"""获取资源信息"""
resource_key = f"resource:{resource_name}"
resource_data = await self.redis.get(resource_key)
if resource_data:
return json.loads(resource_data)
return None
async def _get_agent_info(self, agent_id: str) -> Dict[str, Any]:
"""获取Agent信息"""
agent_key = f"agent:{agent_id}"
agent_data = await self.redis.get(agent_key)
if agent_data:
return json.loads(agent_data)
return {}
async def _read_resource_content(self, uri: str) -> Any:
"""读取资源内容"""
# 这里可以根据URI类型读取不同的资源
# 例如：文件、数据库、API等
if uri.startswith("file://"):
# 读取文件
file_path = uri[7:] # 移除file://前缀
with open(file_path, 'r') as f:
return f.read()
elif uri.startswith("http://") or uri.startswith("https://"):
# 读取HTTP资源
response = await self.http_client.get(uri)
response.raise_for_status()
return response.json()
else:
raise ValueError(f"不支持的资源类型: {uri}")
async def _publish_execution_event(self, event_type: str, data: Dict[str, Any]):
"""发布执行事件"""
try:
if self.nats:
await self.nats.publish(
f"mcp.{event_type}",
json.dumps(data).encode()
)
except Exception as e:
logger.error(f"发布事件失败: {e}")
async def register_tool(self, tool_definition: ToolDefinition) -> bool:
"""注册工具"""
try:
tool_key = f"tool:{tool_definition.name}"
tool_data = tool_definition.dict()
await self.redis.setex(
tool_key,
3600, # 1小时过期
json.dumps(tool_data)
)
logger.info(f"工具注册成功: {tool_definition.name}")
return True
except Exception as e:
logger.error(f"工具注册失败: {e}")
return False
async def register_agent_tool(self, agent_id: str, tool_name: str) -> bool:
"""为Agent注册工具"""
try:
agent_tools_key = f"agent:{agent_id}:tools"
await self.redis.sadd(agent_tools_key, tool_name)
await self.redis.expire(agent_tools_key, 3600)
logger.info(f"Agent {agent_id} 工具注册成功: {tool_name}")
return True
except Exception as e:
logger.error(f"Agent工具注册失败: {e}")
return False
async def close(self):
"""清理资源"""
try:
await self.http_client.aclose()
except Exception as e:
logger.error(f"资源清理失败: {e}")
services/mcp-server/models.py
+292
View File
@@ -0,0 +1,292 @@
"""
数据库模型定义
"""
import uuid
from datetime import datetime
from typing import Dict, List, Optional, Any
from sqlalchemy import (
Column, String, Text, DateTime, Boolean, Integer,
JSON, ForeignKey, Index, UniqueConstraint
)
from sqlalchemy.ext.declarative import declarative_base
from sqlalchemy.orm import relationship
from sqlalchemy.dialects.postgresql import UUID
import sqlalchemy as sa
Base = declarative_base()
class BaseModel:
"""基础模型类"""
id = Column(UUID(as_uuid=True), primary_key=True, default=uuid.uuid4)
created_at = Column(DateTime, default=datetime.utcnow, nullable=False)
updated_at = Column(
DateTime,
default=datetime.utcnow,
onupdate=datetime.utcnow,
nullable=False
)
class User(BaseModel, Base):
"""用户模型"""
__tablename__ = "users"
username = Column(String(50), unique=True, nullable=False)
email = Column(String(255), unique=True, nullable=False)
hashed_password = Column(String(255), nullable=False)
full_name = Column(String(100))
is_active = Column(Boolean, default=True)
is_admin = Column(Boolean, default=False)
# 关联关系
agents = relationship("Agent", back_populates="owner", cascade="all, delete-orphan")
sessions = relationship("Session", back_populates="user", cascade="all, delete-orphan")
# 索引
__table_args__ = (
Index("idx_user_username", username),
Index("idx_user_email", email),
)
class Agent(BaseModel, Base):
"""Agent模型"""
__tablename__ = "agents"
name = Column(String(100), nullable=False)
description = Column(Text)
role = Column(String(200), nullable=False) # Agent的角色定义
goal = Column(Text, nullable=False) # Agent的目标描述
# Agent配置
config = Column(JSON, default=dict) # Agent的配置信息
tools = Column(JSON, default=list) # Agent授权使用的工具列表
capabilities = Column(JSON, default=list) # Agent的能力列表
# 状态信息
status = Column(String(20), default="active") # active, inactive, error
version = Column(String(20), default="1.0.0")
# 性能统计
total_executions = Column(Integer, default=0)
success_rate = Column(sa.Float, default=0.0)
avg_execution_time = Column(sa.Float, default=0.0) # 毫秒
# 关联关系
owner_id = Column(UUID(as_uuid=True), ForeignKey("users.id"), nullable=False)
owner = relationship("User", back_populates="agents")
executions = relationship("Execution", back_populates="agent", cascade="all, delete-orphan")
# 索引和约束
__table_args__ = (
Index("idx_agent_name", name),
Index("idx_agent_owner", owner_id),
Index("idx_agent_status", status),
UniqueConstraint("name", "owner_id", name="uq_agent_name_owner"),
)
class Tool(BaseModel, Base):
"""工具模型"""
__tablename__ = "tools"
name = Column(String(100), nullable=False)
description = Column(Text)
category = Column(String(50)) # api, function, integration等
# 工具定义
schema = Column(JSON, nullable=False) # OpenAPI或Pydantic schema
endpoint = Column(String(500)) # API端点URL
method = Column(String(10), default="POST") # HTTP方法
# 认证信息
auth_type = Column(String(20)) # api_key, oauth, basic等
auth_config = Column(JSON, default=dict)
# 限制和配额
rate_limit = Column(Integer, default=100) # 每分钟调用次数
cost_per_call = Column(sa.Float, default=0.0) # 每次调用成本(EU)
timeout = Column(Integer, default=30) # 超时时间(秒)
# 状态信息
is_active = Column(Boolean, default=True)
is_public = Column(Boolean, default=False) # 是否公开可用
# 统计信息
total_calls = Column(Integer, default=0)
success_rate = Column(sa.Float, default=0.0)
avg_response_time = Column(sa.Float, default=0.0)
# 关联关系
owner_id = Column(UUID(as_uuid=True), ForeignKey("users.id"))
owner = relationship("User")
# 索引
__table_args__ = (
Index("idx_tool_name", name),
Index("idx_tool_category", category),
Index("idx_tool_active", is_active),
)
class Session(BaseModel, Base):
"""会话模型"""
__tablename__ = "sessions"
session_id = Column(String(100), unique=True, nullable=False)
# 会话信息
context = Column(JSON, default=dict) # 会话上下文
metadata = Column(JSON, default=dict) # 元数据
# 状态
status = Column(String(20), default="active") # active, completed, failed
# 关联关系
user_id = Column(UUID(as_uuid=True), ForeignKey("users.id"), nullable=False)
user = relationship("User", back_populates="sessions")
executions = relationship("Execution", back_populates="session", cascade="all, delete-orphan")
# 索引
__table_args__ = (
Index("idx_session_id", session_id),
Index("idx_session_user", user_id),
Index("idx_session_status", status),
)
class Execution(BaseModel, Base):
"""执行记录模型"""
__tablename__ = "executions"
execution_id = Column(String(100), unique=True, nullable=False)
# 执行信息
method = Column(String(50), nullable=False) # MCP方法名
params = Column(JSON, default=dict) # 执行参数
result = Column(JSON, default=dict) # 执行结果
error = Column(Text) # 错误信息
# 时间信息
started_at = Column(DateTime, nullable=False)
completed_at = Column(DateTime)
execution_time = Column(sa.Float) # 执行时间(毫秒)
# 状态
status = Column(String(20), nullable=False) # running, completed, failed
# 资源消耗
cpu_usage = Column(sa.Float, default=0.0) # CPU使用率
memory_usage = Column(sa.Float, default=0.0) # 内存使用(MB)
network_io = Column(sa.Float, default=0.0) # 网络IO(KB)
eu_consumed = Column(sa.Float, default=0.0) # 消耗的执行单元
# 关联关系
agent_id = Column(UUID(as_uuid=True), ForeignKey("agents.id"), nullable=False)
agent = relationship("Agent", back_populates="executions")
session_id = Column(UUID(as_uuid=True), ForeignKey("sessions.id"))
session = relationship("Session", back_populates="executions")
# 索引
__table_args__ = (
Index("idx_execution_id", execution_id),
Index("idx_execution_agent", agent_id),
Index("idx_execution_status", status),
Index("idx_execution_started", started_at),
)
class APIKey(BaseModel, Base):
"""API密钥模型"""
__tablename__ = "api_keys"
name = Column(String(100), nullable=False)
key_hash = Column(String(255), nullable=False) # 哈希后的密钥
prefix = Column(String(20), nullable=False) # 密钥前缀(用于识别)
# 权限和限制
scopes = Column(JSON, default=list) # 权限范围
rate_limit = Column(Integer, default=1000) # 速率限制
is_active = Column(Boolean, default=True)
expires_at = Column(DateTime)
# 使用统计
last_used_at = Column(DateTime)
total_requests = Column(Integer, default=0)
# 关联关系
user_id = Column(UUID(as_uuid=True), ForeignKey("users.id"), nullable=False)
user = relationship("User")
# 索引
__table_args__ = (
Index("idx_api_key_hash", key_hash),
Index("idx_api_key_prefix", prefix),
Index("idx_api_key_user", user_id),
)
class Billing(BaseModel, Base):
"""计费记录模型"""
__tablename__ = "billing"
# 计费信息
eu_consumed = Column(sa.Float, nullable=False) # 消耗的执行单元
cost = Column(sa.Float, nullable=False) # 成本
currency = Column(String(3), default="USD")
# 资源详情
cpu_time = Column(sa.Float, default=0.0) # CPU时间(秒)
memory_max = Column(sa.Float, default=0.0) # 峰值内存(MB)
network_io = Column(sa.Float, default=0.0) # 网络IO(KB)
storage_io = Column(sa.Float, default=0.0) # 存储IO(KB)
# 关联关系
execution_id = Column(UUID(as_uuid=True), ForeignKey("executions.id"), nullable=False)
execution = relationship("Execution")
user_id = Column(UUID(as_uuid=True), ForeignKey("users.id"), nullable=False)
user = relationship("User")
# 索引
__table_args__ = (
Index("idx_billing_execution", execution_id),
Index("idx_billing_user", user_id),
Index("idx_billing_created", created_at),
)
class AuditLog(BaseModel, Base):
"""审计日志模型"""
__tablename__ = "audit_logs"
# 操作信息
action = Column(String(50), nullable=False) # 操作类型
resource_type = Column(String(50), nullable=False) # 资源类型
resource_id = Column(String(100)) # 资源ID
# 详细信息
details = Column(JSON, default=dict) # 操作详情
ip_address = Column(String(45)) # IP地址
user_agent = Column(Text) # 用户代理
# 结果
success = Column(Boolean, nullable=False)
error_message = Column(Text)
# 关联关系
user_id = Column(UUID(as_uuid=True), ForeignKey("users.id"))
user = relationship("User")
# 索引
__table_args__ = (
Index("idx_audit_action", action),
Index("idx_audit_resource", resource_type, resource_id),
Index("idx_audit_user", user_id),
Index("idx_audit_created", created_at),
)
services/mcp-server/requirements.txt
+62
View File
@@ -0,0 +1,62 @@
# Web框架
fastapi==0.104.1
uvicorn[standard]==0.24.0
pydantic==2.5.0
pydantic-settings==2.1.0
# 数据库
sqlalchemy==2.0.23
asyncpg==0.29.0
alembic==1.13.1
# Redis
redis==5.0.1
aioredis==2.0.1
# NATS消息队列
nats-py==2.6.0
# HTTP客户端
httpx==0.25.2
aiohttp==3.9.1
# MCP协议
mcp==1.0.0
json-rpc==1.15.0
# 工具和实用程序
python-multipart==0.0.6
python-jose[cryptography]==3.3.0
passlib[bcrypt]==1.7.4
bcrypt==4.1.2
# 监控和日志
prometheus-client==0.19.0
structlog==23.2.0
rich==13.7.0
# 配置管理
python-dotenv==1.0.0
pyyaml==6.0.1
# 类型检查和验证
typing-extensions==4.8.0
annotated-types==0.6.0
# Agent相关
langchain==0.0.350
langchain-community==0.0.5
openai==1.3.8
anthropic==0.7.8
# API文档处理
openapi-parser==1.1.0
apispec==6.3.0
apispec-webframeworks==0.5.2
# 开发工具
pytest==7.4.3
pytest-asyncio==0.21.1
black==23.11.0
flake8==6.1.0
mypy==1.7.1
services/mcp-server/schemas.py
+346
View File
@@ -0,0 +1,346 @@
"""
Pydantic schemas for API requests and responses
"""
from datetime import datetime
from typing import Any, Dict, List, Optional, Union
from pydantic import BaseModel, Field, validator
import uuid
class BaseSchema(BaseModel):
"""基础schema类"""
class Config:
from_attributes = True
json_encoders = {
datetime: lambda v: v.isoformat(),
uuid.UUID: lambda v: str(v),
}
# ========== MCP协议相关 ==========
class MCPRequest(BaseModel):
"""MCP请求模型"""
jsonrpc: str = "2.0"
id: Union[str, int] = Field(default_factory=lambda: str(uuid.uuid4()))
method: str
params: Optional[Dict[str, Any]] = None
class MCPResponse(BaseModel):
"""MCP响应模型"""
jsonrpc: str = "2.0"
id: Union[str, int]
result: Optional[Any] = None
error: Optional[Dict[str, Any]] = None
class MCPError(BaseModel):
"""MCP错误模型"""
code: int
message: str
data: Optional[Any] = None
# ========== 工具相关 ==========
class ToolParameter(BaseModel):
"""工具参数定义"""
name: str
type: str
description: Optional[str] = None
required: bool = True
default: Optional[Any] = None
enum: Optional[List[Any]] = None
class ToolDefinition(BaseSchema):
"""工具定义"""
name: str
description: str
category: Optional[str] = None
parameters: List[ToolParameter] = []
returns: Optional[Dict[str, Any]] = None
# API相关
endpoint: Optional[str] = None
method: str = "POST"
headers: Optional[Dict[str, str]] = None
# 限制信息
rate_limit: int = 100
timeout: int = 30
cost_per_call: float = 0.0
class ToolExecution(BaseModel):
"""工具执行请求"""
tool_name: str
parameters: Dict[str, Any]
timeout: Optional[int] = None
class ToolResult(BaseSchema):
"""工具执行结果"""
success: bool
result: Optional[Any] = None
error: Optional[str] = None
execution_time: float = 0.0
cost: float = 0.0
# ========== Agent相关 ==========
class AgentCreateRequest(BaseModel):
"""创建Agent请求"""
name: str = Field(..., min_length=1, max_length=100)
description: Optional[str] = None
role: str = Field(..., min_length=1, max_length=200)
goal: str = Field(..., min_length=1)
tools: List[str] = [] # 工具名称列表
config: Dict[str, Any] = {}
capabilities: List[str] = []
owner_id: Optional[uuid.UUID] = None
@validator('name')
def validate_name(cls, v):
"""验证Agent名称"""
if not v.replace('-', '').replace('_', '').isalnum():
raise ValueError('名称只能包含字母、数字、连字符和下划线')
return v
class AgentUpdateRequest(BaseModel):
"""更新Agent请求"""
name: Optional[str] = None
description: Optional[str] = None
role: Optional[str] = None
goal: Optional[str] = None
tools: Optional[List[str]] = None
config: Optional[Dict[str, Any]] = None
capabilities: Optional[List[str]] = None
class AgentCard(BaseSchema):
"""Agent卡片信息"""
id: uuid.UUID
name: str
description: Optional[str]
role: str
goal: str
tools: List[str] = []
capabilities: List[str] = []
# 端点信息
endpoints: Dict[str, str] = {}
# 状态信息
status: str = "active"
version: str = "1.0.0"
# 统计信息
total_executions: int = 0
success_rate: float = 0.0
avg_execution_time: float = 0.0
# 时间信息
created_at: datetime
updated_at: datetime
class AgentExecution(BaseModel):
"""Agent执行请求"""
method: str
params: Optional[Dict[str, Any]] = None
timeout: Optional[int] = None
session_id: Optional[str] = None
class ExecutionResult(BaseSchema):
"""执行结果"""
execution_id: str
success: bool
result: Optional[Any] = None
error: Optional[str] = None
# 性能指标
execution_time: float = 0.0
cpu_usage: float = 0.0
memory_usage: float = 0.0
network_io: float = 0.0
# 成本信息
eu_consumed: float = 0.0
cost: float = 0.0
# 时间戳
started_at: datetime
completed_at: Optional[datetime] = None
# ========== 用户相关 ==========
class UserCreate(BaseModel):
"""创建用户请求"""
username: str = Field(..., min_length=3, max_length=50)
email: str = Field(..., regex=r'^[^@]+@[^@]+\.[^@]+$')
password: str = Field(..., min_length=8)
full_name: Optional[str] = None
class UserUpdate(BaseModel):
"""更新用户请求"""
email: Optional[str] = None
full_name: Optional[str] = None
is_active: Optional[bool] = None
class UserResponse(BaseSchema):
"""用户响应"""
id: uuid.UUID
username: str
email: str
full_name: Optional[str]
is_active: bool
is_admin: bool
created_at: datetime
updated_at: datetime
class UserLogin(BaseModel):
"""用户登录请求"""
username: str
password: str
class Token(BaseModel):
"""访问令牌"""
access_token: str
token_type: str = "bearer"
expires_in: int
# ========== 会话相关 ==========
class SessionCreate(BaseModel):
"""创建会话请求"""
context: Optional[Dict[str, Any]] = None
metadata: Optional[Dict[str, Any]] = None
class SessionResponse(BaseSchema):
"""会话响应"""
id: uuid.UUID
session_id: str
status: str
context: Dict[str, Any]
metadata: Dict[str, Any]
created_at: datetime
updated_at: datetime
# ========== 计费相关 ==========
class BillingRecord(BaseSchema):
"""计费记录"""
id: uuid.UUID
execution_id: uuid.UUID
eu_consumed: float
cost: float
currency: str
# 资源详情
cpu_time: float
memory_max: float
network_io: float
storage_io: float
created_at: datetime
class BillingSummary(BaseModel):
"""计费汇总"""
user_id: uuid.UUID
period_start: datetime
period_end: datetime
total_executions: int
total_eu_consumed: float
total_cost: float
currency: str
# 按服务分类
breakdown_by_agent: Dict[str, float] = {}
breakdown_by_tool: Dict[str, float] = {}
# ========== API响应包装 ==========
class APIResponse(BaseModel):
"""API响应包装"""
success: bool
message: str = ""
data: Optional[Any] = None
timestamp: datetime = Field(default_factory=datetime.utcnow)
class PaginatedResponse(BaseModel):
"""分页响应"""
items: List[Any]
total: int
page: int
page_size: int
has_next: bool
has_prev: bool
# ========== 系统状态 ==========
class HealthCheck(BaseModel):
"""健康检查响应"""
status: str
timestamp: datetime
services: Dict[str, str]
version: str = "1.0.0"
class SystemMetrics(BaseModel):
"""系统指标"""
timestamp: datetime
# 服务指标
active_agents: int
total_executions: int
success_rate: float
avg_response_time: float
# 资源指标
cpu_usage: float
memory_usage: float
disk_usage: float
# 业务指标
daily_active_users: int
total_eu_consumed: float
total_cost: float
# ========== 错误响应 ==========
class ErrorResponse(BaseModel):
"""错误响应"""
error: str
message: str
details: Optional[Dict[str, Any]] = None
timestamp: datetime = Field(default_factory=datetime.utcnow)
class ValidationError(BaseModel):
"""验证错误"""
field: str
message: str
invalid_value: Optional[Any] = None
services/model-gateway/Dockerfile
+35
View File
@@ -0,0 +1,35 @@
FROM python:3.11-slim
# 设置工作目录
WORKDIR /app
# 安装系统依赖
RUN apt-get update && apt-get install -y \
curl \
&& rm -rf /var/lib/apt/lists/*
# 安装LiteLLM
RUN pip install --no-cache-dir \
litellm[proxy]==1.17.0 \
redis==5.0.1 \
prometheus-client==0.19.0
# 复制配置文件
COPY config/ ./config/
# 创建logs目录
RUN mkdir -p logs
# 设置环境变量
ENV LITELLM_MASTER_KEY=sk-taiji-master-key
ENV LITELLM_CONFIG_PATH=/app/config/litellm.yaml
# 暴露端口
EXPOSE 4000
# 健康检查
HEALTHCHECK --interval=30s --timeout=10s --start-period=5s --retries=3 \
CMD curl -f http://localhost:4000/health || exit 1
# 启动LiteLLM代理
CMD ["python", "-m", "litellm", "--config", "/app/config/litellm.yaml", "--port", "4000", "--host", "0.0.0.0"]
services/model-gateway/config/litellm.yaml
+296
View File
@@ -0,0 +1,296 @@
# LiteLLM 网关配置
# taiji-AI-PAD 模型治理层配置
# 基础设置
general_settings:
master_key: "sk-taiji-master-key"
database_url: "postgresql://taiji_user:taiji_pass@postgres:5432/taiji_db"
# 日志设置
set_verbose: true
json_logs: true
log_raw_request_response: false # 生产环境设为false
# 缓存设置
redis_host: "redis"
redis_port: 6379
redis_password: null
# 速率限制
max_budget: 1000.0 # 美元
budget_duration: "30d"
# 回调和监控
success_callback: ["langfuse"]
failure_callback: ["langfuse"]
# 安全设置
allowed_ips: ["127.0.0.1", "172.20.0.0/16"] # Docker网络
# 模型配置
model_list:
# OpenAI 模型组
- model_name: "gpt-3.5-turbo"
litellm_params:
model: "openai/gpt-3.5-turbo"
api_key: "os.environ/OPENAI_API_KEY"
max_tokens: 4000
temperature: 0.7
model_info:
mode: "chat"
supports_function_calling: true
supports_vision: false
max_input_tokens: 16385
max_output_tokens: 4096
input_cost_per_token: 0.0000015
output_cost_per_token: 0.000002
- model_name: "gpt-4"
litellm_params:
model: "openai/gpt-4"
api_key: "os.environ/OPENAI_API_KEY"
max_tokens: 8000
temperature: 0.7
model_info:
mode: "chat"
supports_function_calling: true
supports_vision: false
max_input_tokens: 8192
max_output_tokens: 8192
input_cost_per_token: 0.00003
output_cost_per_token: 0.00006
- model_name: "gpt-4-turbo"
litellm_params:
model: "openai/gpt-4-turbo-preview"
api_key: "os.environ/OPENAI_API_KEY"
max_tokens: 4000
temperature: 0.7
model_info:
mode: "chat"
supports_function_calling: true
supports_vision: true
max_input_tokens: 128000
max_output_tokens: 4096
input_cost_per_token: 0.00001
output_cost_per_token: 0.00003
# Anthropic 模型组
- model_name: "claude-3-haiku"
litellm_params:
model: "anthropic/claude-3-haiku-20240307"
api_key: "os.environ/ANTHROPIC_API_KEY"
max_tokens: 4000
temperature: 0.7
model_info:
mode: "chat"
supports_function_calling: true
supports_vision: true
max_input_tokens: 200000
max_output_tokens: 4096
input_cost_per_token: 0.00000025
output_cost_per_token: 0.00000125
- model_name: "claude-3-sonnet"
litellm_params:
model: "anthropic/claude-3-sonnet-20240229"
api_key: "os.environ/ANTHROPIC_API_KEY"
max_tokens: 4000
temperature: 0.7
model_info:
mode: "chat"
supports_function_calling: true
supports_vision: true
max_input_tokens: 200000
max_output_tokens: 4096
input_cost_per_token: 0.000003
output_cost_per_token: 0.000015
- model_name: "claude-3-opus"
litellm_params:
model: "anthropic/claude-3-opus-20240229"
api_key: "os.environ/ANTHROPIC_API_KEY"
max_tokens: 4000
temperature: 0.7
model_info:
mode: "chat"
supports_function_calling: true
supports_vision: true
max_input_tokens: 200000
max_output_tokens: 4096
input_cost_per_token: 0.000015
output_cost_per_token: 0.000075
# 本地/开源模型（如果可用）
- model_name: "llama-3-8b"
litellm_params:
model: "ollama/llama3"
api_base: "http://ollama:11434"
max_tokens: 2000
model_info:
mode: "chat"
supports_function_calling: false
supports_vision: false
max_input_tokens: 8192
max_output_tokens: 2048
input_cost_per_token: 0.0 # 本地模型无成本
output_cost_per_token: 0.0
# 路由器配置
router_settings:
routing_strategy: "least-busy" # 路由策略: least-busy, round-robin, latency-based
allowed_fails: 3
cooldown_time: 30
retry_after: 10
# 模型组定义
model_group_configs:
- group_name: "gpt-3.5-group"
models:
- model_name: "gpt-3.5-turbo"
weight: 1.0
- group_name: "gpt-4-group"
models:
- model_name: "gpt-4"
weight: 0.7
- model_name: "gpt-4-turbo"
weight: 0.3
- group_name: "claude-group"
models:
- model_name: "claude-3-haiku"
weight: 0.5
- model_name: "claude-3-sonnet"
weight: 0.3
- model_name: "claude-3-opus"
weight: 0.2
- group_name: "fast-models"
models:
- model_name: "gpt-3.5-turbo"
weight: 0.4
- model_name: "claude-3-haiku"
weight: 0.4
- model_name: "llama-3-8b"
weight: 0.2
- group_name: "premium-models"
models:
- model_name: "gpt-4-turbo"
weight: 0.4
- model_name: "claude-3-opus"
weight: 0.3
- model_name: "claude-3-sonnet"
weight: 0.3
# 用户和权限配置
litellm_settings:
# API密钥管理
api_keys:
- key: "sk-taiji-mcp-server"
models: ["gpt-3.5-turbo", "gpt-4", "claude-3-haiku", "claude-3-sonnet"]
max_budget: 100.0
budget_duration: "1d"
metadata:
user_id: "mcp-server"
service: "mcp-server"
- key: "sk-taiji-data-ingestion"
models: ["gpt-3.5-turbo", "claude-3-haiku", "llama-3-8b"]
max_budget: 50.0
budget_duration: "1d"
metadata:
user_id: "data-ingestion"
service: "data-ingestion"
- key: "sk-taiji-agent-dev"
models: ["gpt-3.5-group", "claude-group", "fast-models"]
max_budget: 20.0
budget_duration: "1d"
metadata:
user_id: "agent-development"
service: "agent-development"
- key: "sk-taiji-premium"
models: ["premium-models", "gpt-4-group"]
max_budget: 200.0
budget_duration: "1d"
metadata:
user_id: "premium-user"
service: "premium"
# 回调配置
callbacks:
# 成功回调
success_callback:
- callback_name: "langfuse"
callback_type: "success"
callback_vars:
langfuse_public_key: "os.environ/LANGFUSE_PUBLIC_KEY"
langfuse_secret_key: "os.environ/LANGFUSE_SECRET_KEY"
langfuse_host: "os.environ/LANGFUSE_HOST"
# 失败回调
failure_callback:
- callback_name: "langfuse"
callback_type: "failure"
callback_vars:
langfuse_public_key: "os.environ/LANGFUSE_PUBLIC_KEY"
langfuse_secret_key: "os.environ/LANGFUSE_SECRET_KEY"
langfuse_host: "os.environ/LANGFUSE_HOST"
# 监控和指标
monitoring:
prometheus_port: 4001
health_check_interval: 30
# 自定义指标
custom_metrics:
- name: "taiji_model_requests_total"
type: "counter"
description: "Total model requests"
labels: ["model", "user_id", "status"]
- name: "taiji_model_latency"
type: "histogram"
description: "Model response latency"
labels: ["model", "user_id"]
- name: "taiji_model_cost"
type: "gauge"
description: "Model cost tracking"
labels: ["model", "user_id"]
# 错误处理
error_handling:
# 重试配置
retry_policy:
max_retries: 3
retry_delay: 1.0
exponential_backoff: true
# 超时设置
timeout:
request_timeout: 60
# 回退策略
fallback:
enabled: true
fallback_models:
"gpt-4": ["gpt-4-turbo", "claude-3-sonnet"]
"claude-3-opus": ["claude-3-sonnet", "gpt-4"]
"gpt-3.5-turbo": ["claude-3-haiku", "llama-3-8b"]
# 日志配置
logging:
level: "INFO"
format: "json"
# 请求日志
log_requests: true
log_responses: false # 生产环境关闭
# 敏感信息过滤
redact_messages_in_logs: true
redact_user_api_key_info: true