agent_management/.omc/autopilot/aks-deployment-summary.md

AKS Deployment Guide - Heicode Integration

Files Updated for Deployment

1. Kubernetes Configuration

• ✅ k8s/agent-manager-configmap.yaml - Added Heicode env vars
• ✅ k8s/agent-manager-secret.yaml - Added HEICODE_SERVICE_TOKEN
• ✅ k8s/agent-manager-deployment.yaml - Updated image tag to heicode-v1
• ✅ Dockerfile - Added config/, api/, models/ directories

2. New Environment Variables

ConfigMap (k8s/agent-manager-configmap.yaml):

REDIS_URL: "redis://localhost:6379/0"
HEICODE_NEWAPI_BASE_URL: "https://code.xinghanlab.com"
LITELLM_BASE_URL: "http://litellm-service:8000"
NAMESPACE_PREFIX: "agnet"
MAX_CONCURRENT_DEPLOYMENTS_PER_USER: "10"
MAX_CONCURRENT_DEPLOYMENTS_PER_SCOPE: "50"

Secret (k8s/agent-manager-secret.yaml):

HEICODE_SERVICE_TOKEN: "heicode-prod-token-change-me"

Deployment Steps

Option 1: Automated Deployment (Recommended)

cd /Users/mac/Projects/agent-manager/tools/agent-manager
./.omc/autopilot/deploy-to-aks.sh

Option 2: Manual Deployment

Step 1: Build and Push Docker Image

cd /Users/mac/Projects/agent-manager/tools/agent-manager

# Build image
docker build -t agnettaiji.azurecr.io/ai-agents/agent-manager:heicode-v1 .

# Push to ACR
docker push agnettaiji.azurecr.io/ai-agents/agent-manager:heicode-v1

Step 2: Apply Kubernetes Resources

# Update ConfigMap
kubectl apply -f k8s/agent-manager-configmap.yaml

# Update Secret (IMPORTANT: Change HEICODE_SERVICE_TOKEN first!)
kubectl apply -f k8s/agent-manager-secret.yaml

# Deploy application
kubectl apply -f k8s/agent-manager-deployment.yaml

# Wait for rollout
kubectl rollout status deployment/agent-manager -n agent-manager

Step 3: Verify Deployment

# Check pods
kubectl get pods -n agent-manager

# Check logs
kubectl logs -n agent-manager -l app=agent-manager --tail=50

# Get service
kubectl get svc agent-manager -n agent-manager

Testing the Deployment

1. Port Forward (for local testing)

kubectl port-forward -n agent-manager svc/agent-manager 8000:8000

2. Test Health Endpoint

curl -X GET "http://localhost:8000/api/agnet/health" \
  -H "Authorization: Bearer heicode-prod-token-change-me" \
  -H "X-Correlation-Id: test-123"

Expected response:

{
  "success": true,
  "data": {
    "status": "healthy",
    "service": "agent-manager-agnet",
    "version": "1.0.0",
    "phase": "2-deployments"
  }
}

3. Test Create Deployment

curl -X POST "http://localhost:8000/api/agnet/deployments" \
  -H "Authorization: Bearer heicode-prod-token-change-me" \
  -H "Content-Type: application/json" \
  -H "X-Correlation-Id: test-create-123" \
  -H "X-User-Id: test-user" \
  -H "X-Binding-Scope: test-project" \
  -H "Idempotency-Key: test-idem-456" \
  -d '{
    "orchestration_plan": "Deploy a test agent",
    "agents": [{
      "role": "test-agent",
      "image": "agnettaiji.azurecr.io/agents/test:v1",
      "sk_sources": []
    }],
    "risk_level": "low",
    "budget": {
      "max_usd": 50.0,
      "alert_threshold_pct": 80
    },
    "billing_context": {
      "provider": "newapi",
      "default_model_id": "gpt-4",
      "allowed_model_ids": ["gpt-4", "gpt-3.5-turbo"],
      "secret_ref": "vault:secret/users/test-user/bindings/test-project/newapi-token"
    },
    "resource_grants": [],
    "metadata": 
  }'

4. Test List Deployments

curl -X GET "http://localhost:8000/api/agnet/deployments?user_id=test-user" \
  -H "Authorization: Bearer heicode-prod-token-change-me" \
  -H "X-Correlation-Id: test-list-123"

5. Verify Database

# Connect to PostgreSQL
psql "postgresql://taiji:By@123456.@taijipda.postgres.database.azure.com:5432/taijiagnet"

# Check tables
\dt

# Check deployments
SELECT deployment_id, user_id, status, risk_level, created_at FROM deployments;

# Check audit logs
SELECT audit_id, actor, action, result, occurred_at FROM audit_logs ORDER BY occurred_at DESC LIMIT 10;

Troubleshooting

Issue: Pods not starting

# Check pod status
kubectl describe pod -n agent-manager -l app=agent-manager

# Check logs
kubectl logs -n agent-manager -l app=agent-manager --tail=100

Issue: Database connection failed

• Verify DATABASE_URL in ConfigMap
• Check network connectivity from AKS to Azure PostgreSQL
• Verify firewall rules allow AKS IP range

Issue: Redis connection failed

• Redis is optional - graceful fallback if unavailable
• Check REDIS_URL in ConfigMap
• Deploy Redis if needed: kubectl apply -f k8s/redis-deployment.yaml

Issue: 401 Unauthorized

• Verify HEICODE_SERVICE_TOKEN in Secret matches client token
• Check Authorization header format: Bearer <token>

Monitoring

View Logs

# Real-time logs
kubectl logs -n agent-manager -l app=agent-manager -f

# Last 100 lines
kubectl logs -n agent-manager -l app=agent-manager --tail=100

# Specific pod
kubectl logs -n agent-manager <pod-name>

Check Metrics

# Pod resource usage
kubectl top pods -n agent-manager

# Deployment status
kubectl get deployment agent-manager -n agent-manager

Access Swagger UI

# Port forward
kubectl port-forward -n agent-manager svc/agent-manager 8000:8000

# Open browser
open http://localhost:8000/docs

Rollback

If deployment fails:

# Rollback to previous version
kubectl rollout undo deployment/agent-manager -n agent-manager

# Check rollout history
kubectl rollout history deployment/agent-manager -n agent-manager

Next Steps After Deployment

1. ✅ Verify health endpoint
2. ✅ Test create deployment
3. ✅ Test list deployments
4. ✅ Verify database records
5. ✅ Check audit logs
6. ⏳ Implement Phase 3: Observability endpoints (logs, events, metrics)
7. ⏳ Implement Phase 4: K8s integration (actual pod creation)
8. ⏳ Implement Phase 5: Vault integration

Security Notes

⚠️ IMPORTANT: Before production deployment:

1. Change HEICODE_SERVICE_TOKEN to a strong, random token
2. Coordinate token with mcp-server team
3. Enable HTTPS/TLS for external access
4. Review and restrict RBAC permissions
5. Enable network policies
6. Set up monitoring and alerting

Support

For issues or questions:

• Check logs: kubectl logs -n agent-manager -l app=agent-manager
• Review Phase 1 & 2 summaries in .omc/autopilot/
• Consult implementation plan: .omc/plans/autopilot-impl.md