heicode-win/heicode/docs/deploy/new-api-rebuild-deploy-runbook.md

# NewAPI / Heicode Manager Rebuild and Deploy Runbook

Date: 2026-05-03  
Scope: rebuild and deploy the `heicode` Manager service on an Azure VM while keeping NewAPI as an independent model gateway.  
Security rule: never put real passwords, tokens, SSH keys, Redis keys, PostgreSQL DSNs, NewAPI keys, or cloud credentials in Git, Markdown, shell history, tickets, or logs. Use environment variables, `.env` on the VM, Secret Store references, or `REDACTED` placeholders only.

## 1. Current repo deployment surfaces

| Surface | Purpose | Notes |
|---|---|---|
| `heicode/Dockerfile` | Production image build | Builds default and classic web assets, then compiles the Go binary. |
| `heicode/docker-compose.azure-vm.yml` | Azure VM Manager service | Runs only `heicode`; PostgreSQL and Redis are expected to be managed Azure services. |
| `heicode/docker-compose.override.yml` | Local-source image override | Builds `heicode-manager:local` from the checked-out repo. Keep it in the compose file list when deploying this repo state. |
| `heicode/bin/azure_vm_deploy.sh` | SSH deployment helper | Uses env vars only; can fast-forward a remote branch, performs remote compose up, health gate, and rollback pointer capture. |
| `heicode/bin/acceptance_agnet_local.sh` | Local Agnet control-plane smoke/acceptance probe | Requires an admin session cookie supplied via env; does not store credentials. |
| `heicode/.env.example` | Env-var reference | Placeholder-only reference; production `.env` must stay on the VM and out of Git. |

## 2. Required VM inputs

Prepare these values outside the repo, for example in a secret manager, CI secret store, or an uncommitted VM-local `.env`:

```text
VM_HOST=<azure-vm-host-or-ip>
VM_USER=<ssh-user>
REMOTE_DIR=/opt/heicode/heicode
GIT_REF=main
HEICODE_DATA_ROOT=/var/lib/heicode/heicode
SQL_DSN=postgresql://<user>:REDACTED@<azure-postgres-host>:5432/heicode?sslmode=require
REDIS_CONN_STRING=rediss://:REDACTED@<azure-redis-host>:6380
SESSION_SECRET=REDACTED
HEICODE_AUTH_BASE_URL=https://apimtaiji.azure-api.net/api/mcp
HEICODE_ROOT_EMAILS=<comma-separated-admin-emails-if-needed>
HEICODE_ADMIN_EMAILS=<comma-separated-admin-emails-if-needed>
```

Do not paste real values into this document, commits, chat, or terminal transcripts that will be saved.

## 3. Pre-deploy checklist

Run locally from the repo root unless stated otherwise:

1. Confirm mainline docs boundary:
   - `docs/heicode.md`
   - `docs/plan.md`
   - `docs/README.md`
2. Confirm worktree state:
   - `git status --short`
3. Validate compose syntax without exposing secrets:
   - `cd heicode`
   - `cat > /tmp/heicode.deploy.env <<'EOF_ENV'`
   - `HEICODE_DATA_ROOT=/tmp/heicode-data`
   - `SQL_DSN=REDACTED`
   - `REDIS_CONN_STRING=REDACTED`
   - `SESSION_SECRET=REDACTED`
   - `EOF_ENV`
   - `ENV_FILE=/tmp/heicode.deploy.env IMAGE_TAG=heicode-manager:local docker compose -f docker-compose.azure-vm.yml -f docker-compose.override.yml --env-file /tmp/heicode.deploy.env config --quiet`
   - `rm -f /tmp/heicode.deploy.env`
4. Build verification options:
   - Preferred full image check: `cd heicode && docker compose -f docker-compose.azure-vm.yml -f docker-compose.override.yml build heicode`
   - Backend slice check when full image build is too slow: `cd heicode && go test -count=1 ./controller ./model ./router`
5. Confirm no secrets are staged:
   - `git diff --cached -- . ':!*.jpg' ':!*.png'`
   - Search for known real secret fragments only locally; never copy matches into reports.

## 4. VM preparation checklist

Run on the Azure VM:

```bash
export HEICODE_DATA_ROOT=/var/lib/heicode/heicode
sudo mkdir -p "$HEICODE_DATA_ROOT"/data "$HEICODE_DATA_ROOT"/logs
sudo chown -R "$USER":"$USER" "$HEICODE_DATA_ROOT"
cd /opt/heicode/heicode
```

Create or update VM-local `.env` in `REMOTE_DIR`; keep it untracked:

```bash
cat > .env <<'EOF_ENV'
HEICODE_DATA_ROOT=/var/lib/heicode/heicode
SQL_DSN=postgresql://<user>:REDACTED@<azure-postgres-host>:5432/heicode?sslmode=require
REDIS_CONN_STRING=rediss://:REDACTED@<azure-redis-host>:6380
SESSION_SECRET=REDACTED
HEICODE_AUTH_BASE_URL=https://apimtaiji.azure-api.net/api/mcp
HEICODE_ROOT_EMAILS=
HEICODE_ADMIN_EMAILS=
EOF_ENV
chmod 600 .env
```

Replace `REDACTED` placeholders through a secure VM-local editor or secret injection mechanism only.

## 5. Rebuild and deploy procedure

From the local control machine, after the VM repo has the intended commit:

```bash
cd heicode
VM_HOST=<azure-vm-host-or-ip> \
VM_USER=<ssh-user> \
REMOTE_DIR=/opt/heicode/heicode \
GIT_REF=main \
HEALTH_URL=http://127.0.0.1:3000/api/status \
./bin/azure_vm_deploy.sh
```

The script will:

1. SSH to the VM.
2. Verify the remote repo, compose CLI, compose file, and VM-local env file exist.
3. If `GIT_REF` is set, fast-forward the remote checkout from `origin/<GIT_REF>`.
4. Record the currently running container image in `.last_success_image` when available.
5. Run `ENV_FILE=.env docker compose -f docker-compose.azure-vm.yml -f docker-compose.override.yml --env-file .env up -d --build --force-recreate heicode`.
6. Poll `/api/status` through the VM-local health URL.
7. Update `.last_success_image` only after the health gate passes.

## 5.1 Agnet operator handoff

When Agnet is the executor, Manager should create a high-risk `newapi-rebuild-deploy` deployment using `docs/integration/agnet-platform-request-contract.md` and pass only references:

| Field | Required reference |
|---|---|
| `resource_grants[].secret_ref` | Secret Store reference for SSH access and runtime env access; never plaintext. |
| `resource_grants[].metadata.host_ref` | `env://NEWAPI_VM_HOST` or equivalent platform secret/env ref. |
| `resource_grants[].constraints.rollback_command_ref` | `runbook://newapi/rollback` or this runbook section. |
| `orchestration_plan.constraints.healthcheck_url_ref` | `env://NEWAPI_HEALTHCHECK_URL`, expected to resolve to the VM-local `/api/status` probe. |
| `orchestration_plan.metadata.commit` | Intended Git commit or branch to deploy, such as `origin/main` after push. |

Minimum evidence Agnet must return before Manager marks the operation deployed:

1. Remote commit after fetch/pull.
2. `docker compose ... ps` status for `heicode`.
3. Health check response proving `"success":true`.
4. Redacted log tail or log digest.
5. Rollback pointer (`.last_success_image`) update or explicit note that no previous image existed.

If any item is missing, Manager must keep the deployment in `running`, `failed`, or `pending_operator_review`; it must not report production deployment as completed.

## 6. Post-deploy verification checklist

Run on the VM or through secure operator access:

```bash
cd /opt/heicode/heicode
ENV_FILE=.env docker compose -f docker-compose.azure-vm.yml -f docker-compose.override.yml --env-file .env ps
ENV_FILE=.env docker compose -f docker-compose.azure-vm.yml -f docker-compose.override.yml --env-file .env logs --tail=100 heicode
curl -fsS http://127.0.0.1:3000/api/status
```

Expected health response includes `"success":true`.

Optional Manager / Agnet smoke probe after obtaining a safe admin session cookie without logging it:

```bash
AUTH_COOKIE='REDACTED_SESSION_COOKIE' \
BASE_URL='http://127.0.0.1:3000' \
TENANT_ID='tenant_smoke' \
PROJECT_ID='project_smoke' \
./bin/acceptance_agnet_local.sh
```

Do not commit or report the real cookie.

## 7. Rollback checklist

If the health gate fails or post-deploy checks fail:

1. Keep the failed container logs for local VM inspection only; redact before sharing.
2. Inspect `.last_success_image` on the VM.
3. Re-run compose with the last known good image or revert the repo commit, then:

```bash
ENV_FILE=.env docker compose -f docker-compose.azure-vm.yml --env-file .env up -d --force-recreate heicode
curl -fsS http://127.0.0.1:3000/api/status
```

4. If database migrations were applied by the app, do not manually mutate production data without an explicit backup and approval.

## 8. Known risks / open items

- `go test ./...` can be blocked by missing embedded frontend build artifacts or pre-existing relay tests; use focused checks for the changed slice and a full Docker image build before production.
- Production PostgreSQL and Redis are external managed services; the compose file intentionally does not start local database/cache containers for Azure VM deployment.
- NewAPI remains an independent model gateway and should not be exposed as a normal SaaS user backend.
- Long-lived secrets must move toward Secret Store / Secret Broker; current VM `.env` usage is an operational bridge, not the final P2 target.