Heicode rotated NewAPI service token (fingerprint 25b85d67…b1d6) per §7.16.7.
Walked through §7.7.2.1 flow:
1. Read Docs/heicode-svc-token.txt — fingerprint matched
2. kubectl create secret generic heicode-newapi (key=service-token)
3. rollout restart — completed
4. P4 smoke 4/4 with 55@55.com — balance/models/usage/logs all 200
5. Deleted local Docs/heicode-svc-token.txt
6. This §7.17 ack
Answer to Heicode's lookup question:
Their `zsbgnw@gmail.com → USER_NOT_FOUND` observation was a side-effect of
the stale token: while their new token was staged but our k8s secret still
held the previous value, every NewAPI call returned 401, and our
resolve_user_id_by_email() catches HeicodeNewAPIError and returns None —
which the P4 router translates to 404 HEICODE_USER_NOT_FOUND.
Direct re-test from inside the pod after rotation:
`GET /api/user/search?keyword=zsbgnw@gmail.com&group=` → 200, 1 item,
id=22 email=zsbgnw@gmail.com username=chenchen. Our query path is exactly
what they suggested (`/api/user/search` with empty `group`), and we filter
by email field downstream — implementation is fine.
Bonus finding: post-rotation `/balance` for zsbgnw@gmail.com returns
502 HEICODE_NEWAPI_UPSTREAM_ERROR because user 22 is super-admin and our
admin token holder (user 26) cannot read same-or-higher-level users
("No permission to access users of same or higher level"). NewAPI returns
HTTP 200 with success=false, our client correctly raises HeicodeNewAPIError.
This is an authorization policy on their side, not a bug — three options
proposed in §7.17.3 for product decision.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Found by post-fix audit pass over this session's changes:
1. Predcheck used `effective_username = (req.username or "").strip()` but the
write at the User() construction site still used `req.username` raw. If a
client sent "alice " with trailing whitespace, predcheck queried for "alice"
(clean), missed the conflict, then wrote "alice " back. Now both sites use
the same `effective_username` value — single source of truth.
2. Post-commit `verify_code` was guarded by `except Exception`, but
`asyncio.CancelledError` is a BaseException and propagates through. If the
request task is cancelled (client disconnect / pod shutdown) after DB
commit but before verify_code finishes, the verification code stays in
Redis with full 10-min TTL. Wrapped with `asyncio.shield(...)` so
verify_code completes regardless of cancellation, and an explicit
`except CancelledError: raise` preserves FastAPI's cancellation semantics
for the outer request.
Verified via smoke:
- Register with username "ws_user_$ts " (trailing spaces) → DB stores
"ws_user_$ts" (18 chars, no whitespace). Predcheck and write now agree.
- P4 透传 (balance/models/usage/logs via 55@55.com) still 4/4 — no regression.
Latent bug noticed but NOT introduced this session, deferred:
- auth.py:981 writes ResourceAllocation.resource_id=str(provider.id) for
model allocations, but channel.py:1881 queries by resource_id==model_name.
Pre-existing inconsistency means update_tenant_model_quota never finds
rows created at register time. TenantModelKey row is still updated
correctly so end-user quota is honored; only the ResourceAllocation
audit/reporting view diverges. Fix requires deciding which side is
canonical — out of scope for security hardening.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Two small follow-ups to the register hardening + Heicode P4 work:
1. heicode_client.list_user_models: the path `/api/user/{id}/models`
prescribed in §7.11.2 returns 404 `Invalid URL` on the live Heicode
NewAPI — that path is not registered on their router. Switched to
`/api/user/models` (no path segment), which Heicode binds to the
`New-Api-User: 26` admin header. End-to-end P4 smoke now 4/4 with
user 55@55.com (id=2 on Heicode): /balance /models /usage /logs.
Future: if Heicode ships an "admin-replaces-user" path, switch back
and pass the actual heicode_user_id.
2. routes/auth.register: previously line-744 SELECT only checked
req.username, but line 778 falls back to email.split("@")[0] when
blank — so two users registering with alice@foo.com and alice@bar.com
would both clear the predcheck, then the second would IntegrityError
on flush. Now predcheck uses `effective_username` matching what'll
actually be inserted.
Also append §7.15 to Heicode-对接进度与待办.md:
- 4-item agent-manager / Vault / Workload-Identity audit results
- §7.13 token rotation acknowledgement
- P4 end-to-end first-pass results
- This-session internal security hardening summary
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
A failed POST /api/auth/register returned the SQLAlchemy IntegrityError verbatim
to the caller, which included the full INSERT INTO tenant_model_keys statement
along with every bound parameter — ~50 plaintext LiteLLM API keys per failed
attempt. Same pattern was reproduced in 3 channel.py endpoints that wrap
LiteLLM key INSERTs.
Changes:
- channel.py: assign_resources_to_tenant / assign_model_to_tenant /
update_tenant_model_quota — log full exc_info, return a typed
{code, message} error instead of f"...{str(e)}". 6 leakage points sealed.
- email_verification.py: add peek_verification_code() — checks a code
without burning it. Lets the register handler verify *before* the
multi-step transaction so a downstream failure doesn't waste the user's
one-shot code.
- scripts/cleanup_orphan_litellm_keys.py: one-shot orphan key reaper.
Scans LiteLLM /key/list by metadata.tenant_id (plus a manual list of
the 8 publicly-leaked sk- prefixes from the original incident).
Used to nuke 16 orphan keys for tenant fab9dc27-… on 2026-05-12.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
chenchenandClaude Opus 4.8