xiaohei/model_garden
1
0
Fork 0
mirror of https://github.com/GoogleCloudPlatform/vertex-ai-samples.git synced 2026-09-26 14:42:04 +00:00
Code Issues Packages Projects Releases Wiki Activity

Migrate QWEN3 notebook to use Model Garden SDK

Browse Source 
PiperOrigin-RevId: 800346334
This commit is contained in:
Vertex MG Team
2025-08-28 00:57:59 -07:00
committed by Copybara-Service
parent deaa1ccf2b
commit b77d51b58b
1 changed files with 244 additions and 405 deletions
Download Patch File Download Diff File Expand all files Collapse all files
notebooks/community/model_garden/model_garden_pytorch_qwen3_deployment.ipynb
+244 -405
View File
@@ -59,34 +59,43 @@
"source": [
"## Overview\n",
"\n",
"This notebook demonstrates serving Qwen3 models with [SGLang](https://github.com/sgl-project/sglang). [Qwen3](https://huggingface.co/collections/Qwen/qwen3-67dd247413f0e2e4f653967f) is the latest generation of large language models in Qwen series, offering a comprehensive suite of dense and mixture-of-experts (MoE) models. Built upon extensive training, Qwen3 delivers groundbreaking advancements in reasoning, instruction-following, agent capabilities, and multilingual support\n",
"This notebook demonstrates how to deploy a **Qwen 3** open model on Google Cloud Vertex AI.\n",
"\n",
"### Objectives\n",
"\n",
"### Objective\n",
"- Deploy Qwen 3 using containerized backends like [vLLM](https://github.com/vllm-project/vllm) on GPU.\n",
"- Use the deployed model to serve chat completion requests for both text and multimodal inputs.\n",
"\n",
"- Deploy Qwen3 with SGLang on GPU using single-host serving, and [Spot VMs](https://cloud.google.com/compute/docs/instances/spot) (Optional).\n",
"### File a Bug\n",
"\n",
"### File a bug\n",
"\n",
"File a bug on [GitHub](https://github.com/GoogleCloudPlatform/vertex-ai-samples/issues/new) if you encounter any issue with the notebook.\n",
"If you encounter issues with this notebook, report them on [GitHub](https://github.com/GoogleCloudPlatform/vertex-ai-samples/issues/new).\n",
"\n",
"### Costs\n",
"\n",
"This tutorial uses billable components of Google Cloud:\n",
"\n",
"* Vertex AI\n",
"* Cloud Storage\n",
"- Vertex AI\n",
"- Cloud Storage\n",
"\n",
"Learn about [Vertex AI pricing](https://cloud.google.com/vertex-ai/pricing), [Cloud Storage pricing](https://cloud.google.com/storage/pricing), and use the [Pricing Calculator](https://cloud.google.com/products/calculator/) to generate a cost estimate based on your projected usage."
"Refer to the [Vertex AI pricing](https://cloud.google.com/vertex-ai/pricing) and [Cloud Storage pricing](https://cloud.google.com/storage/pricing) pages for more information. Use the [Pricing Calculator](https://cloud.google.com/products/calculator/) to estimate your projected costs."
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "264c07757582"
"id": "jeYw-Czg-DFy"
},
"source": [
"## Before you begin"
"## Get Started"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "KgyhGvEzBDkj"
},
"source": [
"### Install Vertex AI SDK and other required packages"
]
},
{
@@ -94,17 +103,22 @@
"execution_count": null,
"metadata": {
"cellView": "form",
"id": "ax7zWynUDcjk"
"id": "iCacdLqG-IsH"
},
"outputs": [],
"source": [
"# @title Request for quota\n",
"%pip install --upgrade --force-reinstall --quiet 'google-cloud-aiplatform>=1.106.0' 'openai' 'google-auth' 'requests'"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "HUKCrpBy-3yf"
},
"source": [
"### Authenticate the Notebook Environment (Colab only)\n",
"\n",
"# @markdown To deploy the largest variant of the Qwen3 models, you need 1 host of 8 x H100 machine. Check that you have sufficient quota:\n",
"# @markdown - For Spot VM quota, check [`CustomModelServingPreemptibleH100GPUsPerProjectPerRegion`](https://console.cloud.google.com/iam-admin/quotas?metric=aiplatform.googleapis.com%2Fcustom_model_serving_preemptible_nvidia_h100_gpus).\n",
"# @markdown - For regular VM quota, check [`CustomModelServingH100GPUsPerProjectPerRegion`](https://console.cloud.google.com/iam-admin/quotas?metric=aiplatform.googleapis.com%2Fcustom_model_serving_nvidia_h100_gpus).\n",
"#\n",
"# @markdown If you don't have sufficient quota, request for quota following the instructions at [\"Request a higher quota\"](https://cloud.google.com/docs/quota/view-manage#requesting_higher_quota)."
"If you're running this notebook in Google Colab, run the following cell to authenticate."
]
},
{
@@ -112,80 +126,146 @@
"execution_count": null,
"metadata": {
"cellView": "form",
"id": "YXFGIp1l-qtT"
"id": "JXwCT1kn-3Gu"
},
"outputs": [],
"source": [
"# @title Setup Google Cloud project\n",
"import sys\n",
"\n",
"# @markdown 1. [Make sure that billing is enabled for your project](https://cloud.google.com/billing/docs/how-to/modify-project).\n",
"if \"google.colab\" in sys.modules:\n",
" from google.colab import auth\n",
"\n",
"# @markdown 2. **[Optional]** Set region. If not set, the region will be set automatically according to Colab Enterprise environment.\n",
" auth.authenticate_user()"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "AcW2nwB8-7yC"
},
"source": [
"### Set Google Cloud Project Information\n",
"\n",
"REGION = \"\" # @param {type:\"string\"}\n",
"To get started with Vertex AI, ensure you have an existing Google Cloud project and that the [Vertex AI API is enabled](https://console.cloud.google.com/flows/enableapi?apiid=aiplatform.googleapis.com).\n",
"\n",
"# @markdown 3. If you want to run predictions with A100 80GB or H100 GPUs, we recommend using the regions listed below. **NOTE:** Make sure you have associated quota in selected regions. Click the links to see your current quota for each GPU type: [Nvidia A100 80GB](https://console.cloud.google.com/iam-admin/quotas?metric=aiplatform.googleapis.com%2Fcustom_model_serving_nvidia_a100_80gb_gpus), [Nvidia H100 80GB](https://console.cloud.google.com/iam-admin/quotas?metric=aiplatform.googleapis.com%2Fcustom_model_serving_nvidia_h100_gpus). You can request for quota following the instructions at [\"Request a higher quota\"](https://cloud.google.com/docs/quota/view-manage#requesting_higher_quota).\n",
"\n",
"# @markdown > | Machine Type | Accelerator Type | Recommended Regions |\n",
"# @markdown | ----------- | ----------- | ----------- |\n",
"# @markdown | a2-ultragpu-1g | 1 NVIDIA_A100_80GB | us-central1, us-east4, europe-west4, asia-southeast1, us-east4 |\n",
"# @markdown | a3-highgpu-2g | 2 NVIDIA_H100_80GB | us-west1, asia-southeast1, europe-west4 |\n",
"# @markdown | a3-highgpu-4g | 4 NVIDIA_H100_80GB | us-west1, asia-southeast1, europe-west4 |\n",
"# @markdown | a3-highgpu-8g | 8 NVIDIA_H100_80GB | us-central1, europe-west4, us-west1, asia-southeast1 |\n",
"\n",
"# Upgrade Vertex AI SDK.\n",
"! pip3 install --upgrade --quiet 'google-cloud-aiplatform==1.103.0'\n",
"\n",
"# Import the necessary packages\n",
"import importlib\n",
"See the guide on [setting up your project and development environment](https://cloud.google.com/vertex-ai/docs/start/cloud-environment). Also confirm that [billing is enabled](https://cloud.google.com/billing/docs/how-to/modify-project).\n"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"cellView": "form",
"id": "eIVLp0oE--k-"
},
"outputs": [],
"source": [
"# Use the environment variable if the user doesn't provide Project ID.\n",
"import os\n",
"import time\n",
"from typing import Tuple\n",
"\n",
"import requests\n",
"from google import auth\n",
"from google.cloud import aiplatform\n",
"\n",
"# Upgrade Vertex AI SDK.\n",
"if os.environ.get(\"VERTEX_PRODUCT\") != \"COLAB_ENTERPRISE\":\n",
" ! pip install --upgrade tensorflow\n",
"! git clone https://github.com/GoogleCloudPlatform/vertex-ai-samples.git\n",
"\n",
"common_util = importlib.import_module(\n",
" \"vertex-ai-samples.notebooks.community.model_garden.docker_source_codes.notebook_util.common_util\"\n",
")\n",
"\n",
"LABEL = \"sglang_gpu\"\n",
"models, endpoints = {}, {}\n",
"\n",
"# Get the default cloud project id.\n",
"PROJECT_ID = os.environ[\"GOOGLE_CLOUD_PROJECT\"]\n",
"\n",
"# Get the default region for launching jobs.\n",
"if not REGION:\n",
" REGION = os.environ[\"GOOGLE_CLOUD_REGION\"]\n",
"\n",
"# Initialize Vertex AI API.\n",
"print(\"Initializing Vertex AI API.\")\n",
"aiplatform.init(project=PROJECT_ID, location=REGION)\n",
"\n",
"! gcloud config set project $PROJECT_ID\n",
"\n",
"import vertexai\n",
"\n",
"vertexai.init(\n",
" project=PROJECT_ID,\n",
" location=REGION,\n",
"PROJECT_ID = \"[your-project-id]\" # @param {type: \"string\", placeholder: \"[your-project-id]\", isTemplate: true}\n",
"\n",
"if not PROJECT_ID or PROJECT_ID == \"[your-project-id]\":\n",
" PROJECT_ID = str(os.environ.get(\"GOOGLE_CLOUD_PROJECT\"))\n",
"\n",
"REGION = os.environ.get(\"GOOGLE_CLOUD_REGION\", \"us-central1\")\n",
"\n",
"vertexai.init(project=PROJECT_ID, location=REGION)"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "Q0CXrvcZH_aw"
},
"source": [
"### Import libraries"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"cellView": "form",
"id": "3G2UXB82ICs6"
},
"outputs": [],
"source": [
"from vertexai import model_garden"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "upYRiGtP_-iN"
},
"source": [
"## Deploy model"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "H2WC_0hXDVXc"
},
"source": [
"### Choose model variant"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "u41zbNa2EoFq"
},
"source": [
"You can proceed with the default model variant or select a different one."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"cellView": "form",
"id": "-fgC4NLSDkF7"
},
"outputs": [],
"source": [
"model_version = \"qwen3-4b\" # @param [\"qwen3-0.6b\", \"qwen3-0.6b-base\", \"qwen3-0.6b-fp8\", \"qwen3-1.7b\", \"qwen3-1.7b-base\", \"qwen3-1.7b-fp8\", \"qwen3-14b\", \"qwen3-14b-base\", \"qwen3-14b-fp8\", \"qwen3-235b-a22b\", \"qwen3-235b-a22b-fp8\", \"qwen3-235b-a22b-instruct-2507\", \"qwen3-235b-a22b-instruct-2507-fp8\", \"qwen3-235b-a22b-thinking-2507\", \"qwen3-235b-a22b-thinking-2507-fp8\", \"qwen3-30b-a3b\", \"qwen3-30b-a3b-base\", \"qwen3-30b-a3b-fp8\", \"qwen3-30b-a3b-instruct-2507\", \"qwen3-30b-a3b-instruct-2507-fp8\", \"qwen3-30b-a3b-thinking-2507\", \"qwen3-30b-a3b-thinking-2507-fp8\", \"qwen3-32b\", \"qwen3-32b-fp8\", \"qwen3-4b\", \"qwen3-4b-base\", \"qwen3-4b-fp8\", \"qwen3-4b-instruct-2507\", \"qwen3-4b-instruct-2507-fp8\", \"qwen3-4b-thinking-2507\", \"qwen3-4b-thinking-2507-fp8\", \"qwen3-8b\", \"qwen3-8b-base\", \"qwen3-8b-fp8\"] {isTemplate:true}\n",
"MODEL_NAME = f\"qwen/qwen3@{model_version}\""
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "VRnUgU8LF3_i"
},
"source": [
"To see all deployable model variants available in Model Garden, use:"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"cellView": "form",
"id": "-QLd-wshF6sB"
},
"outputs": [],
"source": [
"all_model_versions = model_garden.list_deployable_models(\n",
" model_filter=\"qwen3\", list_hf_models=False\n",
")"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "3zJJDmldn7rw"
"id": "N0UeFHa2GO63"
},
"source": [
"## Deploy Qwen3 with SGLang"
"Once you've selected a model variant, initialize it:"
]
},
{
@@ -193,73 +273,22 @@
"execution_count": null,
"metadata": {
"cellView": "form",
"id": "_3Swj3pxn7rw"
"id": "GZiV3trBBcA3"
},
"outputs": [],
"source": [
"# @title Select the model variants\n",
"model = model_garden.OpenModel(MODEL_NAME)"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "-0cL378wFlvf"
},
"source": [
"### Check the Deployment Configuration\n",
"\n",
"# @markdown Set the model to deploy.\n",
"\n",
"base_model_name = \"Qwen3-235B-A22B-Instruct-2507\" # @param [\"Qwen3-235B-A22B-Instruct-2507\", \"Qwen3-235B-A22B-Instruct-2507-FP8\", \"Qwen3-235B-A22B-Thinking-2507\", \"Qwen3-235B-A22B-Thinking-2507-FP8\", \"Qwen3-235B-A22B\", \"Qwen3-235B-A22B-FP8\", \"Qwen3-30B-A3B-Instruct-2507\", \"Qwen3-30B-A3B-Instruct-2507-FP8\", \"Qwen3-30B-A3B-Thinking-2507\", \"Qwen3-30B-A3B-Thinking-2507-FP8\", \"Qwen3-30B-A3B\", \"Qwen3-30B-A3B-Base\", \"Qwen3-30B-A3B-FP8\", \"Qwen3-32B\", \"Qwen3-32B-FP8\", \"Qwen3-14B\", \"Qwen3-14B-Base\", \"Qwen3-14B-FP8\", \"Qwen3-8B\", \"Qwen3-8B-Base\", \"Qwen3-8B-FP8\", \"Qwen3-4B-Instruct-2507\", \"Qwen3-4B-Thinking-2507\", \"Qwen3-4B-Instruct-2507-FP8\", \"Qwen3-4B-Thinking-2507-FP8\", \"Qwen3-4B\", \"Qwen3-4B-Base\", \"Qwen3-4B-FP8\", \"Qwen3-1.7B\", \"Qwen3-1.7B-Base\", \"Qwen3-1.7B-FP8\", \"Qwen3-0.6B\", \"Qwen3-0.6B-Base\", \"Qwen3-0.6B-FP8\"] {isTemplate:true}\n",
"model_id = \"Qwen/\" + base_model_name\n",
"hf_model_id = model_id\n",
"\n",
"# The pre-built serving docker images.\n",
"SGLANG_DOCKER_URI = \"us-docker.pkg.dev/deeplearning-platform-release/vertex-model-garden/sglang-serve.cu124.0-4.ubuntu2204.py310:model-garden.sglang-0-4-release_20250718.00_p0\"\n",
"\n",
"# @markdown Choose whether to use a [Spot VM](https://cloud.google.com/compute/docs/instances/spot) for the deployment.\n",
"is_spot = False # @param {type:\"boolean\"}\n",
"\n",
"# @markdown Set use_dedicated_endpoint to False if you don't want to use [dedicated endpoint](https://cloud.google.com/vertex-ai/docs/general/deployment#create-dedicated-endpoint). Note that [dedicated endpoint does not support VPC Service Controls](https://cloud.google.com/vertex-ai/docs/predictions/choose-endpoint-type), uncheck the box if you are using VPC-SC.\n",
"use_dedicated_endpoint = True # @param {type:\"boolean\"}\n",
"\n",
"# @markdown Find Vertex AI prediction supported accelerators and regions at https://cloud.google.com/vertex-ai/docs/predictions/configure-compute.\n",
"accelerator_type = \"NVIDIA_H100_80GB\" # @param [\"NVIDIA_H100_80GB\", \"NVIDIA_L4\"] {isTemplate:true}\n",
"\n",
"PUBLISHER_MODEL_NAME = f\"publishers/qwen/models/qwen3@{base_model_name.lower()}\"\n",
"\n",
"if accelerator_type == \"NVIDIA_H100_80GB\":\n",
" if base_model_name in [\"Qwen3-235B-A22B\", \"Qwen3-235B-A22B-Instruct-2507\", \"Qwen3-235B-A22B-Thinking-2507\"]:\n",
" machine_type = \"a3-highgpu-8g\"\n",
" accelerator_count = 8\n",
" model_id = f\"gs://vertex-model-garden-restricted-us/qwen3/{base_model_name}\"\n",
" elif base_model_name in [\"Qwen3-235B-A22B-FP8\", \"Qwen3-235B-A22B-Instruct-2507-FP8\", \"Qwen3-235B-A22B-Thinking-2507-FP8\"]:\n",
" machine_type = \"a3-highgpu-4g\"\n",
" accelerator_count = 4\n",
" elif base_model_name in [\"Qwen3-30B-A3B-Instruct-2507\", \"Qwen3-30B-A3B-Thinking-2507\", \"Qwen3-30B-A3B\", \"Qwen3-30B-A3B-Base\", \"Qwen3-32B\"]\n",
" machine_type = \"a3-highgpu-2g\"\n",
" accelerator_count = 2\n",
" else:\n",
" machine_type = \"a3-highgpu-1g\"\n",
" accelerator_count = 1\n",
"elif accelerator_type == \"NVIDIA_L4\":\n",
" if base_model_name in [\"Qwen3-235B-A22B\", \"Qwen3-235B-A22B-FP8\", \"Qwen3-235B-A22B-Instruct-2507\", \"Qwen3-235B-A22B-Instruct-2507-FP8\", \"Qwen3-235B-A22B-Thinking-2507\", \"Qwen3-235B-A22B-Thinking-2507-FP8\",]:\n",
" raise ValueError(\"L4s are insufficient to serve Qwen3-235B-A22B models.\")\n",
" elif base_model_name in [\"Qwen3-30B-A3B-Instruct-2507\", \"Qwen3-30B-A3B-Thinking-2507\", \"Qwen3-30B-A3B\", \"Qwen3-30B-A3B-Base\", \"Qwen3-32B\"]:\n",
" machine_type = \"g2-standard-48\"\n",
" accelerator_count = 4\n",
" elif base_model_name in [\"Qwen3-30B-A3B-Thinking-2507-FP8\", \"Qwen3-30B-A3B-Instruct-2507-FP8\", \"Qwen3-14B\", \"Qwen3-14B-Base\", \"Qwen3-30B-A3B-FP8\", \"Qwen3-32B-FP8\"]:\n",
" machine_type = \"g2-standard-24\"\n",
" accelerator_count = 2\n",
" elif base_model_name in [\"Qwen3-8B\", \"Qwen3-8B-Base\",\"Qwen3-4B-Instruct-2507\", \"Qwen3-4B-Thinking-2507\", \"Qwen3-4B-Instruct-2507-FP8\", \"Qwen3-4B-Thinking-2507-FP8\", \"Qwen3-4B\", \"Qwen3-4B-Base\", \"Qwen3-1.7B\", \"Qwen3-1.7B-Base\", \"Qwen3-0.6B\", \"Qwen3-0.6B-Base\", \"Qwen3-14B-FP8\", \"Qwen3-8B-FP8\", \"Qwen3-4B-FP8\", \"Qwen3-1.7B-FP8\", \"Qwen3-0.6B-FP8\"]:\n",
" machine_type = \"g2-standard-12\"\n",
" accelerator_count = 1\n",
" else:\n",
" raise ValueError(f\"Recommended GPU setting not found for: {base_model_name}.\")\n",
"else:\n",
" raise ValueError(f\"Recommended GPU setting not found for: {base_model_name}.\")\n",
"\n",
"common_util.check_quota(\n",
" project_id=PROJECT_ID,\n",
" region=REGION,\n",
" accelerator_type=accelerator_type,\n",
" accelerator_count=accelerator_count,\n",
" is_for_training=False,\n",
" is_spot=is_spot,\n",
")\n",
"\n",
"# @markdown Click \"Show Code\" to see more details."
"Use the `list_deploy_options()` method to view the verified deployment configurations for your selected model. This helps ensure you have sufficient resources (e.g., GPU quota) available to deploy it."
]
},
{
@@ -267,29 +296,61 @@
"execution_count": null,
"metadata": {
"cellView": "form",
"id": "omW0LaC8wWz5"
"id": "zm73g7vFFm9N"
},
"outputs": [],
"source": [
"# @title [Option 1] Deploy with Model Garden SDK\n",
"# @markdown Deploy with Gen AI model-centric SDK. This section uploads the prebuilt model to Model Registry and deploys it to a Vertex AI Endpoint. It takes 15 minutes to 1 hour to finish depending on the size of the model. See [use open models with Vertex AI](https://cloud.google.com/vertex-ai/generative-ai/docs/open-models/use-open-models) for documentation on other use cases.\n",
"deploy_request_timeout = 1800 # 30 minutes\n",
"from vertexai import model_garden\n",
"deploy_options = model.list_deploy_options(concise=True)\n",
"print(deploy_options)"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "WjV499VsGwrD"
},
"source": [
"### Deploy the Model\n",
"\n",
"model = model_garden.OpenModel(PUBLISHER_MODEL_NAME)\n",
"endpoints[LABEL] = model.deploy(\n",
" machine_type=machine_type,\n",
" accelerator_type=accelerator_type,\n",
" accelerator_count=accelerator_count,\n",
"Now that you’ve reviewed the deployment options, use the `deploy()` method to serve the selected open model to a Vertex AI endpoint. Deployment time may vary depending on the model size and infrastructure requirements.\n",
"\n",
"> **Note**: If the model requires accepting a license agreement (EULA), set the `accept_eula=True` flag in the deploy call. Set `use_dedicated_endpoint` to False if you don't want to use [dedicated endpoint](https://cloud.google.com/vertex-ai/docs/general/deployment#create-dedicated-endpoint)."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"cellView": "form",
"id": "wX1itVTvXdEP"
},
"outputs": [],
"source": [
"use_dedicated_endpoint = True"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"cellView": "form",
"id": "MRmPFEPoGzsB"
},
"outputs": [],
"source": [
"endpoint = model.deploy(\n",
" accept_eula=True,\n",
" use_dedicated_endpoint=use_dedicated_endpoint,\n",
" spot=is_spot,\n",
" deploy_request_timeout=deploy_request_timeout,\n",
" accept_eula=False,\n",
")\n",
"\n",
"endpoint = endpoints[LABEL]\n",
"\n",
"# @markdown Click \"Show Code\" to see more details."
")"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "PHBtn8DQp-ID"
},
"source": [
"Alternatively, you can select one of the verified deployment configurations listed above."
]
},
{
@@ -297,249 +358,33 @@
"execution_count": null,
"metadata": {
"cellView": "form",
"id": "3m-tDxgawYhU"
"id": "ADsJG8JYqI6c"
},
"outputs": [],
"source": [
"# @title [Option 2] Deploy with customized configs\n",
"\n",
"# @markdown This section uploads Qwen3 models to Model Registry and deploys them to a Vertex Prediction Endpoint. It takes ~1 hour to finish.\n",
"\n",
"# @markdown It's recommended to use the region selected by the deployment button on the model card. If the deployment button is not available, it's recommended to stay with the default region of the notebook.\n",
"\n",
"\n",
"def poll_operation(op_name: str) -> bool: # noqa: F811\n",
" creds, _ = auth.default()\n",
" auth_req = auth.transport.requests.Request()\n",
" creds.refresh(auth_req)\n",
" headers = {\n",
" \"Authorization\": f\"Bearer {creds.token}\",\n",
" }\n",
" get_resp = requests.get(\n",
" f\"https://{REGION}-aiplatform.googleapis.com/ui/{op_name}\",\n",
" headers=headers,\n",
" )\n",
" opjs = get_resp.json()\n",
" if \"error\" in opjs:\n",
" raise ValueError(f\"Operation failed: {opjs['error']}\")\n",
" return opjs.get(\"done\", False)\n",
"\n",
"\n",
"def poll_and_wait(op_name: str, total_wait: int, interval: int = 60): # noqa: F811\n",
" waited = 0\n",
" while not poll_operation(op_name):\n",
" if waited > total_wait:\n",
" raise TimeoutError(\"Operation timed out\")\n",
" print(\n",
" f\"\\rStill waiting for operation... Waited time in second: {waited:<6}\",\n",
" end=\"\",\n",
" flush=True,\n",
" )\n",
" waited += interval\n",
" time.sleep(interval)\n",
"\n",
"\n",
"def deploy_model_sglang_multihost(\n",
" model_name: str,\n",
" model_id: str,\n",
" publisher: str,\n",
" publisher_model_id: str,\n",
" service_account: str = \"\",\n",
" base_model_id: str = \"\",\n",
" machine_type: str = \"g2-standard-8\",\n",
" accelerator_type: str = \"NVIDIA_L4\",\n",
" accelerator_count: int = 1,\n",
" multihost_gpu_node_count: int = 1,\n",
" gpu_memory_utilization: float | None = None,\n",
" context_length: int | None = None,\n",
" dtype: str | None = None,\n",
" quantization: str | None = None,\n",
" enable_trust_remote_code: bool = False,\n",
" enable_torch_compile: bool = False,\n",
" torch_compile_max_bs: int | None = None,\n",
" attention_backend: str = \"\",\n",
" enable_flashinfer_mla: bool = False,\n",
" disable_cuda_graph: bool = False,\n",
" speculative_algorithm: str | None = None,\n",
" speculative_draft_model_path: str = \"\",\n",
" speculative_num_steps: int = 3,\n",
" speculative_eagle_topk: int = 1,\n",
" speculative_num_draft_tokens: int = 4,\n",
" enable_jit_deepgemm: bool = False,\n",
" enable_dp_attention: bool = False,\n",
" dp_size: int = 1,\n",
" enable_multimodal: bool = False,\n",
" use_dedicated_endpoint: bool = False,\n",
" max_num_seqs: int | None = None,\n",
" is_spot: bool = True,\n",
" tool_call_parser: str | None = None,\n",
") -> Tuple[aiplatform.Model, aiplatform.Endpoint]:\n",
" \"\"\"Deploys trained models with SGLang into Vertex AI.\"\"\"\n",
" endpoint = aiplatform.Endpoint.create(\n",
" display_name=f\"{model_name}-endpoint\",\n",
" dedicated_endpoint_enabled=use_dedicated_endpoint,\n",
" )\n",
"\n",
" if not base_model_id:\n",
" base_model_id = model_id\n",
"\n",
" # See https://docs.sglang.ai/backend/server_arguments.html for a list of possible arguments with descriptions.\n",
" sglang_args = [\n",
" f\"--model={model_id}\",\n",
" f\"--tp={accelerator_count * multihost_gpu_node_count}\",\n",
" f\"--dp={dp_size}\",\n",
" ]\n",
"\n",
" if context_length:\n",
" sglang_args.append(f\"--context-length={context_length}\")\n",
"\n",
" if gpu_memory_utilization:\n",
" sglang_args.append(f\"--mem-fraction-static={gpu_memory_utilization}\")\n",
"\n",
" if max_num_seqs:\n",
" sglang_args.append(f\"--max-running-requests={max_num_seqs}\")\n",
"\n",
" if dtype:\n",
" sglang_args.append(f\"--dtype={dtype}\")\n",
"\n",
" if quantization:\n",
" sglang_args.append(f\"--quantization={quantization}\")\n",
"\n",
" if enable_trust_remote_code:\n",
" sglang_args.append(\"--trust-remote-code\")\n",
"\n",
" if enable_torch_compile:\n",
" sglang_args.append(\"--enable-torch-compile\")\n",
" if torch_compile_max_bs:\n",
" sglang_args.append(f\"--torch-compile-max-bs={torch_compile_max_bs}\")\n",
"\n",
" if attention_backend:\n",
" sglang_args.append(f\"--attention-backend={attention_backend}\")\n",
"\n",
" if enable_flashinfer_mla:\n",
" sglang_args.append(\"--enable-flashinfer-mla\")\n",
"\n",
" if disable_cuda_graph:\n",
" sglang_args.append(\"--disable-cuda-graph\")\n",
"\n",
" if speculative_algorithm:\n",
" sglang_args.append(f\"--speculative-algorithm={speculative_algorithm}\")\n",
" sglang_args.append(\n",
" f\"--speculative-draft-model-path={speculative_draft_model_path}\"\n",
" )\n",
" sglang_args.append(f\"--speculative-num-steps={speculative_num_steps}\")\n",
" sglang_args.append(f\"--speculative-eagle-topk={speculative_eagle_topk}\")\n",
" sglang_args.append(\n",
" f\"--speculative-num-draft-tokens={speculative_num_draft_tokens}\"\n",
" )\n",
"\n",
" if enable_dp_attention:\n",
" sglang_args.append(\"--enable-dp-attention\")\n",
"\n",
" if enable_multimodal:\n",
" sglang_args.append(\"--enable-multimodal\")\n",
"\n",
" if tool_call_parser:\n",
" sglang_args.append(f\"--tool-call-parser={tool_call_parser}\")\n",
"\n",
" env_vars = {\n",
" \"MODEL_ID\": base_model_id,\n",
" \"DEPLOY_SOURCE\": \"notebook\",\n",
" }\n",
"\n",
" if enable_jit_deepgemm:\n",
" env_vars[\"SGL_ENABLE_JIT_DEEPGEMM\"] = \"1\"\n",
"\n",
" # HF_TOKEN is not a compulsory field and may not be defined.\n",
" try:\n",
" if HF_TOKEN:\n",
" env_vars[\"HF_TOKEN\"] = HF_TOKEN\n",
" except NameError:\n",
" pass\n",
"\n",
" model = aiplatform.Model.upload(\n",
" display_name=model_name,\n",
" serving_container_image_uri=SGLANG_DOCKER_URI,\n",
" serving_container_args=sglang_args,\n",
" serving_container_ports=[30000],\n",
" serving_container_predict_route=\"/vertex_generate\",\n",
" serving_container_health_route=\"/health\",\n",
" serving_container_environment_variables=env_vars,\n",
" serving_container_shared_memory_size_mb=(16 * 1024), # 16 GB\n",
" serving_container_deployment_timeout=7200,\n",
" model_garden_source_model_name=(\n",
" f\"publishers/{publisher}/models/{publisher_model_id}\"\n",
" ),\n",
" )\n",
" print(\n",
" f\"Deploying {model_name} on {machine_type} with {int(accelerator_count * multihost_gpu_node_count)} {accelerator_type} GPU(s).\"\n",
" )\n",
"\n",
" creds, _ = auth.default()\n",
" auth_req = auth.transport.requests.Request()\n",
" creds.refresh(auth_req)\n",
"\n",
" url = f\"https://{REGION}-aiplatform.googleapis.com/ui/projects/{PROJECT_ID}/locations/{REGION}/endpoints/{endpoint.name}:deployModel\"\n",
" headers = {\n",
" \"Content-Type\": \"application/json\",\n",
" \"Authorization\": f\"Bearer {creds.token}\",\n",
" }\n",
" data = {\n",
" \"deployedModel\": {\n",
" \"model\": model.resource_name,\n",
" \"displayName\": model_name,\n",
" \"dedicatedResources\": {\n",
" \"machineSpec\": {\n",
" \"machineType\": machine_type,\n",
" \"multihostGpuNodeCount\": multihost_gpu_node_count,\n",
" \"acceleratorType\": accelerator_type,\n",
" \"acceleratorCount\": accelerator_count,\n",
" },\n",
" \"minReplicaCount\": 1,\n",
" \"maxReplicaCount\": 1,\n",
" },\n",
" \"system_labels\": {\n",
" \"NOTEBOOK_NAME\": \"model_garden_pytorch_qwen3_deployment.ipynb\",\n",
" \"NOTEBOOK_ENVIRONMENT\": common_util.get_deploy_source(),\n",
" },\n",
" },\n",
" }\n",
" if service_account:\n",
" data[\"deployedModel\"][\"serviceAccount\"] = service_account\n",
" if is_spot:\n",
" data[\"deployedModel\"][\"dedicatedResources\"][\"spot\"] = True\n",
" response = requests.post(url, headers=headers, json=data)\n",
" print(f\"Deploy Model response: {response.json()}\")\n",
" if response.status_code != 200 or \"name\" not in response.json():\n",
" raise ValueError(f\"Failed to deploy model: {response.text}\")\n",
" poll_and_wait(response.json()[\"name\"], 7200)\n",
" print(\"endpoint_name:\", endpoint.name)\n",
"\n",
" return model, endpoint\n",
"\n",
"\n",
"common_util.check_quota(\n",
" project_id=PROJECT_ID,\n",
" region=REGION,\n",
" accelerator_type=accelerator_type,\n",
" accelerator_count=accelerator_count,\n",
" is_for_training=False,\n",
")\n",
"\n",
"models[LABEL], endpoints[LABEL] = deploy_model_sglang_multihost(\n",
" model_name=common_util.get_job_name_with_datetime(prefix=\"qwen3-serve\"),\n",
" model_id=model_id,\n",
" publisher=\"qwen\",\n",
" publisher_model_id=\"qwen3\",\n",
" base_model_id=hf_model_id,\n",
" machine_type=machine_type,\n",
" accelerator_type=accelerator_type,\n",
" accelerator_count=accelerator_count,\n",
"endpoint = model.deploy(\n",
" accept_eula=True,\n",
" use_dedicated_endpoint=use_dedicated_endpoint,\n",
" is_spot=is_spot,\n",
")\n",
" serving_container_image_uri=\"us-docker.pkg.dev/deeplearning-platform-release/vertex-model-garden/sglang-serve.cu124.0-4.ubuntu2204.py310:20250428-1803-rc0\",\n",
" machine_type=\"a3-highgpu-2g\",\n",
" accelerator_type=\"NVIDIA_H100_80GB\",\n",
" accelerator_count=2,\n",
")"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "kqSUK2CwsImi"
},
"source": [
"To further customize your deployment, you can configure:\n",
"\n",
"# @markdown Click \"Show Code\" to see more details."
"- **Compute Resources**: Machine type, replica count (min/max), accelerator type and quantity.\n",
"- **Infrastructure**: Use Spot VMs, reservation affinity, or dedicated endpoints.\n",
"- **Serving Container**: Customize container image, ports, health checks, and environment variables.\n",
"\n",
"See the [Model Garden SDK README](https://github.com/googleapis/python-aiplatform/blob/main/vertexai/model_garden/README.md) for advanced configuration options."
]
},
{
@@ -617,7 +462,7 @@
" \"min_p\": min_p,\n",
" }\n",
"}\n",
"response = endpoints[\"sglang_gpu\"].predict(\n",
"response = endpoint.predict(\n",
" instances=instances,\n",
" parameters=parameters,\n",
" use_dedicated_endpoint=use_dedicated_endpoint,\n",
@@ -641,8 +486,8 @@
"# @title Chat completion\n",
"\n",
"if use_dedicated_endpoint:\n",
" DEDICATED_ENDPOINT_DNS = endpoints[\"sglang_gpu\"].gca_resource.dedicated_endpoint_dns\n",
"ENDPOINT_RESOURCE_NAME = endpoints[\"sglang_gpu\"].resource_name\n",
" DEDICATED_ENDPOINT_DNS = endpoint.gca_resource.dedicated_endpoint_dns\n",
"ENDPOINT_RESOURCE_NAME = endpoint.resource_name\n",
"\n",
"# @markdown Because the Qwen3 models generate detailed reasoning steps, the output is expected to be long. We recommend using streaming for a better generation experience.\n",
"# @title Chat Completions Inference\n",
@@ -723,18 +568,12 @@
},
"outputs": [],
"source": [
"# @title Delete the models and endpoints\n",
"# @title Delete the endpoints\n",
"\n",
"# @markdown Delete the experiment models and endpoints to recycle the resources\n",
"# @markdown and avoid unnecessary continuous charges that may incur.\n",
"# @markdown Delete the endpoint.\n",
"\n",
"# Undeploy model and delete endpoint.\n",
"for endpoint in endpoints.values():\n",
" endpoint.delete(force=True)\n",
"\n",
"# Delete models.\n",
"for model in models.values():\n",
" model.delete()"
"if endpoint:\n",
" endpoint.delete(force=True)"
]
}
],