mirror of
https://github.com/GoogleCloudPlatform/vertex-ai-samples.git
synced 2026-09-26 14:42:04 +00:00
Deepseek deployment v3_2 notebook
PiperOrigin-RevId: 824868883
This commit is contained in:
Vertex MG Team
committed by
Copybara-Service
Copybara-Service
parent
52444a0933
commit
b6bde41850
@@ -0,0 +1,592 @@
|
||||
{
|
||||
"cells": [
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {
|
||||
"cellView": "form",
|
||||
"id": "YXhbYF11R6oJ"
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"# Copyright 2025 Google LLC\n",
|
||||
"#\n",
|
||||
"# Licensed under the Apache License, Version 2.0 (the \"License\");\n",
|
||||
"# you may not use this file except in compliance with the License.\n",
|
||||
"# You may obtain a copy of the License at\n",
|
||||
"#\n",
|
||||
"# https://www.apache.org/licenses/LICENSE-2.0\n",
|
||||
"#\n",
|
||||
"# Unless required by applicable law or agreed to in writing, software\n",
|
||||
"# distributed under the License is distributed on an \"AS IS\" BASIS,\n",
|
||||
"# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n",
|
||||
"# See the License for the specific language governing permissions and\n",
|
||||
"# limitations under the License."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {
|
||||
"id": "vbPdpEwmShMY"
|
||||
},
|
||||
"source": [
|
||||
" # Vertex AI Model Garden - DeepSeek-V3.2\n",
|
||||
"\n",
|
||||
"<table><tbody><tr>\n",
|
||||
" <td style=\"text-align: center\">\n",
|
||||
" <a href=\"https://console.cloud.google.com/vertex-ai/notebooks/deploy-notebook?download_url=https://raw.githubusercontent.com/GoogleCloudPlatform/vertex-ai-samples/main/notebooks/community/model_garden/model_garden_pytorch_deepseek_v3_2_deployment.ipynb\">\n",
|
||||
" <img alt=\"Workbench logo\" src=\"https://lh3.googleusercontent.com/UiNooY4LUgW_oTvpsNhPpQzsstV5W8F7rYgxgGBD85cWJoLmrOzhVs_ksK_vgx40SHs7jCqkTkCk=e14-rj-sc0xffffff-h130-w32\" width=\"32px\"><br> Run in Workbench\n",
|
||||
" </a>\n",
|
||||
" </td>\n",
|
||||
" <td style=\"text-align: center\">\n",
|
||||
" <a href=\"https://console.cloud.google.com/vertex-ai/colab/import/https:%2F%2Fraw.githubusercontent.com%2FGoogleCloudPlatform%2Fvertex-ai-samples%2Fmain%2Fnotebooks%2Fcommunity%2Fmodel_garden%2Fmodel_garden_pytorch_deepseek_v3_2_deployment.ipynb\">\n",
|
||||
" <img alt=\"Google Cloud Colab Enterprise logo\" src=\"https://lh3.googleusercontent.com/JmcxdQi-qOpctIvWKgPtrzZdJJK-J3sWE1RsfjZNwshCFgE_9fULcNpuXYTilIR2hjwN\" width=\"32px\"><br> Run in Colab Enterprise\n",
|
||||
" </a>\n",
|
||||
" </td>\n",
|
||||
" <td style=\"text-align: center\">\n",
|
||||
" <a href=\"https://github.com/GoogleCloudPlatform/vertex-ai-samples/blob/main/notebooks/community/model_garden/model_garden_pytorch_deepseek_v3_2_deployment.ipynb\">\n",
|
||||
" <img alt=\"GitHub logo\" src=\"https://github.githubassets.com/assets/GitHub-Mark-ea2971cee799.png\" width=\"32px\"><br> View on GitHub\n",
|
||||
" </a>\n",
|
||||
" </td>\n",
|
||||
"</tr></tbody></table>"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {
|
||||
"id": "3de7470326a2"
|
||||
},
|
||||
"source": [
|
||||
"## Overview\n",
|
||||
"\n",
|
||||
"This notebook demonstrates how to deploy a **DeepSeek-V3.2** open model on Google Cloud Vertex AI.\n",
|
||||
"\n",
|
||||
"### Objectives\n",
|
||||
"\n",
|
||||
"- Deploy DeepSeek-V3.2 using containerized backends like [vLLM](https://github.com/vllm-project/vllm) on GPU.\n",
|
||||
"- Use the deployed model to serve chat completion requests for both text and multimodal inputs.\n",
|
||||
"\n",
|
||||
"### File a Bug\n",
|
||||
"\n",
|
||||
"If you encounter issues with this notebook, report them on [GitHub](https://github.com/GoogleCloudPlatform/vertex-ai-samples/issues/new).\n",
|
||||
"\n",
|
||||
"### Costs\n",
|
||||
"\n",
|
||||
"This tutorial uses billable components of Google Cloud:\n",
|
||||
"\n",
|
||||
"- Vertex AI\n",
|
||||
"- Cloud Storage\n",
|
||||
"\n",
|
||||
"Refer to the [Vertex AI pricing](https://cloud.google.com/vertex-ai/pricing) and [Cloud Storage pricing](https://cloud.google.com/storage/pricing) pages for more information. Use the [Pricing Calculator](https://cloud.google.com/products/calculator/) to estimate your projected costs."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {
|
||||
"id": "jeYw-Czg-DFy"
|
||||
},
|
||||
"source": [
|
||||
"## Get Started"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {
|
||||
"id": "KgyhGvEzBDkj"
|
||||
},
|
||||
"source": [
|
||||
"### Install Vertex AI SDK and other required packages"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {
|
||||
"cellView": "form",
|
||||
"id": "iCacdLqG-IsH"
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"%pip install --upgrade --force-reinstall --quiet 'google-cloud-aiplatform>=1.106.0' 'openai' 'google-auth==2.27.0' 'requests==2.32.3'"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {
|
||||
"id": "HUKCrpBy-3yf"
|
||||
},
|
||||
"source": [
|
||||
"### Authenticate the Notebook Environment (Colab only)\n",
|
||||
"\n",
|
||||
"If you're running this notebook in Google Colab, run the following cell to authenticate."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {
|
||||
"cellView": "form",
|
||||
"id": "JXwCT1kn-3Gu"
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"import sys\n",
|
||||
"\n",
|
||||
"if \"google.colab\" in sys.modules:\n",
|
||||
" from google.colab import auth\n",
|
||||
"\n",
|
||||
" auth.authenticate_user()"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {
|
||||
"id": "AcW2nwB8-7yC"
|
||||
},
|
||||
"source": [
|
||||
"### Set Google Cloud Project Information\n",
|
||||
"\n",
|
||||
"To get started with Vertex AI, ensure you have an existing Google Cloud project and that the [Vertex AI API is enabled](https://console.cloud.google.com/flows/enableapi?apiid=aiplatform.googleapis.com).\n",
|
||||
"\n",
|
||||
"See the guide on [setting up your project and development environment](https://cloud.google.com/vertex-ai/docs/start/cloud-environment). Also confirm that [billing is enabled](https://cloud.google.com/billing/docs/how-to/modify-project).\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {
|
||||
"cellView": "form",
|
||||
"id": "eIVLp0oE--k-"
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"# Use the environment variable if the user doesn't provide Project ID.\n",
|
||||
"import os\n",
|
||||
"\n",
|
||||
"import vertexai\n",
|
||||
"\n",
|
||||
"PROJECT_ID = \"\" # @param {type: \"string\", placeholder: \"[your-project-id]\", isTemplate: true}\n",
|
||||
"\n",
|
||||
"if not PROJECT_ID:\n",
|
||||
" PROJECT_ID = os.environ.get(\"GOOGLE_CLOUD_PROJECT\")\n",
|
||||
"\n",
|
||||
"REGION = \"\" # @param {type: \"string\", placeholder: \"[your-region]\", isTemplate: true}\n",
|
||||
"\n",
|
||||
"if not REGION:\n",
|
||||
" REGION = os.environ.get(\"GOOGLE_CLOUD_REGION\", \"us-central1\")\n",
|
||||
"\n",
|
||||
"vertexai.init(project=PROJECT_ID, location=REGION)\n",
|
||||
"\n",
|
||||
"print(f\"Project: {PROJECT_ID}\\nLocation: {REGION}\")"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {
|
||||
"id": "Q0CXrvcZH_aw"
|
||||
},
|
||||
"source": [
|
||||
"### Import libraries"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {
|
||||
"cellView": "form",
|
||||
"id": "3G2UXB82ICs6"
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"from vertexai import model_garden"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {
|
||||
"id": "upYRiGtP_-iN"
|
||||
},
|
||||
"source": [
|
||||
"## Deploy model"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {
|
||||
"id": "H2WC_0hXDVXc"
|
||||
},
|
||||
"source": [
|
||||
"### Choose model variant"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {
|
||||
"id": "u41zbNa2EoFq"
|
||||
},
|
||||
"source": [
|
||||
"You can proceed with the default model variant or select a different one."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {
|
||||
"cellView": "form",
|
||||
"id": "-fgC4NLSDkF7"
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"model_version = \"deepseek-v3-2-exp-base\" # @param [\"deepseek-v3-2-exp\", \"deepseek-v3-2-exp-base\"] {isTemplate:true}\n",
|
||||
"MODEL_NAME = f\"deepseek-ai/deepseek-v3-2@{model_version}\""
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {
|
||||
"id": "VRnUgU8LF3_i"
|
||||
},
|
||||
"source": [
|
||||
"To see all deployable model variants available in Model Garden, use:"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {
|
||||
"cellView": "form",
|
||||
"id": "-QLd-wshF6sB"
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"all_model_versions = model_garden.list_deployable_models(\n",
|
||||
" model_filter=\"deepseek-v3-2\", list_hf_models=False\n",
|
||||
")"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {
|
||||
"id": "N0UeFHa2GO63"
|
||||
},
|
||||
"source": [
|
||||
"Once you've selected a model variant, initialize it:"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {
|
||||
"cellView": "form",
|
||||
"id": "GZiV3trBBcA3"
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"model = model_garden.OpenModel(MODEL_NAME)"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {
|
||||
"id": "-0cL378wFlvf"
|
||||
},
|
||||
"source": [
|
||||
"### Check the Deployment Configuration\n",
|
||||
"\n",
|
||||
"Use the `list_deploy_options()` method to view the verified deployment configurations for your selected model. This helps ensure you have sufficient resources (e.g., GPU quota) available to deploy it."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {
|
||||
"cellView": "form",
|
||||
"id": "zm73g7vFFm9N"
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"deploy_options = model.list_deploy_options(concise=True)\n",
|
||||
"print(deploy_options)"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {
|
||||
"id": "WjV499VsGwrD"
|
||||
},
|
||||
"source": [
|
||||
"### Deploy the Model\n",
|
||||
"\n",
|
||||
"Now that you’ve reviewed the deployment options, use the `deploy()` method to serve the selected open model to a Vertex AI endpoint. Deployment time may vary depending on the model size and infrastructure requirements.\n",
|
||||
"\n",
|
||||
"> **Note**: If the model requires accepting a license agreement (EULA), set the `accept_eula=True` flag in the deploy call. Set `use_dedicated_endpoint` to False if you don't want to use [dedicated endpoint](https://cloud.google.com/vertex-ai/docs/general/deployment#create-dedicated-endpoint)."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {
|
||||
"cellView": "form",
|
||||
"id": "wX1itVTvXdEP"
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"use_dedicated_endpoint = True"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {
|
||||
"cellView": "form",
|
||||
"id": "S0q5fdbietBH"
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"endpoints = {}"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {
|
||||
"cellView": "form",
|
||||
"id": "MRmPFEPoGzsB"
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"endpoints[\"sdk_default\"] = model.deploy(\n",
|
||||
" accept_eula=True,\n",
|
||||
" use_dedicated_endpoint=use_dedicated_endpoint,\n",
|
||||
")"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {
|
||||
"id": "PHBtn8DQp-ID"
|
||||
},
|
||||
"source": [
|
||||
"Alternatively, you can select one of the verified deployment configurations listed above."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {
|
||||
"cellView": "form",
|
||||
"id": "ADsJG8JYqI6c"
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"endpoints[\"sdk_custom\"] = model.deploy(\n",
|
||||
" accept_eula=True,\n",
|
||||
" use_dedicated_endpoint=use_dedicated_endpoint,\n",
|
||||
" serving_container_image_uri=\"us-docker.pkg.dev/vertex-ai/vertex-vision-model-garden-dockers/pytorch-vllm-serve:20250930_0916_RC01\",\n",
|
||||
" machine_type=\"a4-highgpu-8g\",\n",
|
||||
" accelerator_type=\"NVIDIA_B200\",\n",
|
||||
" accelerator_count=8,\n",
|
||||
")"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {
|
||||
"cellView": "form",
|
||||
"id": "OCOHt9ivCdgA"
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"if \"sdk_default\" in endpoints:\n",
|
||||
" endpoint = endpoints[\"sdk_default\"]\n",
|
||||
"elif \"sdk_custom\" in endpoints:\n",
|
||||
" endpoint = endpoints[\"sdk_custom\"]\n",
|
||||
"else:\n",
|
||||
" raise ValueError(\"No endpoint found. Create an endpoint.\")"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {
|
||||
"id": "kqSUK2CwsImi"
|
||||
},
|
||||
"source": [
|
||||
"To further customize your deployment, you can configure:\n",
|
||||
"\n",
|
||||
"- **Compute Resources**: Machine type, replica count (min/max), accelerator type and quantity.\n",
|
||||
"- **Infrastructure**: Use Spot VMs, reservation affinity, or dedicated endpoints.\n",
|
||||
"- **Serving Container**: Customize container image, ports, health checks, and environment variables.\n",
|
||||
"\n",
|
||||
"See the [Model Garden SDK README](https://github.com/googleapis/python-aiplatform/blob/main/vertexai/model_garden/README.md) for advanced configuration options."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {
|
||||
"cellView": "form",
|
||||
"id": "5ptrDoPkSghh"
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"# @title Raw predict\n",
|
||||
"\n",
|
||||
"# @markdown Once deployment succeeds, you can send requests to the endpoint with text prompts. Sampling parameters supported by vLLM can be found [here](https://docs.vllm.ai/en/latest/dev/sampling_params.html).\n",
|
||||
"\n",
|
||||
"# @markdown Example:\n",
|
||||
"\n",
|
||||
"# @markdown ```\n",
|
||||
"# @markdown Human: What is a car?\n",
|
||||
"# @markdown Assistant: A car, or a motor car, is a road-connected human-transportation system used to move people or goods from one place to another. The term also encompasses a wide range of vehicles, including motorboats, trains, and aircrafts. Cars typically have four wheels, a cabin for passengers, and an engine or motor. They have been around since the early 19th century and are now one of the most popular forms of transportation, used for daily commuting, shopping, and other purposes.\n",
|
||||
"# @markdown ```\n",
|
||||
"# @markdown Additionally, you can moderate the generated text with Vertex AI. See [Moderate text documentation](https://cloud.google.com/natural-language/docs/moderating-text) for more details.\n",
|
||||
"\n",
|
||||
"# Loads an existing endpoint instance using the endpoint name:\n",
|
||||
"# - Using `endpoint_name = endpoint.name` allows us to get the\n",
|
||||
"# endpoint name of the endpoint `endpoint` created in the cell\n",
|
||||
"# above.\n",
|
||||
"# - Alternatively, you can set `endpoint_name = \"1234567890123456789\"` to load\n",
|
||||
"# an existing endpoint with the ID 1234567890123456789.\n",
|
||||
"# You may uncomment the code below to load an existing endpoint.\n",
|
||||
"\n",
|
||||
"# endpoint_name = \"\" # @param {type:\"string\"}\n",
|
||||
"# aip_endpoint_name = (\n",
|
||||
"# f\"projects/{PROJECT_ID}/locations/{REGION}/endpoints/{endpoint_name}\"\n",
|
||||
"# )\n",
|
||||
"# endpoint = aiplatform.Endpoint(aip_endpoint_name)\n",
|
||||
"\n",
|
||||
"prompt = \"What is a car?\" # @param {type: \"string\"}\n",
|
||||
"# @markdown If you encounter an issue like `ServiceUnavailable: 503 Took too long to respond when processing`, you can reduce the maximum number of output tokens, by lowering `max_tokens`.\n",
|
||||
"max_tokens = 50 # @param {type:\"integer\"}\n",
|
||||
"temperature = 1.0 # @param {type:\"number\"}\n",
|
||||
"top_p = 1.0 # @param {type:\"number\"}\n",
|
||||
"top_k = 1 # @param {type:\"integer\"}\n",
|
||||
"# @markdown Set `raw_response` to `True` to obtain the raw model output. Set `raw_response` to `False` to apply additional formatting in the structure of `\"Prompt:\\n{prompt.strip()}\\nOutput:\\n{output}\"`.\n",
|
||||
"raw_response = False # @param {type:\"boolean\"}\n",
|
||||
"\n",
|
||||
"# Overrides parameters for inferences.\n",
|
||||
"instances = [\n",
|
||||
" {\n",
|
||||
" \"prompt\": prompt,\n",
|
||||
" \"max_tokens\": max_tokens,\n",
|
||||
" \"temperature\": temperature,\n",
|
||||
" \"top_p\": top_p,\n",
|
||||
" \"top_k\": top_k,\n",
|
||||
" \"raw_response\": raw_response,\n",
|
||||
" },\n",
|
||||
"]\n",
|
||||
"response = endpoint.predict(\n",
|
||||
" instances=instances, use_dedicated_endpoint=use_dedicated_endpoint\n",
|
||||
")\n",
|
||||
"\n",
|
||||
"for prediction in response.predictions:\n",
|
||||
" print(prediction)\n",
|
||||
"\n",
|
||||
"# @markdown Click \"Show Code\" to see more details."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {
|
||||
"cellView": "form",
|
||||
"id": "AMjpQGSfYORn"
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"# @title Chat completion\n",
|
||||
"\n",
|
||||
"if use_dedicated_endpoint:\n",
|
||||
" DEDICATED_ENDPOINT_DNS = endpoint.gca_resource.dedicated_endpoint_dns\n",
|
||||
"ENDPOINT_RESOURCE_NAME = endpoint.resource_name\n",
|
||||
"\n",
|
||||
"# @title Chat Completions Inference\n",
|
||||
"\n",
|
||||
"# @markdown Once deployment succeeds, you can send requests to the endpoint using the OpenAI SDK.\n",
|
||||
"\n",
|
||||
"# @markdown First you will need to install the SDK and some auth-related dependencies.\n",
|
||||
"\n",
|
||||
"! pip install -qU openai google-auth requests\n",
|
||||
"\n",
|
||||
"# @markdown Next fill out some request parameters:\n",
|
||||
"\n",
|
||||
"user_message = \"How is your day going?\" # @param {type: \"string\"}\n",
|
||||
"# @markdown If you encounter the issue like `ServiceUnavailable: 503 Took too long to respond when processing`, you can reduce the maximum number of output tokens, such as set `max_tokens` as 20.\n",
|
||||
"max_tokens = 50 # @param {type: \"integer\"}\n",
|
||||
"temperature = 1.0 # @param {type: \"number\"}\n",
|
||||
"stream = False # @param {type: \"boolean\"}\n",
|
||||
"\n",
|
||||
"# @markdown Now we can send a request.\n",
|
||||
"\n",
|
||||
"import google.auth\n",
|
||||
"import openai\n",
|
||||
"\n",
|
||||
"creds, project = google.auth.default()\n",
|
||||
"auth_req = google.auth.transport.requests.Request()\n",
|
||||
"creds.refresh(auth_req)\n",
|
||||
"\n",
|
||||
"BASE_URL = (\n",
|
||||
" f\"https://{REGION}-aiplatform.googleapis.com/v1beta1/{ENDPOINT_RESOURCE_NAME}\"\n",
|
||||
")\n",
|
||||
"try:\n",
|
||||
" if use_dedicated_endpoint:\n",
|
||||
" BASE_URL = f\"https://{DEDICATED_ENDPOINT_DNS}/v1beta1/{ENDPOINT_RESOURCE_NAME}\"\n",
|
||||
"except NameError:\n",
|
||||
" pass\n",
|
||||
"\n",
|
||||
"client = openai.OpenAI(base_url=BASE_URL, api_key=creds.token)\n",
|
||||
"\n",
|
||||
"model_response = client.chat.completions.create(\n",
|
||||
" model=\"\",\n",
|
||||
" messages=[{\"role\": \"user\", \"content\": user_message}],\n",
|
||||
" temperature=temperature,\n",
|
||||
" max_tokens=max_tokens,\n",
|
||||
" stream=stream,\n",
|
||||
")\n",
|
||||
"\n",
|
||||
"if stream:\n",
|
||||
" usage = None\n",
|
||||
" contents = []\n",
|
||||
" for chunk in model_response:\n",
|
||||
" if chunk.usage is not None:\n",
|
||||
" usage = chunk.usage\n",
|
||||
" continue\n",
|
||||
" print(chunk.choices[0].delta.content, end=\"\")\n",
|
||||
" contents.append(chunk.choices[0].delta.content)\n",
|
||||
" print(f\"\\n\\n{usage}\")\n",
|
||||
"else:\n",
|
||||
" print(model_response)"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {
|
||||
"cellView": "form",
|
||||
"id": "XQ_xFlmPYU_V"
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"# @title Delete the endpoints\n",
|
||||
"\n",
|
||||
"# @markdown Delete the endpoint.\n",
|
||||
"\n",
|
||||
"for endpoint in endpoints.values():\n",
|
||||
" endpoint.delete(force=True)"
|
||||
]
|
||||
}
|
||||
],
|
||||
"metadata": {
|
||||
"colab": {
|
||||
"name": "model_garden_pytorch_deepseek_v3_2_deployment.ipynb",
|
||||
"toc_visible": true
|
||||
},
|
||||
"kernelspec": {
|
||||
"display_name": "Python 3",
|
||||
"name": "python3"
|
||||
}
|
||||
},
|
||||
"nbformat": 4,
|
||||
"nbformat_minor": 0
|
||||
}
|
||||
Reference in New Issue
Block a user