mirror of
https://github.com/GoogleCloudPlatform/vertex-ai-samples.git
synced 2026-09-26 14:42:04 +00:00
feat: Add prediction dedicated endpoint colab sample (#3942)
* feat: Add prediction dedicated endpoint colab sample * Update get_started_with_dedicated_endpoint.ipynb --------- Co-authored-by: TJ(Tianjiao) Liu <tianjiaoliu@google.com>
This commit is contained in:
TJ-Liu
@@ -57,6 +57,7 @@
|
|||||||
/training/tpuv5e_gemma_peft_finetuning_and_serving.ipynb @brianchunkang
|
/training/tpuv5e_gemma_peft_finetuning_and_serving.ipynb @brianchunkang
|
||||||
/training/tpuv5e_llama2_pytorch_finetuning_and_serving.ipynb @brianchunkang @chiefkarlin
|
/training/tpuv5e_llama2_pytorch_finetuning_and_serving.ipynb @brianchunkang @chiefkarlin
|
||||||
/prediction/get_started_with_psc_private_endpoint.ipynb @tianjiaoliu
|
/prediction/get_started_with_psc_private_endpoint.ipynb @tianjiaoliu
|
||||||
|
/prediction/get_started_with_dedicated_endpoint.ipynb @tianjiaoliu
|
||||||
/ray_on_vertex_ai/spark_on_ray_on_vertex_ai.ipynb @ravi-dalal
|
/ray_on_vertex_ai/spark_on_ray_on_vertex_ai.ipynb @ravi-dalal
|
||||||
/generative_ai/mistralai_intro.ipynb @sujituk
|
/generative_ai/mistralai_intro.ipynb @sujituk
|
||||||
/generative_ai/ai21labs_intro.ipynb @miri-bar
|
/generative_ai/ai21labs_intro.ipynb @miri-bar
|
||||||
|
|||||||
@@ -0,0 +1,827 @@
|
|||||||
|
{
|
||||||
|
"cells": [
|
||||||
|
{
|
||||||
|
"cell_type": "code",
|
||||||
|
"execution_count": null,
|
||||||
|
"metadata": {
|
||||||
|
"id": "UUTFwSh5u9OK"
|
||||||
|
},
|
||||||
|
"outputs": [],
|
||||||
|
"source": [
|
||||||
|
"# Copyright 2024 Google LLC\n",
|
||||||
|
"#\n",
|
||||||
|
"# Licensed under the Apache License, Version 2.0 (the \"License\");\n",
|
||||||
|
"# you may not use this file except in compliance with the License.\n",
|
||||||
|
"# You may obtain a copy of the License at\n",
|
||||||
|
"#\n",
|
||||||
|
"# https://www.apache.org/licenses/LICENSE-2.0\n",
|
||||||
|
"#\n",
|
||||||
|
"# Unless required by applicable law or agreed to in writing, software\n",
|
||||||
|
"# distributed under the License is distributed on an \"AS IS\" BASIS,\n",
|
||||||
|
"# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n",
|
||||||
|
"# See the License for the specific language governing permissions and\n",
|
||||||
|
"# limitations under the License."
|
||||||
|
]
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"cell_type": "markdown",
|
||||||
|
"metadata": {
|
||||||
|
"id": "qsLff0QbdE8D"
|
||||||
|
},
|
||||||
|
"source": [
|
||||||
|
"# Online Prediction Dedicated Endpint\n",
|
||||||
|
"\n",
|
||||||
|
"<table align=\"left\">\n",
|
||||||
|
" <td style=\"text-align: center\">\n",
|
||||||
|
" <a href=\"https://colab.research.google.com/github/GoogleCloudPlatform/vertex-ai-samples/blob/main/notebooks/official/prediction/get_started_with_dedicated_endpoint.ipynb\">\n",
|
||||||
|
" <img src=\"https://cloud.google.com/ml-engine/images/colab-logo-32px.png\" alt=\"Google Colaboratory logo\"><br> Open in Colab\n",
|
||||||
|
" </a>\n",
|
||||||
|
" </td>\n",
|
||||||
|
" <td style=\"text-align: center\">\n",
|
||||||
|
" <a href=\"https://console.cloud.google.com/vertex-ai/colab/import/https:%2F%2Fraw.githubusercontent.com%2FGoogleCloudPlatform%2Fvertex-ai-samples%2Fmain%2Fnotebooks%2Fofficial%2Fprediction%2Fget_started_with_dedicated_endpoint.ipynb\">\n",
|
||||||
|
" <img width=\"32px\" src=\"https://cloud.google.com/ml-engine/images/colab-enterprise-logo-32px.png\" alt=\"Google Cloud Colab Enterprise logo\"><br> Open in Colab Enterprise\n",
|
||||||
|
" </a>\n",
|
||||||
|
" </td> \n",
|
||||||
|
" <td style=\"text-align: center\">\n",
|
||||||
|
" <a href=\"https://console.cloud.google.com/vertex-ai/workbench/deploy-notebook?download_url=https://raw.githubusercontent.com/GoogleCloudPlatform/vertex-ai-samples/main/notebooks/official/prediction/get_started_with_dedicated_endpoint.ipynb\">\n",
|
||||||
|
" <img src=\"https://lh3.googleusercontent.com/UiNooY4LUgW_oTvpsNhPpQzsstV5W8F7rYgxgGBD85cWJoLmrOzhVs_ksK_vgx40SHs7jCqkTkCk=e14-rj-sc0xffffff-h130-w32\" alt=\"Vertex AI logo\"><br> Open in Workbench\n",
|
||||||
|
" </a>\n",
|
||||||
|
" </td>\n",
|
||||||
|
" <td style=\"text-align: center\">\n",
|
||||||
|
" <a href=\"https://github.com/GoogleCloudPlatform/vertex-ai-samples/blob/main/notebooks/official/prediction/get_started_with_dedicated_endpoint.ipynb\">\n",
|
||||||
|
" <img src=\"https://cloud.google.com/ml-engine/images/github-logo-32px.png\" alt=\"GitHub logo\"><br> View on GitHub\n",
|
||||||
|
" </a>\n",
|
||||||
|
" </td>\n",
|
||||||
|
"</table>"
|
||||||
|
]
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"cell_type": "markdown",
|
||||||
|
"metadata": {
|
||||||
|
"id": "0C7kTlrH1bO8"
|
||||||
|
},
|
||||||
|
"source": [
|
||||||
|
"## Overview\n",
|
||||||
|
"\n",
|
||||||
|
"A dedicated public endpoint is a public endpoint for online prediction. It offers the following benefits:\n",
|
||||||
|
"\n",
|
||||||
|
"* Dedicated networking: When you send a prediction request to a dedicated public endpoint, it is isolated from other users' traffic.\n",
|
||||||
|
"* Optimized network latency\n",
|
||||||
|
"* Larger payload support: Up to 10 MB.\n",
|
||||||
|
"* Longer request timeouts: Configurable up to 1 hour.\n",
|
||||||
|
"* Generative AI-ready: Streaming and gRPC are supported."
|
||||||
|
]
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"cell_type": "markdown",
|
||||||
|
"metadata": {
|
||||||
|
"id": "D2sWuTr81v9m"
|
||||||
|
},
|
||||||
|
"source": [
|
||||||
|
"## Get started"
|
||||||
|
]
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"cell_type": "markdown",
|
||||||
|
"metadata": {
|
||||||
|
"id": "RakMIliNYh8O"
|
||||||
|
},
|
||||||
|
"source": [
|
||||||
|
"### Install Vertex AI SDK for Python and other required packages"
|
||||||
|
]
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"cell_type": "code",
|
||||||
|
"execution_count": null,
|
||||||
|
"metadata": {
|
||||||
|
"id": "nSHmJT9cTggu"
|
||||||
|
},
|
||||||
|
"outputs": [],
|
||||||
|
"source": [
|
||||||
|
"! pip3 install --upgrade --quiet google-cloud-aiplatform"
|
||||||
|
]
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"cell_type": "markdown",
|
||||||
|
"metadata": {
|
||||||
|
"id": "936Zz5YI2NeA"
|
||||||
|
},
|
||||||
|
"source": [
|
||||||
|
"### Restart runtime (Colab only)\n",
|
||||||
|
"\n",
|
||||||
|
"To use the newly installed packages, you must restart the runtime on Google Colab."
|
||||||
|
]
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"cell_type": "code",
|
||||||
|
"execution_count": null,
|
||||||
|
"metadata": {
|
||||||
|
"id": "66oJ55lG2Tiq"
|
||||||
|
},
|
||||||
|
"outputs": [],
|
||||||
|
"source": [
|
||||||
|
"import sys\n",
|
||||||
|
"\n",
|
||||||
|
"if \"google.colab\" in sys.modules:\n",
|
||||||
|
"\n",
|
||||||
|
" import IPython\n",
|
||||||
|
"\n",
|
||||||
|
" app = IPython.Application.instance()\n",
|
||||||
|
" app.kernel.do_shutdown(True)"
|
||||||
|
]
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"cell_type": "markdown",
|
||||||
|
"metadata": {
|
||||||
|
"id": "Clr61ben2WwY"
|
||||||
|
},
|
||||||
|
"source": [
|
||||||
|
"<div class=\"alert alert-block alert-warning\">\n",
|
||||||
|
"<b>⚠️ The kernel is going to restart. Wait until it's finished before continuing to the next step. ⚠️</b>\n",
|
||||||
|
"</div>\n"
|
||||||
|
]
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"cell_type": "markdown",
|
||||||
|
"metadata": {
|
||||||
|
"id": "v848aGbn2acH"
|
||||||
|
},
|
||||||
|
"source": [
|
||||||
|
"### Authenticate your notebook environment (Colab only)\n",
|
||||||
|
"\n",
|
||||||
|
"Authenticate your environment on Google Colab.\n"
|
||||||
|
]
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"cell_type": "code",
|
||||||
|
"execution_count": null,
|
||||||
|
"metadata": {
|
||||||
|
"id": "IVeoyQPz2cfh"
|
||||||
|
},
|
||||||
|
"outputs": [],
|
||||||
|
"source": [
|
||||||
|
"import sys\n",
|
||||||
|
"\n",
|
||||||
|
"if \"google.colab\" in sys.modules:\n",
|
||||||
|
"\n",
|
||||||
|
" from google.colab import auth\n",
|
||||||
|
"\n",
|
||||||
|
" auth.authenticate_user()"
|
||||||
|
]
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"cell_type": "markdown",
|
||||||
|
"metadata": {
|
||||||
|
"id": "HId-ySlY2jlI"
|
||||||
|
},
|
||||||
|
"source": [
|
||||||
|
"### Set Google Cloud project information and initialize Vertex AI SDK for Python\n",
|
||||||
|
"\n",
|
||||||
|
"To get started using Vertex AI, you must have an existing Google Cloud project and [enable the Vertex AI API](https://console.cloud.google.com/flows/enableapi?apiid=aiplatform.googleapis.com). Learn more about [setting up a project and a development environment](https://cloud.google.com/vertex-ai/docs/start/cloud-environment)."
|
||||||
|
]
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"cell_type": "code",
|
||||||
|
"execution_count": null,
|
||||||
|
"metadata": {
|
||||||
|
"id": "Y4gnZI9OX6VJ"
|
||||||
|
},
|
||||||
|
"outputs": [],
|
||||||
|
"source": [
|
||||||
|
"# Set project id to be the current project ID.\n",
|
||||||
|
"\n",
|
||||||
|
"PROJECT_ID = \"[your-project-id]\" # @param {type:\"string\"}\n",
|
||||||
|
"LOCATION = \"us-central1\" # @param {type:\"string\"}"
|
||||||
|
]
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"cell_type": "code",
|
||||||
|
"execution_count": null,
|
||||||
|
"metadata": {
|
||||||
|
"id": "-NrpFROTjoVL"
|
||||||
|
},
|
||||||
|
"outputs": [],
|
||||||
|
"source": [
|
||||||
|
"from google.cloud import aiplatform\n",
|
||||||
|
"\n",
|
||||||
|
"aiplatform.init(project=PROJECT_ID, location=LOCATION)"
|
||||||
|
]
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"cell_type": "code",
|
||||||
|
"execution_count": null,
|
||||||
|
"metadata": {
|
||||||
|
"id": "moS794OKaaCt"
|
||||||
|
},
|
||||||
|
"outputs": [],
|
||||||
|
"source": [
|
||||||
|
"# Create GCS Bucket\n",
|
||||||
|
"BUCKET_URI = \"gs://your-bucket-name-unique\" # @param {type:\"string\"}\n",
|
||||||
|
"! gsutil mb -l {LOCATION} -p {PROJECT_ID} {BUCKET_URI}"
|
||||||
|
]
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"cell_type": "markdown",
|
||||||
|
"metadata": {
|
||||||
|
"id": "SASm5tKB2JGF"
|
||||||
|
},
|
||||||
|
"source": [
|
||||||
|
"## Create Dedicated Endpoint and Deploy Model"
|
||||||
|
]
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"cell_type": "markdown",
|
||||||
|
"metadata": {
|
||||||
|
"id": "taUmNhwP2S3W"
|
||||||
|
},
|
||||||
|
"source": [
|
||||||
|
"### Create Dedicated Endpoint\n",
|
||||||
|
"When creating endpoint, set `dedicated_endpoint_enabled` to True."
|
||||||
|
]
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"cell_type": "code",
|
||||||
|
"execution_count": null,
|
||||||
|
"metadata": {
|
||||||
|
"id": "IZUfl8Q82crP"
|
||||||
|
},
|
||||||
|
"outputs": [],
|
||||||
|
"source": [
|
||||||
|
"endpoint = aiplatform.Endpoint.create(\n",
|
||||||
|
" display_name=\"test-dedicated-endpoint\",\n",
|
||||||
|
" dedicated_endpoint_enabled=True,\n",
|
||||||
|
")"
|
||||||
|
]
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"cell_type": "markdown",
|
||||||
|
"metadata": {
|
||||||
|
"id": "swERjzZ-a_Nd"
|
||||||
|
},
|
||||||
|
"source": [
|
||||||
|
"### Prepare Test Models\n",
|
||||||
|
"\n",
|
||||||
|
"We prepared a tensorflow test model, feel free to use your own models."
|
||||||
|
]
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"cell_type": "code",
|
||||||
|
"execution_count": null,
|
||||||
|
"metadata": {
|
||||||
|
"id": "0fea4d3018ab"
|
||||||
|
},
|
||||||
|
"outputs": [],
|
||||||
|
"source": [
|
||||||
|
"! pip freeze | grep google-cloud-aiplatform"
|
||||||
|
]
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"cell_type": "code",
|
||||||
|
"execution_count": null,
|
||||||
|
"metadata": {
|
||||||
|
"id": "bA2zhbzJ1R0k"
|
||||||
|
},
|
||||||
|
"outputs": [],
|
||||||
|
"source": [
|
||||||
|
"# List all your models\n",
|
||||||
|
"for my_model in aiplatform.Model.list():\n",
|
||||||
|
" print(my_model.display_name)\n",
|
||||||
|
" print(my_model.gca_resource.name)\n",
|
||||||
|
"\n",
|
||||||
|
"# If you want to use an existing model, use the resource id\n",
|
||||||
|
"# model = aiplatform.Model('projects/12345/locations/us-central1/models/456789')"
|
||||||
|
]
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"cell_type": "code",
|
||||||
|
"execution_count": null,
|
||||||
|
"metadata": {
|
||||||
|
"id": "3PtFCQNHbloQ"
|
||||||
|
},
|
||||||
|
"outputs": [],
|
||||||
|
"source": [
|
||||||
|
"# @title Upload a new tensorflow model\n",
|
||||||
|
"# @markdown You can skip this if use an existing model.\n",
|
||||||
|
"\n",
|
||||||
|
"# TF Model\n",
|
||||||
|
"DISPLAY_NAME = \"tensorflow model\" # @param {type:\"string\"}\n",
|
||||||
|
"ARTIFACT_URI = BUCKET_URI + \"/tensorflow\"\n",
|
||||||
|
"IMAGE_URI = \"us-docker.pkg.dev/vertex-ai/prediction/tf2-cpu.2-12:latest\"\n",
|
||||||
|
"\n",
|
||||||
|
"# Copy Test Models to the Bucket\n",
|
||||||
|
"! gsutil -m cp -r \"gs://cloud-samples-data/vertex-ai/prediction/test-models-requests/tensorflow/*\" {ARTIFACT_URI}\n",
|
||||||
|
"\n",
|
||||||
|
"model = aiplatform.Model.upload(\n",
|
||||||
|
" display_name=DISPLAY_NAME,\n",
|
||||||
|
" artifact_uri=ARTIFACT_URI,\n",
|
||||||
|
" serving_container_image_uri=IMAGE_URI,\n",
|
||||||
|
")"
|
||||||
|
]
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"cell_type": "markdown",
|
||||||
|
"metadata": {
|
||||||
|
"id": "USWCW-SNo-9M"
|
||||||
|
},
|
||||||
|
"source": [
|
||||||
|
"#### Deploy Model\n",
|
||||||
|
"\n",
|
||||||
|
"Deploy the model to the endpoint. If it is your first model with dedicated endpoint, it will take ~30 min."
|
||||||
|
]
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"cell_type": "code",
|
||||||
|
"execution_count": null,
|
||||||
|
"metadata": {
|
||||||
|
"id": "S_eRJglhpVfL"
|
||||||
|
},
|
||||||
|
"outputs": [],
|
||||||
|
"source": [
|
||||||
|
"endpoint.deploy(model=model, traffic_percentage=100, machine_type=\"e2-standard-8\")"
|
||||||
|
]
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"cell_type": "markdown",
|
||||||
|
"metadata": {
|
||||||
|
"id": "civyNQaPr4QD"
|
||||||
|
},
|
||||||
|
"source": [
|
||||||
|
"## Make Predictions\n",
|
||||||
|
"\n",
|
||||||
|
"Dedicated endpoint cannot be accessed with the shared `aiplatform.googleapis.com` DNS. \\\n",
|
||||||
|
"Instead, it has its own dedicated DNS\n",
|
||||||
|
"```\n",
|
||||||
|
"DEDICATED_DNS = f\"https://{endpoint.gca_resource.dedicated_endpoint_dns}/v1beta1/{endpoint.resource_name}\"\n",
|
||||||
|
"# or\n",
|
||||||
|
"DEDICATED_DNS = f\"https://{ENDPOINT_ID}.{LOCATION}-{PROJECT_NUMBER}.prediction.vertexai.goog\"\n",
|
||||||
|
"```\n",
|
||||||
|
"\n",
|
||||||
|
"Python SDK has been integrated to support dedicated endpoint using a flag `use_dedicated_endpoint`.\n",
|
||||||
|
"\n",
|
||||||
|
"Alternatively, you can send HTTP/GRPC request directly to this DNS in any language you prefer."
|
||||||
|
]
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"cell_type": "code",
|
||||||
|
"execution_count": null,
|
||||||
|
"metadata": {
|
||||||
|
"cellView": "form",
|
||||||
|
"id": "I-O9U63juoWE"
|
||||||
|
},
|
||||||
|
"outputs": [],
|
||||||
|
"source": [
|
||||||
|
"# @title Predict\n",
|
||||||
|
"# @markdown You can use client library.\n",
|
||||||
|
"\n",
|
||||||
|
"use_python_sdk = True # @param {type:\"boolean\"}\n",
|
||||||
|
"# @markdown response = my_endpoint.predict( \\\n",
|
||||||
|
"# @markdown   instances=[{\"feat_1\":val_1, \"feat_2\":val_2}]}, \\\n",
|
||||||
|
"# @markdown   headers = {'Content-Type':'application/json'}, \\\n",
|
||||||
|
"# @markdown   dedicated_endpoint=True, \\\n",
|
||||||
|
"# @markdown )\n",
|
||||||
|
"\n",
|
||||||
|
"if use_python_sdk:\n",
|
||||||
|
" instances = [\n",
|
||||||
|
" {\"dense_input\": [14.0, 7.0, 2545.461893666405, 54.2, 48.5, 0.0, 61.3, 0.0, 0.0, 0.0]},\n",
|
||||||
|
" {\"dense_input\": [28.0, 14.0, 1234, 27.1, 90, 0.0, 61.3, 0.0, 0.0, 0.0]}\n",
|
||||||
|
" ]\n",
|
||||||
|
" response = endpoint.predict(instances=instances, use_dedicated_endpoint=True)\n",
|
||||||
|
" print(response)\n",
|
||||||
|
"\n",
|
||||||
|
"# @markdown You can also make HTTP/GRPC request directly.\n",
|
||||||
|
"\n",
|
||||||
|
"use_http = True # @param {type:\"boolean\"}\n",
|
||||||
|
"# @markdown POST request to the following URL:\\\n",
|
||||||
|
"# @markdown `https://ENDPOINT_ID.us-central1-PROJECT_NUMBER.prediction.vertexai.goog/v1/projects/PROJECT_NUMBER/locations/LOCATION/endpoints/ENDPOINT_ID:predict`\n",
|
||||||
|
"\n",
|
||||||
|
"if use_http:\n",
|
||||||
|
" request_path = f\"https://{endpoint.gca_resource.dedicated_endpoint_dns}/v1/{endpoint.resource_name}:predict\"\n",
|
||||||
|
" ! curl {request_path} -X POST -H \"Content-Type: application/json\" -H \"Authorization: Bearer `gcloud auth print-access-token`\" -d \\\n",
|
||||||
|
" '{{ \\\n",
|
||||||
|
" \"instances\": [ \\\n",
|
||||||
|
" {{\"dense_input\": [14.0, 7.0, 2545.461893666405, 54.2, 48.5, 0.0, 61.3, 0.0, 0.0, 0.0]}}, \\\n",
|
||||||
|
" {{\"dense_input\": [28.0, 14.0, 1234, 27.1, 90, 0.0, 61.3, 0.0, 0.0, 0.0]}} \\\n",
|
||||||
|
" ] \\\n",
|
||||||
|
" }}'"
|
||||||
|
]
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"cell_type": "code",
|
||||||
|
"execution_count": null,
|
||||||
|
"metadata": {
|
||||||
|
"cellView": "form",
|
||||||
|
"id": "RRoiziQ9FAw6"
|
||||||
|
},
|
||||||
|
"outputs": [],
|
||||||
|
"source": [
|
||||||
|
"# @title Raw Predict\n",
|
||||||
|
"# @markdown You can use client library.\n",
|
||||||
|
"\n",
|
||||||
|
"use_python_sdk = True # @param {type:\"boolean\"}\n",
|
||||||
|
"# @markdown response = my_endpoint.raw_predict( \\\n",
|
||||||
|
"# @markdown    body = b'{\"instances\":[{\"feat_1\":val_1, \"feat_2\":val_2}]}', \\\n",
|
||||||
|
"# @markdown    headers = {'Content-Type':'application/json'}, \\\n",
|
||||||
|
"# @markdown    dedicated_endpoint=True, \\\n",
|
||||||
|
"# @markdown ) \\\n",
|
||||||
|
"# @markdown status_code = response.status_code \\\n",
|
||||||
|
"# @markdown results = json.dumps(response.text)\n",
|
||||||
|
"\n",
|
||||||
|
"if use_python_sdk:\n",
|
||||||
|
" body = b'{ \\\n",
|
||||||
|
" \"instances\": [ \\\n",
|
||||||
|
" {\"dense_input\": [14.0, 7.0, 2545.461893666405, 54.2, 48.5, 0.0, 61.3, 0.0, 0.0, 0.0]}, \\\n",
|
||||||
|
" {\"dense_input\": [28.0, 14.0, 1234, 27.1, 90, 0.0, 61.3, 0.0, 0.0, 0.0]} \\\n",
|
||||||
|
" ]\\\n",
|
||||||
|
" }'\n",
|
||||||
|
" import os\n",
|
||||||
|
" token = !gcloud auth print-access-token\n",
|
||||||
|
" token = token[0]\n",
|
||||||
|
" headers = {\"content-type\": \"application/json\", \"Authorization\": f\"Bearer {token}\"}\n",
|
||||||
|
" response = endpoint.raw_predict(body=body, headers=headers, use_dedicated_endpoint=True)\n",
|
||||||
|
" # print(response.status_code)\n",
|
||||||
|
" print(response.text)\n",
|
||||||
|
"\n",
|
||||||
|
"# @markdown You can also make HTTP/GRPC request directly.\n",
|
||||||
|
"\n",
|
||||||
|
"use_http = True # @param {type:\"boolean\"}\n",
|
||||||
|
"# @markdown POST request to the following URL:\\\n",
|
||||||
|
"# @markdown `https://ENDPOINT_ID.us-central1-PROJECT_NUMBER.prediction.vertexai.goog/v1/projects/PROJECT_NUMBER/locations/LOCATION/endpoints/ENDPOINT_ID:rawPredict`\n",
|
||||||
|
"\n",
|
||||||
|
"if use_http:\n",
|
||||||
|
" request_path = f\"https://{endpoint.gca_resource.dedicated_endpoint_dns}/v1/{endpoint.resource_name}:rawPredict\"\n",
|
||||||
|
" ! curl {request_path} -X POST -H \"Content-Type: application/json\" -H \"Authorization: Bearer `gcloud auth print-access-token`\" -d \\\n",
|
||||||
|
" '{{ \\\n",
|
||||||
|
" \"instances\": [ \\\n",
|
||||||
|
" {{\"dense_input\": [14.0, 7.0, 2545.461893666405, 54.2, 48.5, 0.0, 61.3, 0.0, 0.0, 0.0]}}, \\\n",
|
||||||
|
" {{\"dense_input\": [28.0, 14.0, 1234, 27.1, 90, 0.0, 61.3, 0.0, 0.0, 0.0]}} \\\n",
|
||||||
|
" ] \\\n",
|
||||||
|
" }}'"
|
||||||
|
]
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"cell_type": "code",
|
||||||
|
"execution_count": null,
|
||||||
|
"metadata": {
|
||||||
|
"cellView": "form",
|
||||||
|
"id": "IbZDIJlIRl1b"
|
||||||
|
},
|
||||||
|
"outputs": [],
|
||||||
|
"source": [
|
||||||
|
"# @title Stream Raw Predict\n",
|
||||||
|
"# @markdown You can use client library.\n",
|
||||||
|
"\n",
|
||||||
|
"use_python_sdk = True # @param {type:\"boolean\"}\n",
|
||||||
|
"# @markdown for stream_response in my_endpoint.stream_raw_predict( \\\n",
|
||||||
|
"# @markdown    body = b'{\"instances\":[{\"feat_1\":val_1, \"feat_2\":val_2}]}', \\\n",
|
||||||
|
"# @markdown    headers = {'Content-Type':'application/json'}, \\\n",
|
||||||
|
"# @markdown    use_dedicated_endpoint=True, \\\n",
|
||||||
|
"# @markdown ): \\\n",
|
||||||
|
"# @markdown    status_code = response.status_code \\\n",
|
||||||
|
"# @markdown    stream_result = json.dumps(response.text)\n",
|
||||||
|
"\n",
|
||||||
|
"if use_python_sdk:\n",
|
||||||
|
" body = b'{ \\\n",
|
||||||
|
" \"instances\": [ \\\n",
|
||||||
|
" {\"dense_input\": [14.0, 7.0, 2545.461893666405, 54.2, 48.5, 0.0, 61.3, 0.0, 0.0, 0.0]}, \\\n",
|
||||||
|
" {\"dense_input\": [28.0, 14.0, 1234, 27.1, 90, 0.0, 61.3, 0.0, 0.0, 0.0]} \\\n",
|
||||||
|
" ]\\\n",
|
||||||
|
" }'\n",
|
||||||
|
" import os\n",
|
||||||
|
" token = !gcloud auth print-access-token\n",
|
||||||
|
" token = token[0]\n",
|
||||||
|
" headers = {\"content-type\": \"application/json\", \"Authorization\": f\"Bearer {token}\"}\n",
|
||||||
|
" for stream_response in endpoint.raw_predict(body=body, headers=headers, use_dedicated_endpoint=True):\n",
|
||||||
|
" # print(response.status_code)\n",
|
||||||
|
" print(response.text)\n",
|
||||||
|
"\n",
|
||||||
|
"# @markdown You can also make HTTP/GRPC request directly.\n",
|
||||||
|
"\n",
|
||||||
|
"use_http = True # @param {type:\"boolean\"}\n",
|
||||||
|
"# @markdown POST request to the following URL:\\\n",
|
||||||
|
"# @markdown `https://ENDPOINT_ID.us-central1-PROJECT_NUMBER.prediction.vertexai.goog/v1/projects/PROJECT_NUMBER/locations/LOCATION/endpoints/ENDPOINT_ID:streamRawPredict`\n",
|
||||||
|
"\n",
|
||||||
|
"if use_http:\n",
|
||||||
|
" request_path = f\"https://{endpoint.gca_resource.dedicated_endpoint_dns}/v1/{endpoint.resource_name}:streamRawPredict\"\n",
|
||||||
|
" ! curl {request_path} -X POST -H \"Content-Type: application/json\" -H \"Authorization: Bearer `gcloud auth print-access-token`\" -d \\\n",
|
||||||
|
" '{{ \\\n",
|
||||||
|
" \"instances\": [ \\\n",
|
||||||
|
" {{\"dense_input\": [14.0, 7.0, 2545.461893666405, 54.2, 48.5, 0.0, 61.3, 0.0, 0.0, 0.0]}}, \\\n",
|
||||||
|
" {{\"dense_input\": [28.0, 14.0, 1234, 27.1, 90, 0.0, 61.3, 0.0, 0.0, 0.0]}} \\\n",
|
||||||
|
" ] \\\n",
|
||||||
|
" }}'"
|
||||||
|
]
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"cell_type": "markdown",
|
||||||
|
"metadata": {
|
||||||
|
"id": "35gPjIcSfIVY"
|
||||||
|
},
|
||||||
|
"source": [
|
||||||
|
"### Chat Completion (Model Garden Only)\n",
|
||||||
|
"**You can use OpenAI client library to do chat completion.**\n",
|
||||||
|
"\n",
|
||||||
|
"```\n",
|
||||||
|
"client = openai.OpenAI(base_url=DEDICATED_DNS, api_key=creds.token)\n",
|
||||||
|
"model_response = client.chat.completions.create(\n",
|
||||||
|
" model=\"your model\",\n",
|
||||||
|
" messages=[{\"role\": \"user\", \"content\": user_message}],\n",
|
||||||
|
" temperature=temperature,\n",
|
||||||
|
" max_tokens=max_tokens,\n",
|
||||||
|
" stream=stream,\n",
|
||||||
|
")\n",
|
||||||
|
"```\n",
|
||||||
|
"\n",
|
||||||
|
"**You can also make HTTP/GRPC request directly.**\n",
|
||||||
|
"\n",
|
||||||
|
"POST request to the following URL:\n",
|
||||||
|
"```\n",
|
||||||
|
"curl -X POST -H \"Content-Type: application/json\" \\\n",
|
||||||
|
" -H \"Authorization: Bearer `gcloud auth print-access-token`\" \\\n",
|
||||||
|
" https://DEDICATED_DNS/v1beta1/projects/PROJECT_NUMBER/locations/LOCATION/endpoints/ENDPOINT_ID/chat/completions -d PAYLOAD\n",
|
||||||
|
"```\n"
|
||||||
|
]
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"cell_type": "markdown",
|
||||||
|
"metadata": {
|
||||||
|
"id": "Ka-pVhA1Xf-g"
|
||||||
|
},
|
||||||
|
"source": [
|
||||||
|
"## Supported Features"
|
||||||
|
]
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"cell_type": "markdown",
|
||||||
|
"metadata": {
|
||||||
|
"id": "_TCktMxbA4mb"
|
||||||
|
},
|
||||||
|
"source": [
|
||||||
|
"### Traffic split\n",
|
||||||
|
"\n",
|
||||||
|
"Deploy another model, and update the traffic split to be 50:50, this should take less than 5 min using the test model. After the deployment is done, you can rerun the prediction again for multiple times, you should be able to see the deployed_model_id are different."
|
||||||
|
]
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"cell_type": "code",
|
||||||
|
"execution_count": null,
|
||||||
|
"metadata": {
|
||||||
|
"id": "bVLgpRiRBEL7"
|
||||||
|
},
|
||||||
|
"outputs": [],
|
||||||
|
"source": [
|
||||||
|
"endpoint.deploy(model=model, traffic_percentage=50, machine_type=\"e2-standard-8\")"
|
||||||
|
]
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"cell_type": "code",
|
||||||
|
"execution_count": null,
|
||||||
|
"metadata": {
|
||||||
|
"id": "2dx975IkCv7v"
|
||||||
|
},
|
||||||
|
"outputs": [],
|
||||||
|
"source": [
|
||||||
|
"instances = [\n",
|
||||||
|
" {\n",
|
||||||
|
" \"dense_input\": [\n",
|
||||||
|
" 14.0,\n",
|
||||||
|
" 7.0,\n",
|
||||||
|
" 2545.461893666405,\n",
|
||||||
|
" 54.2,\n",
|
||||||
|
" 48.5,\n",
|
||||||
|
" 0.0,\n",
|
||||||
|
" 61.3,\n",
|
||||||
|
" 0.0,\n",
|
||||||
|
" 0.0,\n",
|
||||||
|
" 0.0,\n",
|
||||||
|
" ]\n",
|
||||||
|
" },\n",
|
||||||
|
" {\"dense_input\": [28.0, 14.0, 1234, 27.1, 90, 0.0, 61.3, 0.0, 0.0, 0.0]},\n",
|
||||||
|
"]\n",
|
||||||
|
"counter = {}\n",
|
||||||
|
"for i in range(1000):\n",
|
||||||
|
" response = endpoint.predict(instances=instances, use_dedicated_endpoint=True)\n",
|
||||||
|
" if response.deployed_model_id in counter.keys():\n",
|
||||||
|
" counter[response.deployed_model_id] += 1\n",
|
||||||
|
" else:\n",
|
||||||
|
" counter[response.deployed_model_id] = 1\n",
|
||||||
|
"print(counter)"
|
||||||
|
]
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"cell_type": "markdown",
|
||||||
|
"metadata": {
|
||||||
|
"id": "2Hld7iDmEyiF"
|
||||||
|
},
|
||||||
|
"source": [
|
||||||
|
"You can update the traffic split with the following command and run the code above again."
|
||||||
|
]
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"cell_type": "code",
|
||||||
|
"execution_count": null,
|
||||||
|
"metadata": {
|
||||||
|
"id": "HXPI-2q9Eh6X"
|
||||||
|
},
|
||||||
|
"outputs": [],
|
||||||
|
"source": [
|
||||||
|
"deployed_model_id_0 = list(counter)[0]\n",
|
||||||
|
"deployed_model_id_1 = list(counter)[1]\n",
|
||||||
|
"\n",
|
||||||
|
"endpoint.update(traffic_split={deployed_model_id_0: 20, deployed_model_id_1: 80})"
|
||||||
|
]
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"cell_type": "markdown",
|
||||||
|
"metadata": {
|
||||||
|
"id": "oXG3G-7D4hUJ"
|
||||||
|
},
|
||||||
|
"source": [
|
||||||
|
"### Custom Timeout\n",
|
||||||
|
"\n",
|
||||||
|
"Default 600s(10 min) timeout will be applied if the inference timeout is 0. Max timeout allowed is 1h. \\\n",
|
||||||
|
"\n",
|
||||||
|
"Use `EndpointService.UpdateEndpointLongRunning` to change the setting \\\n",
|
||||||
|
"`EndpointService.UpdateEndpoint` doesn't support request/response logging change.\n",
|
||||||
|
"\n",
|
||||||
|
"```\n",
|
||||||
|
"timeout_endpoint = aiplatform.Endpoint.create(\n",
|
||||||
|
" display_name=\"test-dedicated-endpoint-with-timeout\",\n",
|
||||||
|
" dedicated_endpoint_enabled=True,\n",
|
||||||
|
" inference_timeout=1800, # Unit: Second.\n",
|
||||||
|
")\n",
|
||||||
|
"```\n",
|
||||||
|
"Alternatively, you can create the endpoint using HTTP.\n",
|
||||||
|
"\n",
|
||||||
|
"Create endpoint with timeout sample:\n",
|
||||||
|
"```\n",
|
||||||
|
"curl -X POST -H \"Content-Type: application/json\" \\\n",
|
||||||
|
" -H \"Authorization: Bearer `gcloud auth print-access-token`\" \\\n",
|
||||||
|
" https://${DEDICATED_DNS}/v1beta1/projects/${PROJECT_NUMBER}/locations/us-central1/endpoints -d \\\n",
|
||||||
|
" '{ \\\n",
|
||||||
|
" displayName: \"test-dedicated-endpoint-with-timeout\", \\\n",
|
||||||
|
" dedicatedEndpointEnabled: true, \\\n",
|
||||||
|
" client_connection_config: \\\n",
|
||||||
|
" { \\\n",
|
||||||
|
" \"inferenceTimeout\": \\\n",
|
||||||
|
" {\"seconds\": 20, \"nanos\": 0} \\\n",
|
||||||
|
" } \\\n",
|
||||||
|
" }'\n",
|
||||||
|
"```\n",
|
||||||
|
"Update endpoint long running sample:\n",
|
||||||
|
"```\n",
|
||||||
|
"curl -X POST \\\n",
|
||||||
|
" -H \"Authorization: Bearer $(gcloud auth print-access-token)\" \\\n",
|
||||||
|
" -H \"Content-Type: application/json\" \\\n",
|
||||||
|
" https://${DEDICATED_DNS}/v1beta1/projects/${PROJECT_ID}/locations/${REGION}/endpoints/${ENDPOINT_ID}:update \\\n",
|
||||||
|
" -d '{ \\\n",
|
||||||
|
" \"endpoint\": { \\\n",
|
||||||
|
" \"clientConnectionConfig\": { \\\n",
|
||||||
|
" \"inferenceTimeout\": {\"seconds\": 5, \"nanos\": 0} \\\n",
|
||||||
|
" }, \\\n",
|
||||||
|
" } \\\n",
|
||||||
|
" }'\n",
|
||||||
|
"```"
|
||||||
|
]
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"cell_type": "markdown",
|
||||||
|
"metadata": {
|
||||||
|
"id": "yDNTq4Zh39MS"
|
||||||
|
},
|
||||||
|
"source": [
|
||||||
|
"### Request&Response Logging\n",
|
||||||
|
"\n",
|
||||||
|
"Request/Response logging will skip payload that exceed 10M which is the big query limit.\n",
|
||||||
|
"\n",
|
||||||
|
"Use `EndpointService.UpdateEndpointLongRunning` to change the setting \\\n",
|
||||||
|
"`EndpointService.UpdateEndpoint` doesn't support request/response logging change.\n",
|
||||||
|
"\n",
|
||||||
|
"```\n",
|
||||||
|
"logging_endpoint = aiplatform.Endpoint.create(\n",
|
||||||
|
" display_name=\"test-dedicated-endpoint-with-logging\",\n",
|
||||||
|
" dedicated_endpoint_enabled=True,\n",
|
||||||
|
" enable_request_response_logging=True,\n",
|
||||||
|
" request_response_logging_sampling_rate=1.0, # default 0.0\n",
|
||||||
|
" request_response_logging_bq_destination_table=\"bq://test_logging\",\n",
|
||||||
|
" # if not set, a new table will be created with name `bq://{project_id}.logging_{endpoint_display_name}_{endpoint_id}.request_response_logging`\n",
|
||||||
|
")\n",
|
||||||
|
"```\n",
|
||||||
|
"\n",
|
||||||
|
"Alternatively, use HTTP directly.\n",
|
||||||
|
"\n",
|
||||||
|
"Create endpoint sample\n",
|
||||||
|
"```\n",
|
||||||
|
"curl -X POST \\\n",
|
||||||
|
" -H \"Authorization: Bearer $(gcloud auth print-access-token)\" -H \"Content-Type: application/json\" \\\n",
|
||||||
|
" https://${DEDICATED_DNS}/v1beta1/projects/${PROJECT_ID}/locations/us-central1/endpoints \\\n",
|
||||||
|
" -d '{ \\\n",
|
||||||
|
" \"display_name\": \"test-dedicated-endpoint-with-logging\", \\\n",
|
||||||
|
" \"dedicatedEndpointEnabled\": true, \\\n",
|
||||||
|
" \"predict_request_response_logging_config\": { \\\n",
|
||||||
|
" \"enabled\": true, \\\n",
|
||||||
|
" \"sampling_rate\": 0.5, \\\n",
|
||||||
|
" \"bigquery_destination\": { \\\n",
|
||||||
|
" \"output_uri\": \"bq://my-project\" \\\n",
|
||||||
|
" } \\\n",
|
||||||
|
" } \\\n",
|
||||||
|
" }'\n",
|
||||||
|
"```\n",
|
||||||
|
"\n",
|
||||||
|
"Update endpoint long running request sample\n",
|
||||||
|
"```\n",
|
||||||
|
"curl -X POST \\\n",
|
||||||
|
" -H \"Authorization: Bearer $(gcloud auth print-access-token)\" \\\n",
|
||||||
|
" -H \"Content-Type: application/json\" \\\n",
|
||||||
|
" https://${DEDICATED_DNS}/v1beta1/projects/${PROJECT_ID}/locations/${REGION}/endpoints/${ENDPOINT_ID}:update \\\n",
|
||||||
|
" -d '{ \\\n",
|
||||||
|
" \"endpoint\": { \\\n",
|
||||||
|
" \"predict_request_response_logging_config\": { \\\n",
|
||||||
|
" \"enabled\": true, \\\n",
|
||||||
|
" \"sampling_rate\": 0.5, \\\n",
|
||||||
|
" \"bigquery_destination\": { \\\n",
|
||||||
|
" \"output_uri\": \"bq://my-project\" \\\n",
|
||||||
|
" } \\\n",
|
||||||
|
" }, \\\n",
|
||||||
|
" } \\\n",
|
||||||
|
" }'\n",
|
||||||
|
"```\n"
|
||||||
|
]
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"cell_type": "markdown",
|
||||||
|
"metadata": {
|
||||||
|
"id": "XW_BtPnEFPp4"
|
||||||
|
},
|
||||||
|
"source": [
|
||||||
|
"## Cleanup"
|
||||||
|
]
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"cell_type": "code",
|
||||||
|
"execution_count": null,
|
||||||
|
"metadata": {
|
||||||
|
"id": "E4Ik3eKqdI_2"
|
||||||
|
},
|
||||||
|
"outputs": [],
|
||||||
|
"source": [
|
||||||
|
"endpoint.undeploy_all()\n",
|
||||||
|
"endpoint.delete()\n",
|
||||||
|
"model.delete()"
|
||||||
|
]
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"cell_type": "markdown",
|
||||||
|
"metadata": {
|
||||||
|
"id": "BTz-2N1XunXB"
|
||||||
|
},
|
||||||
|
"source": [
|
||||||
|
"Delete the bucket if needed."
|
||||||
|
]
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"cell_type": "code",
|
||||||
|
"execution_count": null,
|
||||||
|
"metadata": {
|
||||||
|
"id": "NPQT5Wv9lC3O"
|
||||||
|
},
|
||||||
|
"outputs": [],
|
||||||
|
"source": [
|
||||||
|
"! gsutil rm -r {BUCKET_URI}"
|
||||||
|
]
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"cell_type": "markdown",
|
||||||
|
"metadata": {
|
||||||
|
"id": "2iR_Q5K_ksWi"
|
||||||
|
},
|
||||||
|
"source": [
|
||||||
|
"Optionally, you can use the following command to clean up all endpoint and models if needed."
|
||||||
|
]
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"cell_type": "code",
|
||||||
|
"execution_count": null,
|
||||||
|
"metadata": {
|
||||||
|
"id": "vkyvlwTgky0J"
|
||||||
|
},
|
||||||
|
"outputs": [],
|
||||||
|
"source": [
|
||||||
|
"# for e in aiplatform.Endpoint.list():\n",
|
||||||
|
"# e.undeploy_all()\n",
|
||||||
|
"# e.delete()"
|
||||||
|
]
|
||||||
|
}
|
||||||
|
],
|
||||||
|
"metadata": {
|
||||||
|
"colab": {
|
||||||
|
"name": "get_started_with_dedicated_endpoint.ipynb",
|
||||||
|
"toc_visible": true
|
||||||
|
},
|
||||||
|
"kernelspec": {
|
||||||
|
"display_name": "Python 3",
|
||||||
|
"name": "python3"
|
||||||
|
}
|
||||||
|
},
|
||||||
|
"nbformat": 4,
|
||||||
|
"nbformat_minor": 0
|
||||||
|
}
|
||||||
Reference in New Issue
Block a user