From 22709379dd2eda5f1f3609607de1693b43676d7b Mon Sep 17 00:00:00 2001 From: TJ-Liu Date: Wed, 14 May 2025 18:14:40 -0700 Subject: [PATCH] feat: Add prediction dedicated endpoint colab sample (#3942) * feat: Add prediction dedicated endpoint colab sample * Update get_started_with_dedicated_endpoint.ipynb --------- Co-authored-by: TJ(Tianjiao) Liu --- notebooks/official/CODEOWNERS | 1 + .../get_started_with_dedicated_endpoint.ipynb | 827 ++++++++++++++++++ 2 files changed, 828 insertions(+) create mode 100644 notebooks/official/prediction/get_started_with_dedicated_endpoint.ipynb diff --git a/notebooks/official/CODEOWNERS b/notebooks/official/CODEOWNERS index cc1e13d50..44bb96a2e 100644 --- a/notebooks/official/CODEOWNERS +++ b/notebooks/official/CODEOWNERS @@ -57,6 +57,7 @@ /training/tpuv5e_gemma_peft_finetuning_and_serving.ipynb @brianchunkang /training/tpuv5e_llama2_pytorch_finetuning_and_serving.ipynb @brianchunkang @chiefkarlin /prediction/get_started_with_psc_private_endpoint.ipynb @tianjiaoliu +/prediction/get_started_with_dedicated_endpoint.ipynb @tianjiaoliu /ray_on_vertex_ai/spark_on_ray_on_vertex_ai.ipynb @ravi-dalal /generative_ai/mistralai_intro.ipynb @sujituk /generative_ai/ai21labs_intro.ipynb @miri-bar diff --git a/notebooks/official/prediction/get_started_with_dedicated_endpoint.ipynb b/notebooks/official/prediction/get_started_with_dedicated_endpoint.ipynb new file mode 100644 index 000000000..f758701a8 --- /dev/null +++ b/notebooks/official/prediction/get_started_with_dedicated_endpoint.ipynb @@ -0,0 +1,827 @@ +{ + "cells": [ + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "UUTFwSh5u9OK" + }, + "outputs": [], + "source": [ + "# Copyright 2024 Google LLC\n", + "#\n", + "# Licensed under the Apache License, Version 2.0 (the \"License\");\n", + "# you may not use this file except in compliance with the License.\n", + "# You may obtain a copy of the License at\n", + "#\n", + "# https://www.apache.org/licenses/LICENSE-2.0\n", + "#\n", + "# Unless required by applicable law or agreed to in writing, software\n", + "# distributed under the License is distributed on an \"AS IS\" BASIS,\n", + "# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n", + "# See the License for the specific language governing permissions and\n", + "# limitations under the License." + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "qsLff0QbdE8D" + }, + "source": [ + "# Online Prediction Dedicated Endpint\n", + "\n", + "\n", + " \n", + " \n", + " \n", + " \n", + "
\n", + " \n", + " \"Google
Open in Colab\n", + " \n", + " 		\n", + " \n", + " \"Google
Open in Colab Enterprise\n", + " \n", + " 		\n", + " \n", + " \"Vertex
Open in Workbench\n", + " \n", + " 		\n", + " \n", + " \"GitHub
View on GitHub\n", + " \n", + "
" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "0C7kTlrH1bO8" + }, + "source": [ + "## Overview\n", + "\n", + "A dedicated public endpoint is a public endpoint for online prediction. It offers the following benefits:\n", + "\n", + "* Dedicated networking: When you send a prediction request to a dedicated public endpoint, it is isolated from other users' traffic.\n", + "* Optimized network latency\n", + "* Larger payload support: Up to 10 MB.\n", + "* Longer request timeouts: Configurable up to 1 hour.\n", + "* Generative AI-ready: Streaming and gRPC are supported." + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "D2sWuTr81v9m" + }, + "source": [ + "## Get started" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "RakMIliNYh8O" + }, + "source": [ + "### Install Vertex AI SDK for Python and other required packages" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "nSHmJT9cTggu" + }, + "outputs": [], + "source": [ + "! pip3 install --upgrade --quiet google-cloud-aiplatform" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "936Zz5YI2NeA" + }, + "source": [ + "### Restart runtime (Colab only)\n", + "\n", + "To use the newly installed packages, you must restart the runtime on Google Colab." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "66oJ55lG2Tiq" + }, + "outputs": [], + "source": [ + "import sys\n", + "\n", + "if \"google.colab\" in sys.modules:\n", + "\n", + " import IPython\n", + "\n", + " app = IPython.Application.instance()\n", + " app.kernel.do_shutdown(True)" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "Clr61ben2WwY" + }, + "source": [ + "
\n", + "⚠️ The kernel is going to restart. Wait until it's finished before continuing to the next step. ⚠️\n", + "
\n" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "v848aGbn2acH" + }, + "source": [ + "### Authenticate your notebook environment (Colab only)\n", + "\n", + "Authenticate your environment on Google Colab.\n" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "IVeoyQPz2cfh" + }, + "outputs": [], + "source": [ + "import sys\n", + "\n", + "if \"google.colab\" in sys.modules:\n", + "\n", + " from google.colab import auth\n", + "\n", + " auth.authenticate_user()" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "HId-ySlY2jlI" + }, + "source": [ + "### Set Google Cloud project information and initialize Vertex AI SDK for Python\n", + "\n", + "To get started using Vertex AI, you must have an existing Google Cloud project and [enable the Vertex AI API](https://console.cloud.google.com/flows/enableapi?apiid=aiplatform.googleapis.com). Learn more about [setting up a project and a development environment](https://cloud.google.com/vertex-ai/docs/start/cloud-environment)." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "Y4gnZI9OX6VJ" + }, + "outputs": [], + "source": [ + "# Set project id to be the current project ID.\n", + "\n", + "PROJECT_ID = \"[your-project-id]\" # @param {type:\"string\"}\n", + "LOCATION = \"us-central1\" # @param {type:\"string\"}" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "-NrpFROTjoVL" + }, + "outputs": [], + "source": [ + "from google.cloud import aiplatform\n", + "\n", + "aiplatform.init(project=PROJECT_ID, location=LOCATION)" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "moS794OKaaCt" + }, + "outputs": [], + "source": [ + "# Create GCS Bucket\n", + "BUCKET_URI = \"gs://your-bucket-name-unique\" # @param {type:\"string\"}\n", + "! gsutil mb -l {LOCATION} -p {PROJECT_ID} {BUCKET_URI}" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "SASm5tKB2JGF" + }, + "source": [ + "## Create Dedicated Endpoint and Deploy Model" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "taUmNhwP2S3W" + }, + "source": [ + "### Create Dedicated Endpoint\n", + "When creating endpoint, set `dedicated_endpoint_enabled` to True." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "IZUfl8Q82crP" + }, + "outputs": [], + "source": [ + "endpoint = aiplatform.Endpoint.create(\n", + " display_name=\"test-dedicated-endpoint\",\n", + " dedicated_endpoint_enabled=True,\n", + ")" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "swERjzZ-a_Nd" + }, + "source": [ + "### Prepare Test Models\n", + "\n", + "We prepared a tensorflow test model, feel free to use your own models." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "0fea4d3018ab" + }, + "outputs": [], + "source": [ + "! pip freeze | grep google-cloud-aiplatform" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "bA2zhbzJ1R0k" + }, + "outputs": [], + "source": [ + "# List all your models\n", + "for my_model in aiplatform.Model.list():\n", + " print(my_model.display_name)\n", + " print(my_model.gca_resource.name)\n", + "\n", + "# If you want to use an existing model, use the resource id\n", + "# model = aiplatform.Model('projects/12345/locations/us-central1/models/456789')" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "3PtFCQNHbloQ" + }, + "outputs": [], + "source": [ + "# @title Upload a new tensorflow model\n", + "# @markdown You can skip this if use an existing model.\n", + "\n", + "# TF Model\n", + "DISPLAY_NAME = \"tensorflow model\" # @param {type:\"string\"}\n", + "ARTIFACT_URI = BUCKET_URI + \"/tensorflow\"\n", + "IMAGE_URI = \"us-docker.pkg.dev/vertex-ai/prediction/tf2-cpu.2-12:latest\"\n", + "\n", + "# Copy Test Models to the Bucket\n", + "! gsutil -m cp -r \"gs://cloud-samples-data/vertex-ai/prediction/test-models-requests/tensorflow/*\" {ARTIFACT_URI}\n", + "\n", + "model = aiplatform.Model.upload(\n", + " display_name=DISPLAY_NAME,\n", + " artifact_uri=ARTIFACT_URI,\n", + " serving_container_image_uri=IMAGE_URI,\n", + ")" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "USWCW-SNo-9M" + }, + "source": [ + "#### Deploy Model\n", + "\n", + "Deploy the model to the endpoint. If it is your first model with dedicated endpoint, it will take ~30 min." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "S_eRJglhpVfL" + }, + "outputs": [], + "source": [ + "endpoint.deploy(model=model, traffic_percentage=100, machine_type=\"e2-standard-8\")" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "civyNQaPr4QD" + }, + "source": [ + "## Make Predictions\n", + "\n", + "Dedicated endpoint cannot be accessed with the shared `aiplatform.googleapis.com` DNS. \\\n", + "Instead, it has its own dedicated DNS\n", + "```\n", + "DEDICATED_DNS = f\"https://{endpoint.gca_resource.dedicated_endpoint_dns}/v1beta1/{endpoint.resource_name}\"\n", + "# or\n", + "DEDICATED_DNS = f\"https://{ENDPOINT_ID}.{LOCATION}-{PROJECT_NUMBER}.prediction.vertexai.goog\"\n", + "```\n", + "\n", + "Python SDK has been integrated to support dedicated endpoint using a flag `use_dedicated_endpoint`.\n", + "\n", + "Alternatively, you can send HTTP/GRPC request directly to this DNS in any language you prefer." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "cellView": "form", + "id": "I-O9U63juoWE" + }, + "outputs": [], + "source": [ + "# @title Predict\n", + "# @markdown You can use client library.\n", + "\n", + "use_python_sdk = True # @param {type:\"boolean\"}\n", + "# @markdown response = my_endpoint.predict( \\\n", + "# @markdown    instances=[{\"feat_1\":val_1, \"feat_2\":val_2}]}, \\\n", + "# @markdown    headers = {'Content-Type':'application/json'}, \\\n", + "# @markdown    dedicated_endpoint=True, \\\n", + "# @markdown )\n", + "\n", + "if use_python_sdk:\n", + " instances = [\n", + " {\"dense_input\": [14.0, 7.0, 2545.461893666405, 54.2, 48.5, 0.0, 61.3, 0.0, 0.0, 0.0]},\n", + " {\"dense_input\": [28.0, 14.0, 1234, 27.1, 90, 0.0, 61.3, 0.0, 0.0, 0.0]}\n", + " ]\n", + " response = endpoint.predict(instances=instances, use_dedicated_endpoint=True)\n", + " print(response)\n", + "\n", + "# @markdown You can also make HTTP/GRPC request directly.\n", + "\n", + "use_http = True # @param {type:\"boolean\"}\n", + "# @markdown POST request to the following URL:\\\n", + "# @markdown `https://ENDPOINT_ID.us-central1-PROJECT_NUMBER.prediction.vertexai.goog/v1/projects/PROJECT_NUMBER/locations/LOCATION/endpoints/ENDPOINT_ID:predict`\n", + "\n", + "if use_http:\n", + " request_path = f\"https://{endpoint.gca_resource.dedicated_endpoint_dns}/v1/{endpoint.resource_name}:predict\"\n", + " ! curl {request_path} -X POST -H \"Content-Type: application/json\" -H \"Authorization: Bearer `gcloud auth print-access-token`\" -d \\\n", + " '{{ \\\n", + " \"instances\": [ \\\n", + " {{\"dense_input\": [14.0, 7.0, 2545.461893666405, 54.2, 48.5, 0.0, 61.3, 0.0, 0.0, 0.0]}}, \\\n", + " {{\"dense_input\": [28.0, 14.0, 1234, 27.1, 90, 0.0, 61.3, 0.0, 0.0, 0.0]}} \\\n", + " ] \\\n", + " }}'" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "cellView": "form", + "id": "RRoiziQ9FAw6" + }, + "outputs": [], + "source": [ + "# @title Raw Predict\n", + "# @markdown You can use client library.\n", + "\n", + "use_python_sdk = True # @param {type:\"boolean\"}\n", + "# @markdown response = my_endpoint.raw_predict( \\\n", + "# @markdown     body = b'{\"instances\":[{\"feat_1\":val_1, \"feat_2\":val_2}]}', \\\n", + "# @markdown     headers = {'Content-Type':'application/json'}, \\\n", + "# @markdown     dedicated_endpoint=True, \\\n", + "# @markdown ) \\\n", + "# @markdown status_code = response.status_code \\\n", + "# @markdown results = json.dumps(response.text)\n", + "\n", + "if use_python_sdk:\n", + " body = b'{ \\\n", + " \"instances\": [ \\\n", + " {\"dense_input\": [14.0, 7.0, 2545.461893666405, 54.2, 48.5, 0.0, 61.3, 0.0, 0.0, 0.0]}, \\\n", + " {\"dense_input\": [28.0, 14.0, 1234, 27.1, 90, 0.0, 61.3, 0.0, 0.0, 0.0]} \\\n", + " ]\\\n", + " }'\n", + " import os\n", + " token = !gcloud auth print-access-token\n", + " token = token[0]\n", + " headers = {\"content-type\": \"application/json\", \"Authorization\": f\"Bearer {token}\"}\n", + " response = endpoint.raw_predict(body=body, headers=headers, use_dedicated_endpoint=True)\n", + " # print(response.status_code)\n", + " print(response.text)\n", + "\n", + "# @markdown You can also make HTTP/GRPC request directly.\n", + "\n", + "use_http = True # @param {type:\"boolean\"}\n", + "# @markdown POST request to the following URL:\\\n", + "# @markdown `https://ENDPOINT_ID.us-central1-PROJECT_NUMBER.prediction.vertexai.goog/v1/projects/PROJECT_NUMBER/locations/LOCATION/endpoints/ENDPOINT_ID:rawPredict`\n", + "\n", + "if use_http:\n", + " request_path = f\"https://{endpoint.gca_resource.dedicated_endpoint_dns}/v1/{endpoint.resource_name}:rawPredict\"\n", + " ! curl {request_path} -X POST -H \"Content-Type: application/json\" -H \"Authorization: Bearer `gcloud auth print-access-token`\" -d \\\n", + " '{{ \\\n", + " \"instances\": [ \\\n", + " {{\"dense_input\": [14.0, 7.0, 2545.461893666405, 54.2, 48.5, 0.0, 61.3, 0.0, 0.0, 0.0]}}, \\\n", + " {{\"dense_input\": [28.0, 14.0, 1234, 27.1, 90, 0.0, 61.3, 0.0, 0.0, 0.0]}} \\\n", + " ] \\\n", + " }}'" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "cellView": "form", + "id": "IbZDIJlIRl1b" + }, + "outputs": [], + "source": [ + "# @title Stream Raw Predict\n", + "# @markdown You can use client library.\n", + "\n", + "use_python_sdk = True # @param {type:\"boolean\"}\n", + "# @markdown for stream_response in my_endpoint.stream_raw_predict( \\\n", + "# @markdown     body = b'{\"instances\":[{\"feat_1\":val_1, \"feat_2\":val_2}]}', \\\n", + "# @markdown     headers = {'Content-Type':'application/json'}, \\\n", + "# @markdown     use_dedicated_endpoint=True, \\\n", + "# @markdown ): \\\n", + "# @markdown     status_code = response.status_code \\\n", + "# @markdown     stream_result = json.dumps(response.text)\n", + "\n", + "if use_python_sdk:\n", + " body = b'{ \\\n", + " \"instances\": [ \\\n", + " {\"dense_input\": [14.0, 7.0, 2545.461893666405, 54.2, 48.5, 0.0, 61.3, 0.0, 0.0, 0.0]}, \\\n", + " {\"dense_input\": [28.0, 14.0, 1234, 27.1, 90, 0.0, 61.3, 0.0, 0.0, 0.0]} \\\n", + " ]\\\n", + " }'\n", + " import os\n", + " token = !gcloud auth print-access-token\n", + " token = token[0]\n", + " headers = {\"content-type\": \"application/json\", \"Authorization\": f\"Bearer {token}\"}\n", + " for stream_response in endpoint.raw_predict(body=body, headers=headers, use_dedicated_endpoint=True):\n", + " # print(response.status_code)\n", + " print(response.text)\n", + "\n", + "# @markdown You can also make HTTP/GRPC request directly.\n", + "\n", + "use_http = True # @param {type:\"boolean\"}\n", + "# @markdown POST request to the following URL:\\\n", + "# @markdown `https://ENDPOINT_ID.us-central1-PROJECT_NUMBER.prediction.vertexai.goog/v1/projects/PROJECT_NUMBER/locations/LOCATION/endpoints/ENDPOINT_ID:streamRawPredict`\n", + "\n", + "if use_http:\n", + " request_path = f\"https://{endpoint.gca_resource.dedicated_endpoint_dns}/v1/{endpoint.resource_name}:streamRawPredict\"\n", + " ! curl {request_path} -X POST -H \"Content-Type: application/json\" -H \"Authorization: Bearer `gcloud auth print-access-token`\" -d \\\n", + " '{{ \\\n", + " \"instances\": [ \\\n", + " {{\"dense_input\": [14.0, 7.0, 2545.461893666405, 54.2, 48.5, 0.0, 61.3, 0.0, 0.0, 0.0]}}, \\\n", + " {{\"dense_input\": [28.0, 14.0, 1234, 27.1, 90, 0.0, 61.3, 0.0, 0.0, 0.0]}} \\\n", + " ] \\\n", + " }}'" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "35gPjIcSfIVY" + }, + "source": [ + "### Chat Completion (Model Garden Only)\n", + "**You can use OpenAI client library to do chat completion.**\n", + "\n", + "```\n", + "client = openai.OpenAI(base_url=DEDICATED_DNS, api_key=creds.token)\n", + "model_response = client.chat.completions.create(\n", + " model=\"your model\",\n", + " messages=[{\"role\": \"user\", \"content\": user_message}],\n", + " temperature=temperature,\n", + " max_tokens=max_tokens,\n", + " stream=stream,\n", + ")\n", + "```\n", + "\n", + "**You can also make HTTP/GRPC request directly.**\n", + "\n", + "POST request to the following URL:\n", + "```\n", + "curl -X POST -H \"Content-Type: application/json\" \\\n", + " -H \"Authorization: Bearer `gcloud auth print-access-token`\" \\\n", + " https://DEDICATED_DNS/v1beta1/projects/PROJECT_NUMBER/locations/LOCATION/endpoints/ENDPOINT_ID/chat/completions -d PAYLOAD\n", + "```\n" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "Ka-pVhA1Xf-g" + }, + "source": [ + "## Supported Features" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "_TCktMxbA4mb" + }, + "source": [ + "### Traffic split\n", + "\n", + "Deploy another model, and update the traffic split to be 50:50, this should take less than 5 min using the test model. After the deployment is done, you can rerun the prediction again for multiple times, you should be able to see the deployed_model_id are different." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "bVLgpRiRBEL7" + }, + "outputs": [], + "source": [ + "endpoint.deploy(model=model, traffic_percentage=50, machine_type=\"e2-standard-8\")" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "2dx975IkCv7v" + }, + "outputs": [], + "source": [ + "instances = [\n", + " {\n", + " \"dense_input\": [\n", + " 14.0,\n", + " 7.0,\n", + " 2545.461893666405,\n", + " 54.2,\n", + " 48.5,\n", + " 0.0,\n", + " 61.3,\n", + " 0.0,\n", + " 0.0,\n", + " 0.0,\n", + " ]\n", + " },\n", + " {\"dense_input\": [28.0, 14.0, 1234, 27.1, 90, 0.0, 61.3, 0.0, 0.0, 0.0]},\n", + "]\n", + "counter = {}\n", + "for i in range(1000):\n", + " response = endpoint.predict(instances=instances, use_dedicated_endpoint=True)\n", + " if response.deployed_model_id in counter.keys():\n", + " counter[response.deployed_model_id] += 1\n", + " else:\n", + " counter[response.deployed_model_id] = 1\n", + "print(counter)" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "2Hld7iDmEyiF" + }, + "source": [ + "You can update the traffic split with the following command and run the code above again." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "HXPI-2q9Eh6X" + }, + "outputs": [], + "source": [ + "deployed_model_id_0 = list(counter)[0]\n", + "deployed_model_id_1 = list(counter)[1]\n", + "\n", + "endpoint.update(traffic_split={deployed_model_id_0: 20, deployed_model_id_1: 80})" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "oXG3G-7D4hUJ" + }, + "source": [ + "### Custom Timeout\n", + "\n", + "Default 600s(10 min) timeout will be applied if the inference timeout is 0. Max timeout allowed is 1h. \\\n", + "\n", + "Use `EndpointService.UpdateEndpointLongRunning` to change the setting \\\n", + "`EndpointService.UpdateEndpoint` doesn't support request/response logging change.\n", + "\n", + "```\n", + "timeout_endpoint = aiplatform.Endpoint.create(\n", + " display_name=\"test-dedicated-endpoint-with-timeout\",\n", + " dedicated_endpoint_enabled=True,\n", + " inference_timeout=1800, # Unit: Second.\n", + ")\n", + "```\n", + "Alternatively, you can create the endpoint using HTTP.\n", + "\n", + "Create endpoint with timeout sample:\n", + "```\n", + "curl -X POST -H \"Content-Type: application/json\" \\\n", + " -H \"Authorization: Bearer `gcloud auth print-access-token`\" \\\n", + " https://${DEDICATED_DNS}/v1beta1/projects/${PROJECT_NUMBER}/locations/us-central1/endpoints -d \\\n", + " '{ \\\n", + " displayName: \"test-dedicated-endpoint-with-timeout\", \\\n", + " dedicatedEndpointEnabled: true, \\\n", + " client_connection_config: \\\n", + " { \\\n", + " \"inferenceTimeout\": \\\n", + " {\"seconds\": 20, \"nanos\": 0} \\\n", + " } \\\n", + " }'\n", + "```\n", + "Update endpoint long running sample:\n", + "```\n", + "curl -X POST \\\n", + " -H \"Authorization: Bearer $(gcloud auth print-access-token)\" \\\n", + " -H \"Content-Type: application/json\" \\\n", + " https://${DEDICATED_DNS}/v1beta1/projects/${PROJECT_ID}/locations/${REGION}/endpoints/${ENDPOINT_ID}:update \\\n", + " -d '{ \\\n", + " \"endpoint\": { \\\n", + " \"clientConnectionConfig\": { \\\n", + " \"inferenceTimeout\": {\"seconds\": 5, \"nanos\": 0} \\\n", + " }, \\\n", + " } \\\n", + " }'\n", + "```" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "yDNTq4Zh39MS" + }, + "source": [ + "### Request&Response Logging\n", + "\n", + "Request/Response logging will skip payload that exceed 10M which is the big query limit.\n", + "\n", + "Use `EndpointService.UpdateEndpointLongRunning` to change the setting \\\n", + "`EndpointService.UpdateEndpoint` doesn't support request/response logging change.\n", + "\n", + "```\n", + "logging_endpoint = aiplatform.Endpoint.create(\n", + " display_name=\"test-dedicated-endpoint-with-logging\",\n", + " dedicated_endpoint_enabled=True,\n", + " enable_request_response_logging=True,\n", + " request_response_logging_sampling_rate=1.0, # default 0.0\n", + " request_response_logging_bq_destination_table=\"bq://test_logging\",\n", + " # if not set, a new table will be created with name `bq://{project_id}.logging_{endpoint_display_name}_{endpoint_id}.request_response_logging`\n", + ")\n", + "```\n", + "\n", + "Alternatively, use HTTP directly.\n", + "\n", + "Create endpoint sample\n", + "```\n", + "curl -X POST \\\n", + " -H \"Authorization: Bearer $(gcloud auth print-access-token)\" -H \"Content-Type: application/json\" \\\n", + " https://${DEDICATED_DNS}/v1beta1/projects/${PROJECT_ID}/locations/us-central1/endpoints \\\n", + " -d '{ \\\n", + " \"display_name\": \"test-dedicated-endpoint-with-logging\", \\\n", + " \"dedicatedEndpointEnabled\": true, \\\n", + " \"predict_request_response_logging_config\": { \\\n", + " \"enabled\": true, \\\n", + " \"sampling_rate\": 0.5, \\\n", + " \"bigquery_destination\": { \\\n", + " \"output_uri\": \"bq://my-project\" \\\n", + " } \\\n", + " } \\\n", + " }'\n", + "```\n", + "\n", + "Update endpoint long running request sample\n", + "```\n", + "curl -X POST \\\n", + " -H \"Authorization: Bearer $(gcloud auth print-access-token)\" \\\n", + " -H \"Content-Type: application/json\" \\\n", + " https://${DEDICATED_DNS}/v1beta1/projects/${PROJECT_ID}/locations/${REGION}/endpoints/${ENDPOINT_ID}:update \\\n", + " -d '{ \\\n", + " \"endpoint\": { \\\n", + " \"predict_request_response_logging_config\": { \\\n", + " \"enabled\": true, \\\n", + " \"sampling_rate\": 0.5, \\\n", + " \"bigquery_destination\": { \\\n", + " \"output_uri\": \"bq://my-project\" \\\n", + " } \\\n", + " }, \\\n", + " } \\\n", + " }'\n", + "```\n" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "XW_BtPnEFPp4" + }, + "source": [ + "## Cleanup" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "E4Ik3eKqdI_2" + }, + "outputs": [], + "source": [ + "endpoint.undeploy_all()\n", + "endpoint.delete()\n", + "model.delete()" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "BTz-2N1XunXB" + }, + "source": [ + "Delete the bucket if needed." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "NPQT5Wv9lC3O" + }, + "outputs": [], + "source": [ + "! gsutil rm -r {BUCKET_URI}" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "2iR_Q5K_ksWi" + }, + "source": [ + "Optionally, you can use the following command to clean up all endpoint and models if needed." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "vkyvlwTgky0J" + }, + "outputs": [], + "source": [ + "# for e in aiplatform.Endpoint.list():\n", + "# e.undeploy_all()\n", + "# e.delete()" + ] + } + ], + "metadata": { + "colab": { + "name": "get_started_with_dedicated_endpoint.ipynb", + "toc_visible": true + }, + "kernelspec": { + "display_name": "Python 3", + "name": "python3" + } + }, + "nbformat": 4, + "nbformat_minor": 0 +}