{
  "cells": [
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "UUTFwSh5u9OK"
      },
      "outputs": [],
      "source": [
        "# Copyright 2024 Google LLC\n",
        "#\n",
        "# Licensed under the Apache License, Version 2.0 (the \"License\");\n",
        "# you may not use this file except in compliance with the License.\n",
        "# You may obtain a copy of the License at\n",
        "#\n",
        "#     https://www.apache.org/licenses/LICENSE-2.0\n",
        "#\n",
        "# Unless required by applicable law or agreed to in writing, software\n",
        "# distributed under the License is distributed on an \"AS IS\" BASIS,\n",
        "# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n",
        "# See the License for the specific language governing permissions and\n",
        "# limitations under the License."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "qsLff0QbdE8D"
      },
      "source": [
        "# Online Prediction Dedicated Endpint\n",
        "\n",
        "<table align=\"left\">\n",
        "  <td style=\"text-align: center\">\n",
        "    <a href=\"https://colab.research.google.com/github/GoogleCloudPlatform/vertex-ai-samples/blob/main/notebooks/official/prediction/get_started_with_dedicated_endpoint.ipynb\">\n",
        "      <img src=\"https://www.gstatic.com/pantheon/images/bigquery/welcome_page/colab-logo.svg\" alt=\"Google Colaboratory logo\"><br> Open in Colab\n",
        "    </a>\n",
        "  </td>\n",
        "  <td style=\"text-align: center\">\n",
        "    <a href=\"https://console.cloud.google.com/vertex-ai/colab/import/https:%2F%2Fraw.githubusercontent.com%2FGoogleCloudPlatform%2Fvertex-ai-samples%2Fmain%2Fnotebooks%2Fofficial%2Fprediction%2Fget_started_with_dedicated_endpoint.ipynb\">\n",
        "      <img width=\"32px\" src=\"https://lh3.googleusercontent.com/JmcxdQi-qOpctIvWKgPtrzZdJJK-J3sWE1RsfjZNwshCFgE_9fULcNpuXYTilIR2hjwN\" alt=\"Google Cloud Colab Enterprise logo\"><br> Open in Colab Enterprise\n",
        "    </a>\n",
        "  </td>    \n",
        "  <td style=\"text-align: center\">\n",
        "    <a href=\"https://console.cloud.google.com/vertex-ai/workbench/deploy-notebook?download_url=https://raw.githubusercontent.com/GoogleCloudPlatform/vertex-ai-samples/main/notebooks/official/prediction/get_started_with_dedicated_endpoint.ipynb\">\n",
        "      <img src=\"https://www.gstatic.com/images/branding/gcpiconscolors/vertexai/v1/32px.svg\" alt=\"Vertex AI logo\"><br> Open in Workbench\n",
        "    </a>\n",
        "  </td>\n",
        "  <td style=\"text-align: center\">\n",
        "    <a href=\"https://github.com/GoogleCloudPlatform/vertex-ai-samples/blob/main/notebooks/official/prediction/get_started_with_dedicated_endpoint.ipynb\">\n",
        "      <img width=\"32px\" src=\"https://www.svgrepo.com/download/217753/github.svg\" alt=\"GitHub logo\"><br> View on GitHub\n",
        "    </a>\n",
        "  </td>\n",
        "</table>"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "0C7kTlrH1bO8"
      },
      "source": [
        "## Overview\n",
        "\n",
        "A dedicated public endpoint is a public endpoint for online prediction. It offers the following benefits:\n",
        "\n",
        "* Dedicated networking: When you send a prediction request to a dedicated public endpoint, it is isolated from other users' traffic.\n",
        "* Optimized network latency\n",
        "* Larger payload support: Up to 10 MB.\n",
        "* Longer request timeouts: Configurable up to 1 hour.\n",
        "* Generative AI-ready: Streaming and gRPC are supported."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "D2sWuTr81v9m"
      },
      "source": [
        "## Get started"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "RakMIliNYh8O"
      },
      "source": [
        "### Install Vertex AI SDK for Python and other required packages"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "nSHmJT9cTggu"
      },
      "outputs": [],
      "source": [
        "! pip3 install --upgrade --quiet google-cloud-aiplatform"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "936Zz5YI2NeA"
      },
      "source": [
        "### Restart runtime (Colab only)\n",
        "\n",
        "To use the newly installed packages, you must restart the runtime on Google Colab."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "66oJ55lG2Tiq"
      },
      "outputs": [],
      "source": [
        "import sys\n",
        "\n",
        "if \"google.colab\" in sys.modules:\n",
        "\n",
        "    import IPython\n",
        "\n",
        "    app = IPython.Application.instance()\n",
        "    app.kernel.do_shutdown(True)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Clr61ben2WwY"
      },
      "source": [
        "<div class=\"alert alert-block alert-warning\">\n",
        "<b>⚠️ The kernel is going to restart. Wait until it's finished before continuing to the next step. ⚠️</b>\n",
        "</div>\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "v848aGbn2acH"
      },
      "source": [
        "### Authenticate your notebook environment (Colab only)\n",
        "\n",
        "Authenticate your environment on Google Colab.\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "IVeoyQPz2cfh"
      },
      "outputs": [],
      "source": [
        "import sys\n",
        "\n",
        "if \"google.colab\" in sys.modules:\n",
        "\n",
        "    from google.colab import auth\n",
        "\n",
        "    auth.authenticate_user()"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "HId-ySlY2jlI"
      },
      "source": [
        "### Set Google Cloud project information and initialize Vertex AI SDK for Python\n",
        "\n",
        "To get started using Vertex AI, you must have an existing Google Cloud project and [enable the Vertex AI API](https://console.cloud.google.com/flows/enableapi?apiid=aiplatform.googleapis.com). Learn more about [setting up a project and a development environment](https://cloud.google.com/vertex-ai/docs/start/cloud-environment)."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "Y4gnZI9OX6VJ"
      },
      "outputs": [],
      "source": [
        "# Set project id to be the current project ID.\n",
        "\n",
        "PROJECT_ID = \"[your-project-id]\"  # @param {type:\"string\"}\n",
        "LOCATION = \"us-central1\"  # @param {type:\"string\"}"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "-NrpFROTjoVL"
      },
      "outputs": [],
      "source": [
        "from google.cloud import aiplatform\n",
        "\n",
        "aiplatform.init(project=PROJECT_ID, location=LOCATION)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "moS794OKaaCt"
      },
      "outputs": [],
      "source": [
        "# Create GCS Bucket\n",
        "BUCKET_URI = \"gs://your-bucket-name-unique\"  # @param {type:\"string\"}\n",
        "! gcloud storage buckets create --location={LOCATION} --project={PROJECT_ID} {BUCKET_URI}"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "SASm5tKB2JGF"
      },
      "source": [
        "## Create Dedicated Endpoint and Deploy Model"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "taUmNhwP2S3W"
      },
      "source": [
        "### Create Dedicated Endpoint\n",
        "When creating endpoint, set `dedicated_endpoint_enabled` to True."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "IZUfl8Q82crP"
      },
      "outputs": [],
      "source": [
        "endpoint = aiplatform.Endpoint.create(\n",
        "    display_name=\"test-dedicated-endpoint\",\n",
        "    dedicated_endpoint_enabled=True,\n",
        ")"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "swERjzZ-a_Nd"
      },
      "source": [
        "### Prepare Test Models\n",
        "\n",
        "We prepared a tensorflow test model, feel free to use your own models."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "0fea4d3018ab"
      },
      "outputs": [],
      "source": [
        "! pip freeze | grep google-cloud-aiplatform"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "bA2zhbzJ1R0k"
      },
      "outputs": [],
      "source": [
        "# List all your models\n",
        "for my_model in aiplatform.Model.list():\n",
        "    print(my_model.display_name)\n",
        "    print(my_model.gca_resource.name)\n",
        "\n",
        "# If you want to use an existing model, use the resource id\n",
        "# model = aiplatform.Model('projects/12345/locations/us-central1/models/456789')"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "3PtFCQNHbloQ"
      },
      "outputs": [],
      "source": [
        "# @title Upload a new tensorflow model\n",
        "# @markdown You can skip this if use an existing model.\n",
        "\n",
        "# TF Model\n",
        "DISPLAY_NAME = \"tensorflow model\"  # @param {type:\"string\"}\n",
        "ARTIFACT_URI = BUCKET_URI + \"/tensorflow\"\n",
        "IMAGE_URI = \"us-docker.pkg.dev/vertex-ai/prediction/tf2-cpu.2-12:latest\"\n",
        "\n",
        "# Copy Test Models to the Bucket\n",
        "! gcloud storage cp --recursive \"gs://cloud-samples-data/vertex-ai/prediction/test-models-requests/tensorflow/*\" {ARTIFACT_URI}\n",
        "\n",
        "model = aiplatform.Model.upload(\n",
        "    display_name=DISPLAY_NAME,\n",
        "    artifact_uri=ARTIFACT_URI,\n",
        "    serving_container_image_uri=IMAGE_URI,\n",
        ")"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "USWCW-SNo-9M"
      },
      "source": [
        "#### Deploy Model\n",
        "\n",
        "Deploy the model to the endpoint. If it is your first model with dedicated endpoint, it will take ~30 min."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "S_eRJglhpVfL"
      },
      "outputs": [],
      "source": [
        "endpoint.deploy(model=model, traffic_percentage=100, machine_type=\"e2-standard-8\")"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "civyNQaPr4QD"
      },
      "source": [
        "## Make Predictions\n",
        "\n",
        "Dedicated endpoint cannot be accessed with the shared `aiplatform.googleapis.com` DNS. \\\n",
        "Instead, it has its own dedicated DNS\n",
        "```\n",
        "DEDICATED_DNS = f\"https://{endpoint.gca_resource.dedicated_endpoint_dns}/v1beta1/{endpoint.resource_name}\"\n",
        "# or\n",
        "DEDICATED_DNS = f\"https://{ENDPOINT_ID}.{LOCATION}-{PROJECT_NUMBER}.prediction.vertexai.goog\"\n",
        "```\n",
        "\n",
        "Python SDK has been integrated to support dedicated endpoint using a flag `use_dedicated_endpoint`.\n",
        "\n",
        "Alternatively, you can send HTTP/GRPC request directly to this DNS in any language you prefer."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "cellView": "form",
        "id": "I-O9U63juoWE"
      },
      "outputs": [],
      "source": [
        "# @title Predict\n",
        "# @markdown You can use client library.\n",
        "\n",
        "use_python_sdk = True # @param {type:\"boolean\"}\n",
        "# @markdown response = my_endpoint.predict( \\\n",
        "# @markdown &nbsp;&ensp; instances=[{\"feat_1\":val_1, \"feat_2\":val_2}]}, \\\n",
        "# @markdown &nbsp;&ensp; headers = {'Content-Type':'application/json'}, \\\n",
        "# @markdown &nbsp;&ensp; dedicated_endpoint=True, \\\n",
        "# @markdown )\n",
        "\n",
        "if use_python_sdk:\n",
        "    instances = [\n",
        "        {\"dense_input\": [14.0, 7.0, 2545.461893666405, 54.2, 48.5, 0.0, 61.3, 0.0, 0.0, 0.0]},\n",
        "        {\"dense_input\": [28.0, 14.0, 1234, 27.1, 90, 0.0, 61.3, 0.0, 0.0, 0.0]}\n",
        "    ]\n",
        "    response = endpoint.predict(instances=instances, use_dedicated_endpoint=True)\n",
        "    print(response)\n",
        "\n",
        "# @markdown You can also make HTTP/GRPC request directly.\n",
        "\n",
        "use_http = True # @param {type:\"boolean\"}\n",
        "# @markdown POST request to the following URL:\\\n",
        "# @markdown `https://ENDPOINT_ID.us-central1-PROJECT_NUMBER.prediction.vertexai.goog/v1/projects/PROJECT_NUMBER/locations/LOCATION/endpoints/ENDPOINT_ID:predict`\n",
        "\n",
        "if use_http:\n",
        "    request_path = f\"https://{endpoint.gca_resource.dedicated_endpoint_dns}/v1/{endpoint.resource_name}:predict\"\n",
        "    ! curl {request_path} -X POST  -H \"Content-Type: application/json\" -H \"Authorization: Bearer `gcloud auth print-access-token`\" -d \\\n",
        "    '{{ \\\n",
        "      \"instances\": [ \\\n",
        "        {{\"dense_input\": [14.0, 7.0, 2545.461893666405, 54.2, 48.5, 0.0, 61.3, 0.0, 0.0, 0.0]}}, \\\n",
        "        {{\"dense_input\": [28.0, 14.0, 1234, 27.1, 90, 0.0, 61.3, 0.0, 0.0, 0.0]}} \\\n",
        "      ] \\\n",
        "    }}'"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "cellView": "form",
        "id": "RRoiziQ9FAw6"
      },
      "outputs": [],
      "source": [
        "# @title Raw Predict\n",
        "# @markdown You can use client library.\n",
        "\n",
        "use_python_sdk = True # @param {type:\"boolean\"}\n",
        "# @markdown response = my_endpoint.raw_predict( \\\n",
        "# @markdown &nbsp;&ensp;&nbsp;&ensp;body = b'{\"instances\":[{\"feat_1\":val_1, \"feat_2\":val_2}]}', \\\n",
        "# @markdown &nbsp;&ensp;&nbsp;&ensp;headers = {'Content-Type':'application/json'}, \\\n",
        "# @markdown &nbsp;&ensp;&nbsp;&ensp;dedicated_endpoint=True, \\\n",
        "# @markdown ) \\\n",
        "# @markdown status_code = response.status_code \\\n",
        "# @markdown results = json.dumps(response.text)\n",
        "\n",
        "if use_python_sdk:\n",
        "    body = b'{ \\\n",
        "      \"instances\": [ \\\n",
        "        {\"dense_input\": [14.0, 7.0, 2545.461893666405, 54.2, 48.5, 0.0, 61.3, 0.0, 0.0, 0.0]}, \\\n",
        "        {\"dense_input\": [28.0, 14.0, 1234, 27.1, 90, 0.0, 61.3, 0.0, 0.0, 0.0]} \\\n",
        "      ]\\\n",
        "    }'\n",
        "    import os\n",
        "    token = !gcloud auth print-access-token\n",
        "    token = token[0]\n",
        "    headers = {\"content-type\": \"application/json\", \"Authorization\": f\"Bearer {token}\"}\n",
        "    response = endpoint.raw_predict(body=body, headers=headers, use_dedicated_endpoint=True)\n",
        "    # print(response.status_code)\n",
        "    print(response.text)\n",
        "\n",
        "# @markdown You can also make HTTP/GRPC request directly.\n",
        "\n",
        "use_http = True # @param {type:\"boolean\"}\n",
        "# @markdown POST request to the following URL:\\\n",
        "# @markdown `https://ENDPOINT_ID.us-central1-PROJECT_NUMBER.prediction.vertexai.goog/v1/projects/PROJECT_NUMBER/locations/LOCATION/endpoints/ENDPOINT_ID:rawPredict`\n",
        "\n",
        "if use_http:\n",
        "    request_path = f\"https://{endpoint.gca_resource.dedicated_endpoint_dns}/v1/{endpoint.resource_name}:rawPredict\"\n",
        "    ! curl {request_path} -X POST  -H \"Content-Type: application/json\" -H \"Authorization: Bearer `gcloud auth print-access-token`\" -d \\\n",
        "    '{{ \\\n",
        "      \"instances\": [ \\\n",
        "        {{\"dense_input\": [14.0, 7.0, 2545.461893666405, 54.2, 48.5, 0.0, 61.3, 0.0, 0.0, 0.0]}}, \\\n",
        "        {{\"dense_input\": [28.0, 14.0, 1234, 27.1, 90, 0.0, 61.3, 0.0, 0.0, 0.0]}} \\\n",
        "      ] \\\n",
        "    }}'"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "cellView": "form",
        "id": "IbZDIJlIRl1b"
      },
      "outputs": [],
      "source": [
        "# @title Stream Raw Predict\n",
        "# @markdown You can use client library.\n",
        "\n",
        "use_python_sdk = True # @param {type:\"boolean\"}\n",
        "# @markdown for stream_response in my_endpoint.stream_raw_predict( \\\n",
        "# @markdown &nbsp;&ensp;&nbsp;&ensp;body = b'{\"instances\":[{\"feat_1\":val_1, \"feat_2\":val_2}]}', \\\n",
        "# @markdown &nbsp;&ensp;&nbsp;&ensp;headers = {'Content-Type':'application/json'}, \\\n",
        "# @markdown &nbsp;&ensp;&nbsp;&ensp;use_dedicated_endpoint=True, \\\n",
        "# @markdown ): \\\n",
        "# @markdown &nbsp;&ensp;&nbsp;&ensp;status_code = response.status_code \\\n",
        "# @markdown &nbsp;&ensp;&nbsp;&ensp;stream_result = json.dumps(response.text)\n",
        "\n",
        "if use_python_sdk:\n",
        "    body = b'{ \\\n",
        "      \"instances\": [ \\\n",
        "        {\"dense_input\": [14.0, 7.0, 2545.461893666405, 54.2, 48.5, 0.0, 61.3, 0.0, 0.0, 0.0]}, \\\n",
        "        {\"dense_input\": [28.0, 14.0, 1234, 27.1, 90, 0.0, 61.3, 0.0, 0.0, 0.0]} \\\n",
        "      ]\\\n",
        "    }'\n",
        "    import os\n",
        "    token = !gcloud auth print-access-token\n",
        "    token = token[0]\n",
        "    headers = {\"content-type\": \"application/json\", \"Authorization\": f\"Bearer {token}\"}\n",
        "    for stream_response in endpoint.raw_predict(body=body, headers=headers, use_dedicated_endpoint=True):\n",
        "        # print(response.status_code)\n",
        "        print(response.text)\n",
        "\n",
        "# @markdown You can also make HTTP/GRPC request directly.\n",
        "\n",
        "use_http = True # @param {type:\"boolean\"}\n",
        "# @markdown POST request to the following URL:\\\n",
        "# @markdown `https://ENDPOINT_ID.us-central1-PROJECT_NUMBER.prediction.vertexai.goog/v1/projects/PROJECT_NUMBER/locations/LOCATION/endpoints/ENDPOINT_ID:streamRawPredict`\n",
        "\n",
        "if use_http:\n",
        "    request_path = f\"https://{endpoint.gca_resource.dedicated_endpoint_dns}/v1/{endpoint.resource_name}:streamRawPredict\"\n",
        "    ! curl {request_path} -X POST  -H \"Content-Type: application/json\" -H \"Authorization: Bearer `gcloud auth print-access-token`\" -d \\\n",
        "    '{{ \\\n",
        "      \"instances\": [ \\\n",
        "        {{\"dense_input\": [14.0, 7.0, 2545.461893666405, 54.2, 48.5, 0.0, 61.3, 0.0, 0.0, 0.0]}}, \\\n",
        "        {{\"dense_input\": [28.0, 14.0, 1234, 27.1, 90, 0.0, 61.3, 0.0, 0.0, 0.0]}} \\\n",
        "      ] \\\n",
        "    }}'"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "35gPjIcSfIVY"
      },
      "source": [
        "### Chat Completion (Model Garden Only)\n",
        "**You can use OpenAI client library to do chat completion.**\n",
        "\n",
        "```\n",
        "client = openai.OpenAI(base_url=DEDICATED_DNS, api_key=creds.token)\n",
        "model_response = client.chat.completions.create(\n",
        "    model=\"your model\",\n",
        "    messages=[{\"role\": \"user\", \"content\": user_message}],\n",
        "    temperature=temperature,\n",
        "    max_tokens=max_tokens,\n",
        "    stream=stream,\n",
        ")\n",
        "```\n",
        "\n",
        "**You can also make HTTP/GRPC request directly.**\n",
        "\n",
        "POST request to the following URL:\n",
        "```\n",
        "curl -X POST -H \"Content-Type: application/json\" \\\n",
        "  -H \"Authorization: Bearer `gcloud auth print-access-token`\" \\\n",
        "  https://DEDICATED_DNS/v1beta1/projects/PROJECT_NUMBER/locations/LOCATION/endpoints/ENDPOINT_ID/chat/completions -d PAYLOAD\n",
        "```\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Ka-pVhA1Xf-g"
      },
      "source": [
        "## Supported Features"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "_TCktMxbA4mb"
      },
      "source": [
        "### Traffic split\n",
        "\n",
        "Deploy another model, and update the traffic split to be 50:50, this should take less than 5 min using the test model. After the deployment is done, you can rerun the prediction again for multiple times, you should be able to see the deployed_model_id are different."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "bVLgpRiRBEL7"
      },
      "outputs": [],
      "source": [
        "endpoint.deploy(model=model, traffic_percentage=50, machine_type=\"e2-standard-8\")"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "2dx975IkCv7v"
      },
      "outputs": [],
      "source": [
        "instances = [\n",
        "    {\n",
        "        \"dense_input\": [\n",
        "            14.0,\n",
        "            7.0,\n",
        "            2545.461893666405,\n",
        "            54.2,\n",
        "            48.5,\n",
        "            0.0,\n",
        "            61.3,\n",
        "            0.0,\n",
        "            0.0,\n",
        "            0.0,\n",
        "        ]\n",
        "    },\n",
        "    {\"dense_input\": [28.0, 14.0, 1234, 27.1, 90, 0.0, 61.3, 0.0, 0.0, 0.0]},\n",
        "]\n",
        "counter = {}\n",
        "for i in range(1000):\n",
        "    response = endpoint.predict(instances=instances, use_dedicated_endpoint=True)\n",
        "    if response.deployed_model_id in counter.keys():\n",
        "        counter[response.deployed_model_id] += 1\n",
        "    else:\n",
        "        counter[response.deployed_model_id] = 1\n",
        "print(counter)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "2Hld7iDmEyiF"
      },
      "source": [
        "You can update the traffic split with the following command and run the code above again."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "HXPI-2q9Eh6X"
      },
      "outputs": [],
      "source": [
        "deployed_model_id_0 = list(counter)[0]\n",
        "deployed_model_id_1 = list(counter)[1]\n",
        "\n",
        "endpoint.update(traffic_split={deployed_model_id_0: 20, deployed_model_id_1: 80})"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "oXG3G-7D4hUJ"
      },
      "source": [
        "### Custom Timeout\n",
        "\n",
        "Default 600s(10 min) timeout will be applied if the inference timeout is 0. Max timeout allowed is 1h. \\\n",
        "\n",
        "Use `EndpointService.UpdateEndpointLongRunning` to change the setting \\\n",
        "`EndpointService.UpdateEndpoint` doesn't support request/response logging change.\n",
        "\n",
        "```\n",
        "timeout_endpoint = aiplatform.Endpoint.create(\n",
        "    display_name=\"test-dedicated-endpoint-with-timeout\",\n",
        "    dedicated_endpoint_enabled=True,\n",
        "    inference_timeout=1800, # Unit: Second.\n",
        ")\n",
        "```\n",
        "Alternatively, you can create the endpoint using HTTP.\n",
        "\n",
        "Create endpoint with timeout sample:\n",
        "```\n",
        "curl -X POST -H \"Content-Type: application/json\" \\\n",
        "  -H \"Authorization: Bearer `gcloud auth print-access-token`\" \\\n",
        "  https://${DEDICATED_DNS}/v1beta1/projects/${PROJECT_NUMBER}/locations/us-central1/endpoints -d \\\n",
        "  '{ \\\n",
        "      displayName: \"test-dedicated-endpoint-with-timeout\", \\\n",
        "      dedicatedEndpointEnabled: true, \\\n",
        "      client_connection_config: \\\n",
        "          { \\\n",
        "            \"inferenceTimeout\": \\\n",
        "              {\"seconds\": 20, \"nanos\": 0} \\\n",
        "          } \\\n",
        "    }'\n",
        "```\n",
        "Update endpoint long running sample:\n",
        "```\n",
        "curl -X POST \\\n",
        "  -H \"Authorization: Bearer $(gcloud auth print-access-token)\" \\\n",
        "  -H \"Content-Type: application/json\" \\\n",
        "  https://${DEDICATED_DNS}/v1beta1/projects/${PROJECT_ID}/locations/${REGION}/endpoints/${ENDPOINT_ID}:update \\\n",
        "  -d '{ \\\n",
        "      \"endpoint\": { \\\n",
        "        \"clientConnectionConfig\": { \\\n",
        "            \"inferenceTimeout\": {\"seconds\": 5, \"nanos\": 0} \\\n",
        "          }, \\\n",
        "      } \\\n",
        "    }'\n",
        "```"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "yDNTq4Zh39MS"
      },
      "source": [
        "### Request&Response Logging\n",
        "\n",
        "Request/Response logging will skip payload that exceed 10M which is the big query limit.\n",
        "\n",
        "Use `EndpointService.UpdateEndpointLongRunning` to change the setting \\\n",
        "`EndpointService.UpdateEndpoint` doesn't support request/response logging change.\n",
        "\n",
        "```\n",
        "logging_endpoint = aiplatform.Endpoint.create(\n",
        "    display_name=\"test-dedicated-endpoint-with-logging\",\n",
        "    dedicated_endpoint_enabled=True,\n",
        "    enable_request_response_logging=True,\n",
        "    request_response_logging_sampling_rate=1.0, # default 0.0\n",
        "    request_response_logging_bq_destination_table=\"bq://test_logging\",\n",
        "    # if not set, a new table will be created with name `bq://{project_id}.logging_{endpoint_display_name}_{endpoint_id}.request_response_logging`\n",
        ")\n",
        "```\n",
        "\n",
        "Alternatively, use HTTP directly.\n",
        "\n",
        "Create endpoint sample\n",
        "```\n",
        "curl -X POST \\\n",
        "  -H \"Authorization: Bearer $(gcloud auth print-access-token)\" -H \"Content-Type: application/json\" \\\n",
        "  https://${DEDICATED_DNS}/v1beta1/projects/${PROJECT_ID}/locations/us-central1/endpoints \\\n",
        "  -d '{ \\\n",
        "    \"display_name\": \"test-dedicated-endpoint-with-logging\", \\\n",
        "    \"dedicatedEndpointEnabled\": true, \\\n",
        "    \"predict_request_response_logging_config\": { \\\n",
        "      \"enabled\": true, \\\n",
        "      \"sampling_rate\": 0.5, \\\n",
        "      \"bigquery_destination\": { \\\n",
        "        \"output_uri\": \"bq://my-project\" \\\n",
        "        } \\\n",
        "      } \\\n",
        "    }'\n",
        "```\n",
        "\n",
        "Update endpoint long running request sample\n",
        "```\n",
        "curl -X POST \\\n",
        "  -H \"Authorization: Bearer $(gcloud auth print-access-token)\" \\\n",
        "  -H \"Content-Type: application/json\" \\\n",
        "  https://${DEDICATED_DNS}/v1beta1/projects/${PROJECT_ID}/locations/${REGION}/endpoints/${ENDPOINT_ID}:update \\\n",
        "  -d '{ \\\n",
        "      \"endpoint\": { \\\n",
        "        \"predict_request_response_logging_config\": { \\\n",
        "          \"enabled\": true, \\\n",
        "          \"sampling_rate\": 0.5, \\\n",
        "          \"bigquery_destination\": { \\\n",
        "            \"output_uri\": \"bq://my-project\" \\\n",
        "          } \\\n",
        "        }, \\\n",
        "      } \\\n",
        "    }'\n",
        "```\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "XW_BtPnEFPp4"
      },
      "source": [
        "## Cleanup"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "E4Ik3eKqdI_2"
      },
      "outputs": [],
      "source": [
        "endpoint.undeploy_all()\n",
        "endpoint.delete()\n",
        "model.delete()"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "BTz-2N1XunXB"
      },
      "source": [
        "Delete the bucket if needed."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "NPQT5Wv9lC3O"
      },
      "outputs": [],
      "source": [
        "! gcloud storage rm --recursive {BUCKET_URI}"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "2iR_Q5K_ksWi"
      },
      "source": [
        "Optionally, you can use the following command to clean up all endpoint and models if needed."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "vkyvlwTgky0J"
      },
      "outputs": [],
      "source": [
        "# for e in aiplatform.Endpoint.list():\n",
        "#   e.undeploy_all()\n",
        "#   e.delete()"
      ]
    }
  ],
  "metadata": {
    "colab": {
      "name": "get_started_with_dedicated_endpoint.ipynb",
      "toc_visible": true
    },
    "kernelspec": {
      "display_name": "Python 3",
      "name": "python3"
    }
  },
  "nbformat": 4,
  "nbformat_minor": 0
}