diff --git a/notebooks/official/CODEOWNERS b/notebooks/official/CODEOWNERS index a743d2902..9d6775509 100644 --- a/notebooks/official/CODEOWNERS +++ b/notebooks/official/CODEOWNERS @@ -60,3 +60,4 @@ /generative_ai/mistralai_intro.ipynb @sujituk /generative_ai/ai21labs_intro.ipynb @sujituk /forecasting/starry_net_pipeline.ipynb @tsteve +/generative_ai/nvidianims_vertexai.ipynb @sujituk diff --git a/notebooks/official/generative_ai/nvidianims_vertexai.ipynb b/notebooks/official/generative_ai/nvidianims_vertexai.ipynb new file mode 100644 index 000000000..e310cf3d1 --- /dev/null +++ b/notebooks/official/generative_ai/nvidianims_vertexai.ipynb @@ -0,0 +1,769 @@ +{ + "cells": [ + { + "cell_type": "code", + "execution_count": null, + "id": "FHI7XiOc8L9W", + "metadata": { + "id": "FHI7XiOc8L9W" + }, + "outputs": [], + "source": [ + "# Copyright 2024 Google LLC\n", + "#\n", + "# Licensed under the Apache License, Version 2.0 (the \"License\");\n", + "# you may not use this file except in compliance with the License.\n", + "# You may obtain a copy of the License at\n", + "#\n", + "# https://www.apache.org/licenses/LICENSE-2.0\n", + "#\n", + "# Unless required by applicable law or agreed to in writing, software\n", + "# distributed under the License is distributed on an \"AS IS\" BASIS,\n", + "# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n", + "# See the License for the specific language governing permissions and\n", + "# limitations under the License." + ] + }, + { + "cell_type": "markdown", + "id": "bXIc5UUP0IdQ", + "metadata": { + "id": "bXIc5UUP0IdQ" + }, + "source": [ + "# **NVIDIA NIMs on Google Cloud Vertex AI**\n", + "\n", + "\n", + " \n", + " \n", + "
\n", + " \n", + " \"Google
Open in Colab Enterprise\n", + " \n", + " 		\n", + " \n", + " \"GitHub
View on GitHub\n", + " \n", + "
" + ] + }, + { + "cell_type": "markdown", + "id": "eSsj1dJ48bgo", + "metadata": { + "id": "eSsj1dJ48bgo" + }, + "source": [ + "[Vertex AI](https://cloud.google.com/vertex-ai/docs/start/introduction-unified-platform) is Google Cloud's unified machine learning platform. It streamlines the process of building, training, and deploying AI models, making it easier to bring your AI projects to life.\n", + "\n", + "[NVIDIA Inference Microservices (NIMs)](https://www.nvidia.com/en-us/ai/) are pre-trained and optimized AI models packaged as microservices. They're designed to simplify the deployment of high-performance, production-ready AI into applications.\n", + "\n", + "In this colab notebook, we will demonstrate how to run a *meta/llama-3.1-8b-instruct* NIM on Vertex AI supported by NVIDIA GPUs. We will also showcase a sample inference in both batch and streaming modes. You can run this using Colab Enterprise in Vertex AI. In summary, NVIDIA NIMs are packaged as container images, which need to be pulled into your Google Cloud environment and deployed to a Vertex AI endpoint. These endpoints are Open API compatible REST endpoints that can be used with your application for different use cases.\n" + ] + }, + { + "cell_type": "markdown", + "id": "qeziXkLk5IbE", + "metadata": { + "id": "qeziXkLk5IbE" + }, + "source": [ + "## Prerequisites\n", + "### Hardware\n", + "\n", + "You will need access to the [NVIDIA L4 GPU](https://cloud.google.com/compute/docs/gpus#l4-gpus) accelerator, accessible through Google Cloud G2 VM family, `g2-standard-24` VM type. This is a minimum requirement to run the meta/llama-3.1-8b-instruct NIM.\n", + "\n", + "### Software\n", + "\n", + "1. [Google Cloud Project](https://cloud.google.com/resource-manager/docs/creating-managing-projects#creating_a_project) with a billing ID\n", + "2. [Vertex AI](https://cloud.google.com/vertex-ai/docs/start/introduction-unified-platform)\n", + " - [Vertex AI Model Resource](https://cloud.google.com/vertex-ai/docs/model-registry/introduction)\n", + " - [Vertex AI Endpoint](https://cloud.google.com/vertex-ai/docs/general/deployment)\n", + "3. [Colab Enterprise](https://cloud.google.com/colab/docs/create-console-quickstart) using a default Runtime\n", + "4. [Artifact Registry](https://cloud.google.com/artifact-registry)\n", + "5. NVIDIA NGC API Key\n", + "\n", + " **Note:** Please sign up for the [NVIDIA Developer Program](https://developer.nvidia.com/developer-program) which provides developers with tools and resources to build more efficiently and quickly using NVIDIA technology. You will get a NGC API Key that is required to access NIMs.\n", + "\n", + "### Required Security roles\n", + "\n", + "\n", + "To ensure that your user account has the necessary permissions perform all operations required for this colab, ask your administrator to grant your user account below roles:\n", + " - Colab Enterprise Admin (*roles/aiplatform.colabEnterpriseAdmin*)\n", + " - Vertex AI Platform User (*roles/aiplatform.user*)\n", + "\n", + "The Vertex AI Workbench instance runs under the default Compute Engine service account [*\\-compute@developer.gserviceaccount.com*]. Ask your administrator to grant below role(s) to the default Compute engine service account:\n", + " - Artifact Registry Writer (*roles/artifactregistry.writer*)\n", + " - Compute Network Admin (*roles/compute.networkAdmin*) [Optional, if there is no default network]\n", + "\n", + "\n", + "## Outline\n", + "\n", + "\n", + "1. [Getting started](#step1): Download the NIM container image to Artifact Registry. Vertex AI Workbench instance is pre-installed with docker to perform operations such as pulling, tagging and pushing images to repositories. The NIM container image will be pushed in the Artifact registry.\n", + "\n", + " *If you already have the NIM image in Google's Artifact Registry, you can move to [Step 3](#step3).*\n", + "\n", + "2. [Prerequisites](#step2): Enable API's for Google Cloud Products and authenticate for subsequent steps.\n", + "\n", + "4. [Configure](#step3): Parameters such as GPU Accelerator types, machine types are configured to host the NIM in Vertex AI\n", + "\n", + "5. [Deploy](#step4): The model needs to be uploaded to Vertex AI Model and deployed to a Vertex AI endpoint.\n", + "\n", + "6. [Test inference](#step7): Use sample prompts to test the model inferencing in [batch](#step7a) and [streaming](#step7b) mode.\n", + "\n", + "6. [Teardown](#step8): Teardown all the resources such as Vertex AI Endpoint, Model, Artifact registry repo and Workbench instance. " + ] + }, + { + "cell_type": "markdown", + "id": "H-dYYYhtiyZx", + "metadata": { + "id": "H-dYYYhtiyZx" + }, + "source": [ + "## Step 1: Getting Started\n", + "\n", + "\n", + "### a. Enable APIs\n", + "Enable the APIs listed below from the Google Cloud Console.\n", + "\n", + " - [Artifact Registry](https://console.cloud.google.com/flows/enableapi?apiid=artifactregistry.googleapis.com&redirect=https://console.cloud.google.com&_ga=2.153348347.214183506.1726972544-2083916923.1726802364)\n", + " - [Compute Engine](https://console.cloud.google.com/flows/enableapi?apiid=compute.googleapis.com&redirect=https://console.cloud.google.com&_ga=2.153677051.214183506.1726972544-2083916923.1726802364)\n", + " - [Dataform](https://console.cloud.google.com/flows/enableapi?apiid=dataform.googleapis.com&redirect=https://console.cloud.google.com&_ga=2.203088210.214183506.1726972544-2083916923.1726802364)\n", + " - [Notebook](https://console.cloud.google.com/flows/enableapi?apiid=notebooks.googleapis.com&redirect=https://console.cloud.google.com&_ga=2.153677051.214183506.1726972544-2083916923.1726802364)\n", + " - [Vertex AI](https://console.cloud.google.com/flows/enableapi?apiid=aiplatform.googleapis.com&redirect=https://console.cloud.google.com&_ga=2.153348347.214183506.1726972544-2083916923.1726802364)\n", + "\n", + "### b. Colab Notebook\n", + "\n", + "Follow [this link](https://cloud.google.com/colab/docs/create-runtime#create) to create a runtime in Colab Enterprise.\n", + "\n", + "### c. Setup environment\n", + "Install Python packages as shown below and provide values for your environment.\n", + "\n", + "

[go to top](#outline)

\n" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "sNU6x_VFBYVi", + "metadata": { + "id": "sNU6x_VFBYVi" + }, + "outputs": [], + "source": [ + "! pip install --upgrade --user --quiet \\\n", + " google-cloud-aiplatform \\\n", + " google-cloud-artifact-registry \\\n", + " openai==1.35.15" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "f0hgKwc6CLrb", + "metadata": { + "id": "f0hgKwc6CLrb" + }, + "outputs": [], + "source": [ + "import sys\n", + "\n", + "if \"google.colab\" in sys.modules:\n", + "\n", + " import IPython\n", + "\n", + " app = IPython.Application.instance()\n", + " app.kernel.do_shutdown(True)" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "zPe2WjwQzut-", + "metadata": { + "id": "zPe2WjwQzut-" + }, + "outputs": [], + "source": [ + "PROJECT_ID = \"[your-project-id]\" # @param {type:\"string\"}\n", + "ZONE = \"us-central1-a\" # @param [\"us-central1-a\"] {\"allow-input\":true}\n", + "LOCATION = \"-\".join(ZONE.split(\"-\")[:-1])\n", + "\n", + "REPOSITORY_NAME = \"nim\"\n", + "\n", + "WORKBENCH_NAME = \"wb-nims\" # @param {\"type\":\"string\"}\n", + "MACHINE_TYPE = \"e2-standard-4\" # @param {\"type\":\"string\"}" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "ng0UmyfrD8VV", + "metadata": { + "id": "ng0UmyfrD8VV" + }, + "outputs": [], + "source": [ + "! gcloud artifacts repositories create {REPOSITORY_NAME} \\\n", + " --repository-format=docker \\\n", + " --location={LOCATION} \\\n", + " --project={PROJECT_ID}" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "2aUBUukfN2dH", + "metadata": { + "id": "2aUBUukfN2dH" + }, + "outputs": [], + "source": [ + "import google.cloud.aiplatform as aiplatform\n", + "\n", + "aiplatform.init(project=PROJECT_ID, location=LOCATION)" + ] + }, + { + "cell_type": "markdown", + "id": "CncZoi96CoWz", + "metadata": { + "id": "CncZoi96CoWz" + }, + "source": [ + "## Step 2: Create Vertex AI Workbench instance\n", + "\n", + "\n", + "A temporary Vertex AI Workbench instance is created to pull the docker image from NVIDIA GPU Catalog (NGC) using the NGC API Key. If there is no default network setup, you can optionally create and specify below.\n", + "\n", + "**Note:**\n", + "Click on the link below that opens up the new Vertex AI Workbench instance.\n", + "

[go to top](#outline)

" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "iTd24cMfkqtV", + "metadata": { + "id": "iTd24cMfkqtV" + }, + "outputs": [], + "source": [ + "# Optional to create network and specify if there is no default network created\n", + "NETWORK_NAME = f\"vpc-{WORKBENCH_NAME}\"\n", + "\n", + "!gcloud compute networks create {NETWORK_NAME} \\\n", + " --project={PROJECT_ID} \\\n", + " --subnet-mode=auto" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "igcBIoHW9RnQ", + "metadata": { + "id": "igcBIoHW9RnQ" + }, + "outputs": [], + "source": [ + "! gcloud workbench instances create {WORKBENCH_NAME} \\\n", + " --project={PROJECT_ID} \\\n", + " --location={ZONE} \\\n", + " --machine-type={MACHINE_TYPE}\n", + " #--network={NETWORK_NAME}\n", + "\n", + "from IPython.display import HTML\n", + "\n", + "url = !gcloud workbench instances describe {WORKBENCH_NAME} \\\n", + " --project={PROJECT_ID} \\\n", + " --location={ZONE} \\\n", + " --format=\"value(gceSetup.metadata.proxy-url)\"\n", + "\n", + "url = url[0]\n", + "display(HTML(f'{url}'))" + ] + }, + { + "cell_type": "markdown", + "id": "bciR3s8p56QF", + "metadata": { + "id": "bciR3s8p56QF" + }, + "source": [ + "## Step 3: Push image to Artifact registry\n", + "\n", + "\n", + "The below code has two sections.\n", + " - [Configure NIM models](#step3a): Execute this in **this colab** and **the Vertex AI workbench**.\n", + " - [Pull and push Docker image](#step3b): Execute this in the **Vertex AI workbench only**. You will need the NGC API Key. It will fail if tried to execute in the colab. In the Workbench instance, follow these steps:\n", + " - Click Notebooks -> Python3 (ipykernel)\n", + " - Paste code from both cells\n", + " - Provide values for *PROJECT_ID*, *LOCATION*, if not already\n", + " - Execute all at once\n", + "\n", + "**Note:** This step can take from 20-30 mins depending on the size of the image.\n", + "

[go to top](#outline)

" + ] + }, + { + "cell_type": "markdown", + "id": "phbVb2UWiblT", + "metadata": { + "id": "phbVb2UWiblT" + }, + "source": [ + "### a. Configure NIM models\n", + "\n", + "\n", + "Note:: Execute both in Colab and the Vertex AI Workbench notebook." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "b1u5-MWu3wTc", + "metadata": { + "id": "b1u5-MWu3wTc" + }, + "outputs": [], + "source": [ + "LOCATION = \"[GCP region from Step 1c]\"\n", + "PROJECT_ID = \"[GCP project_id from Step 1c]\"\n", + "\n", + "NVCR_REGISTRY = \"nvcr.io\"\n", + "REPOSITORY_NAME = \"nim\"\n", + "NIM_MODEL = \"meta/llama-3.1-8b-instruct:latest\" # @param [\"meta/llama-3.1-8b-instruct:latest\",\"meta/llama-3.1-70b-instruct:1.1\",\"meta/llama-3.1-405b-instruct:1.2.0\",\"meta/llama3-8b-instruct:1.0.3\",\"meta/llama3-70b-instruct:1.0.3\"] {\"allow-input\":true}\n", + "NGC_API_KEY = \"[nvidia-ngc-api-key]\" # @param {\"type\":\"string\"}\n", + "\n", + "NIM_IMAGE = NVCR_REGISTRY + \"/\" + REPOSITORY_NAME + \"/\" + NIM_MODEL\n", + "NIM_IMAGE_GAR = (\n", + " LOCATION + \"-docker.pkg.dev/\" + PROJECT_ID + \"/\" + REPOSITORY_NAME + \"/\" + NIM_MODEL\n", + ")" + ] + }, + { + "cell_type": "markdown", + "id": "PJK5VHrZinb_", + "metadata": { + "id": "PJK5VHrZinb_" + }, + "source": [ + "### b. Pull and push Docker image\n", + "\n", + "\n", + "**Note:**: Execute only in the Vertex AI Workbench notebook." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "j7WQ5XzviAyX", + "metadata": { + "id": "j7WQ5XzviAyX" + }, + "outputs": [], + "source": [ + "! gcloud auth configure-docker {LOCATION}-docker.pkg.dev --quiet\n", + "\n", + "! docker login -u '$oauthtoken' --password-stdin nvcr.io <<< \"$NGC_API_KEY\"\n", + "! docker pull $NIM_IMAGE\n", + "! docker tag $NIM_IMAGE $NIM_IMAGE_GAR\n", + "! docker push $NIM_IMAGE_GAR" + ] + }, + { + "cell_type": "markdown", + "id": "Ow_OnrCaXN_f", + "metadata": { + "id": "Ow_OnrCaXN_f" + }, + "source": [ + "## Step 4. Upload NIM to Vertex AI\n", + "\n", + "Uploading a NIM to Vertex AI provides a streamlined and efficient way to deploy and manage your generative AI models.\n", + "

[go to top](#outline)

" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "XV_BtH9qMrxf", + "metadata": { + "id": "XV_BtH9qMrxf" + }, + "outputs": [], + "source": [ + "MACHINE_TYPE = \"g2-standard-24\" # @param {type:\"string\"}\n", + "GPU_ACCELERATOR_TYPE = \"NVIDIA_L4\" # @param {type:\"string\"}\n", + "GPU_ACCELERATOR_COUNT = 2 # @param {\"type\":\"number\"}\n", + "\n", + "SELECTED_PROFILE = \"vllm-fp16-tp2\" # @param {type:\"string\"}\n", + "API_ENDPOINT = \"{}-aiplatform.googleapis.com\".format(LOCATION)\n", + "\n", + "endpoint_name = NIM_MODEL.replace(\":\", \"_\")\n", + "model_wo_tag = NIM_MODEL.split(\":\")[0]" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "DiE5bGiMIkkE", + "metadata": { + "id": "DiE5bGiMIkkE" + }, + "outputs": [], + "source": [ + "from google.api_core.future.polling import DEFAULT_POLLING\n", + "from google.cloud.aiplatform import Endpoint, Model\n", + "\n", + "DEFAULT_POLLING._timeout = 360000\n", + "model = None\n", + "models = Model.list(filter=f'displayName=\"{NIM_MODEL}\"')\n", + "\n", + "if models:\n", + " model = models[0]\n", + "else:\n", + " try:\n", + " model = aiplatform.Model.upload(\n", + " display_name=f\"{NIM_MODEL}\",\n", + " serving_container_image_uri=f\"{NIM_IMAGE_GAR}\",\n", + " serving_container_predict_route=\"/v1/chat/completions\",\n", + " serving_container_health_route=\"/v1/health/ready\",\n", + " serving_container_environment_variables={\n", + " \"NGC_API_KEY\": f\"{NGC_API_KEY}\",\n", + " \"PORT\": \"8000\",\n", + " \"shm-size\": \"16GB\",\n", + " },\n", + " serving_container_shared_memory_size_mb=16000,\n", + " serving_container_ports=[8000],\n", + " sync=True,\n", + " )\n", + " model.wait()\n", + "\n", + " except Exception as e:\n", + " print(f\"An error occurred: {str(e)}\")\n", + "\n", + "if model:\n", + " print(\"Model:\")\n", + " print(f\"\\tDisplay name: {model.display_name}\")\n", + " print(f\"\\tResource name: {model.resource_name}\")\n", + " MODEL_ID = model.resource_name" + ] + }, + { + "cell_type": "markdown", + "id": "KkJ9kWaGZVnN", + "metadata": { + "id": "KkJ9kWaGZVnN" + }, + "source": [ + "## Step 5. Create Vertex Endpoint\n", + "\n", + "The Vertex AI Endpoint components expose the functionalities of the Vertex AI Model through an Endpoint resource.\n", + "

[go to top](#outline)

" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "B-ekGzFGQomr", + "metadata": { + "id": "B-ekGzFGQomr" + }, + "outputs": [], + "source": [ + "endpoints = Endpoint.list(filter=f'displayName=\"{endpoint_name}\"')\n", + "print(endpoints)\n", + "if endpoints:\n", + " endpoint = endpoints[0]\n", + "else:\n", + " print(f\"Endpoint {endpoint_name} doesn't exist, creating...\")\n", + " endpoint = aiplatform.Endpoint.create(display_name=endpoint_name)\n", + "\n", + "if endpoint:\n", + " print(\"Endpoint:\")\n", + " print(f\"\\tDisplay name: {endpoint.display_name}\")\n", + " print(f\"\\tResource name: {endpoint.resource_name}\")\n", + "\n", + " ENDPOINT_ID = endpoint.resource_name" + ] + }, + { + "cell_type": "markdown", + "id": "ALnX28YdZacH", + "metadata": { + "id": "ALnX28YdZacH" + }, + "source": [ + "## Step 6. Deploy NIM\n", + "\n", + "\n", + "Models must be deployed to an endpoint before they can be used for online predictions.\n", + "\n", + "**Note:** This step can take 20-30 minutes.\n", + "

[go to top](#outline)

\n" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "oPQ-3kMoRFPr", + "metadata": { + "id": "oPQ-3kMoRFPr" + }, + "outputs": [], + "source": [ + "try:\n", + " model.deploy(\n", + " endpoint=endpoint,\n", + " deployed_model_display_name=f\"{NIM_MODEL}\",\n", + " traffic_percentage=100,\n", + " machine_type=f\"{MACHINE_TYPE}\",\n", + " min_replica_count=1,\n", + " max_replica_count=1,\n", + " accelerator_type=f\"{GPU_ACCELERATOR_TYPE}\",\n", + " accelerator_count=GPU_ACCELERATOR_COUNT,\n", + " enable_access_logging=True,\n", + " sync=True,\n", + " )\n", + " print(f\"Model {model.display_name} deployed at endpoint {endpoint.display_name}.\")\n", + "except Exception as e:\n", + " print(f\"An error occurred: {str(e)}\")" + ] + }, + { + "cell_type": "markdown", + "id": "dkMEzG2GZepL", + "metadata": { + "id": "dkMEzG2GZepL" + }, + "source": [ + "## Step 7. Run Inference\n", + "\n", + "You need to format your input instance as JSON run sample inference.\n", + "\n", + "

[go to top](#outline)

" + ] + }, + { + "cell_type": "markdown", + "id": "8pDhek1LqQTm", + "metadata": { + "id": "8pDhek1LqQTm" + }, + "source": [ + "### a. Create Payload" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "sby_UTdWWXGo", + "metadata": { + "id": "sby_UTdWWXGo" + }, + "outputs": [], + "source": [ + "import json\n", + "\n", + "messages = [\n", + " {\n", + " \"content\": \"You are a polite and respectful chatbot helping people plan a vacation.\",\n", + " \"role\": \"system\",\n", + " },\n", + " {\"content\": \"What should I do for a 4 day vacation in Spain?\", \"role\": \"user\"},\n", + "]\n", + "\n", + "payload = {\"model\": model_wo_tag, \"messages\": messages, \"max_tokens\": 4096, \"top_p\": 1}\n", + "\n", + "with open(\"request.json\", \"w\") as outfile:\n", + " json.dump(payload, outfile)\n", + "\n", + "# Streaming\n", + "payload_s = {\n", + " \"model\": model_wo_tag,\n", + " \"messages\": messages,\n", + " \"max_tokens\": 4096,\n", + " \"top_p\": 1,\n", + " \"stream\": True,\n", + "}\n", + "\n", + "with open(\"request_stream.json\", \"w\") as outfile:\n", + " json.dump(payload_s, outfile)" + ] + }, + { + "cell_type": "markdown", + "id": "dJbAepESZkWH", + "metadata": { + "id": "dJbAepESZkWH" + }, + "source": [ + "### b. Test Inference\n", + "" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "WR-AjyX0WgTf", + "metadata": { + "id": "WR-AjyX0WgTf" + }, + "outputs": [], + "source": [ + "import json\n", + "from pprint import pprint\n", + "\n", + "from google.api import httpbody_pb2\n", + "from google.cloud import aiplatform_v1\n", + "\n", + "client_options = {\"api_endpoint\": API_ENDPOINT}\n", + "\n", + "http_body = httpbody_pb2.HttpBody(\n", + " data=json.dumps(payload).encode(\"utf-8\"),\n", + " content_type=\"application/json\",\n", + ")\n", + "\n", + "try:\n", + " req = aiplatform_v1.RawPredictRequest(\n", + " http_body=http_body, endpoint=endpoint.resource_name\n", + " )\n", + "\n", + " print(\"Request:\")\n", + " pprint(json.loads(req.http_body.data))\n", + "\n", + " pred_client = aiplatform.gapic.PredictionServiceClient(\n", + " client_options=client_options\n", + " )\n", + "\n", + " response = pred_client.raw_predict(req)\n", + "\n", + " print(\n", + " \"--------------------------------------------------------------------------------------\"\n", + " )\n", + " print(\"Response:\")\n", + " pprint(json.loads(response.data))\n", + "except Exception as e:\n", + " print(f\"An error occurred: {str(e)}\")" + ] + }, + { + "cell_type": "markdown", + "id": "ZEHodX6eZs-R", + "metadata": { + "id": "ZEHodX6eZs-R" + }, + "source": [ + "### c. Test Inferencing (Streaming)\n", + "" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "bs2g_UJnWqYx", + "metadata": { + "id": "bs2g_UJnWqYx" + }, + "outputs": [], + "source": [ + "# Streaming\n", + "\n", + "import json\n", + "from pprint import pprint\n", + "\n", + "from google.api import httpbody_pb2\n", + "from google.cloud import aiplatform_v1\n", + "\n", + "client_options = {\"api_endpoint\": API_ENDPOINT}\n", + "\n", + "http_body = httpbody_pb2.HttpBody(\n", + " data=json.dumps(payload_s).encode(\"utf-8\"),\n", + " content_type=\"application/json\",\n", + ")\n", + "\n", + "try:\n", + " req = aiplatform_v1.RawPredictRequest(\n", + " http_body=http_body, endpoint=endpoint.resource_name\n", + " )\n", + "\n", + " print(\"Request:\")\n", + " pprint(json.loads(req.http_body.data))\n", + "\n", + " pred_client = aiplatform.gapic.PredictionServiceClient(\n", + " client_options=client_options\n", + " )\n", + "\n", + " response = pred_client.raw_predict(req)\n", + " print(\n", + " \"--------------------------------------------------------------------------------------\"\n", + " )\n", + " print(\"Response:\")\n", + " print(response.data.decode(\"utf-8\"))\n", + "except Exception as e:\n", + " print(f\"An error occurred: {str(e)}\")" + ] + }, + { + "cell_type": "markdown", + "id": "-xYIV31DCZ8k", + "metadata": { + "id": "-xYIV31DCZ8k" + }, + "source": [ + "## Step 8. Teardown the setup\n", + "\n", + "\n", + "This is an optional step to delete all the infrastructure provisioned in the previous steps.\n", + "

[go to top](#outline)

\n" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "Z2y_n_dSCgKt", + "metadata": { + "id": "Z2y_n_dSCgKt" + }, + "outputs": [], + "source": [ + "if endpoint:\n", + " print(f\"Deleting endpoint {endpoint.display_name}\")\n", + " endpoint.undeploy_all()\n", + " endpoint.delete()\n", + "\n", + "if model:\n", + " print(f\"Deleting model {model.display_name}\")\n", + " model.delete()\n", + "\n", + "! gcloud artifacts docker images delete \\\n", + " --delete-tags {NIM_IMAGE_GAR} \\\n", + " --quiet\n", + "\n", + "! gcloud artifacts repositories delete {REPOSITORY_NAME} \\\n", + " --location={LOCATION} \\\n", + " --quiet\n", + "\n", + "!gcloud workbench instances delete {WORKBENCH_NAME} \\\n", + " --project={PROJECT_ID} \\\n", + " --location={ZONE}" + ] + } + ], + "metadata": { + "colab": { + "name": "nvidianims_vertexai.ipynb", + "toc_visible": true + }, + "kernelspec": { + "display_name": "Python 3", + "name": "python3" + } + }, + "nbformat": 4, + "nbformat_minor": 0 +}