diff --git a/notebooks/community/prediction/custom_prediction_routines/SDK_Triton_PyTorch_Local_Prediction.ipynb b/notebooks/community/prediction/custom_prediction_routines/SDK_Triton_PyTorch_Local_Prediction.ipynb new file mode 100644 index 000000000..badfd871f --- /dev/null +++ b/notebooks/community/prediction/custom_prediction_routines/SDK_Triton_PyTorch_Local_Prediction.ipynb @@ -0,0 +1,1857 @@ +{ + "cells": [ + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "ur8xi4C7S06n" + }, + "outputs": [], + "source": [ + "# Copyright 2022 Google LLC\n", + "#\n", + "# Licensed under the Apache License, Version 2.0 (the \"License\");\n", + "# you may not use this file except in compliance with the License.\n", + "# You may obtain a copy of the License at\n", + "#\n", + "# https://www.apache.org/licenses/LICENSE-2.0\n", + "#\n", + "# Unless required by applicable law or agreed to in writing, software\n", + "# distributed under the License is distributed on an \"AS IS\" BASIS,\n", + "# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n", + "# See the License for the specific language governing permissions and\n", + "# limitations under the License." + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "JAPoU8Sm5E6e" + }, + "source": [ + "\n", + " \n", + " \n", + "
\n", + " \n", + " \"GitHub\n", + " View on GitHub\n", + " \n", + " \n", + " \n", + " \"Vertex\n", + " Open in Vertex AI Workbench\n", + " \n", + "
" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "tvgnzT1CKxrO" + }, + "source": [ + "## Overview\n", + "\n", + "This tutorial demonstrates how to use Vertex AI SDK to locally test [NVIDIA Triton inference server](https://developer.nvidia.com/nvidia-triton-inference-server) to serve a PyTorch model and deploy it to Vertex AI Predictions. This is currently an **experimental** feature and is not yet officially supported by the Vertex AI SDK. In this tutorial, you will be installing the Vertex AI SDK from an experimental branch on github. \n", + "\n", + "\n", + "\n", + "### Dataset\n", + "\n", + "This tutorial uses R.A. Fisher's Iris dataset, a small dataset that is popular for trying out machine learning techniques. Each instance has four numerical features, which are different measurements of a flower, and a target label that\n", + "marks it as one of three types of iris: Iris setosa, Iris versicolour, or Iris virginica.\n", + "\n", + "This tutorial uses [the Iris dataset](https://archive.ics.uci.edu/ml/machine-learning-databases/iris).\n", + "\n", + "### Objective\n", + "\n", + "The goal is to:\n", + "- Train a model that uses a flower's measurements as input to predict what type of iris it is with PyTorch.\n", + "- Save the model.\n", + "- Test the NVIDIA Triton inference server locally.\n", + "- Upload and deploy Triton inference server as custom container to Vertex Prediction.\n", + "\n", + "This tutorial focuses more on deploying this model with Vertex AI than on\n", + "the design of the model itself.\n", + "\n", + "### Costs \n", + "\n", + "This tutorial uses billable components of Google Cloud:\n", + "\n", + "* Vertex AI\n", + "\n", + "Learn about [Vertex AI pricing](https://cloud.google.com/vertex-ai/pricing), and use the [Pricing Calculator](https://cloud.google.com/products/calculator/) to generate a cost estimate based on your projected usage." + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "ze4-nDLfK4pw" + }, + "source": [ + "### Set up your local development environment\n", + "\n", + "**If you are using Vertex AI Workbench Notebooks**, your environment already meets\n", + "all the requirements to run this notebook. You can skip this step." + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "gCuSR8GkAgzl" + }, + "source": [ + "**Otherwise**, make sure your environment meets this notebook's requirements.\n", + "You need the following:\n", + "\n", + "* Docker\n", + "* Git\n", + "* Google Cloud SDK (gcloud)\n", + "* Python 3\n", + "* virtualenv\n", + "* Jupyter notebook running in a virtual environment with Python 3\n", + "\n", + "The Google Cloud guide to [Setting up a Python development\n", + "environment](https://cloud.google.com/python/setup) and the [Jupyter\n", + "installation guide](https://jupyter.org/install) provide detailed instructions\n", + "for meeting these requirements. The following steps provide a condensed set of\n", + "instructions:\n", + "\n", + "1. [Install and initialize the Cloud SDK.](https://cloud.google.com/sdk/docs/)\n", + "\n", + "1. [Install Python 3.](https://cloud.google.com/python/setup#installing_python)\n", + "\n", + "1. [Install\n", + " virtualenv](https://cloud.google.com/python/setup#installing_and_using_virtualenv)\n", + " and create a virtual environment that uses Python 3. Activate the virtual environment.\n", + "\n", + "1. To install Jupyter, run `pip install jupyter` on the\n", + "command-line in a terminal shell.\n", + "\n", + "1. To launch Jupyter, run `jupyter notebook` on the command-line in a terminal shell.\n", + "\n", + "1. Open this notebook in the Jupyter Notebook Dashboard." + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "i7EUnXsZhAGF" + }, + "source": [ + "### Install additional packages\n", + "\n", + "Install additional package dependencies not installed in your notebook environment, such as NumPy, Scikit-learn, FastAPI, Uvicorn, and joblib." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "6d0fb1cd95f4" + }, + "outputs": [], + "source": [ + "%%writefile requirements.txt\n", + "pandas\n", + "torch==1.11.0\n", + "google-cloud-storage>=1.26.0,<2.0.0dev\n", + "google-cloud-aiplatform[prediction] @ git+https://github.com/googleapis/python-aiplatform.git@custom-prediction-routine" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "356f45ac7942" + }, + "source": [ + "**The model you deploy will have a different set of dependencies pre-installed than your notebook environment has. You should not assume that because things work in the notebook, they will work in the model. Instead, you will be very explicit about the dependencies for the model by listing them in requirements.txt and then use `pip install` to install the exact same dependencies in the notebook. Please note, of course, that there is a chance that you miss a dependency in requirements.txt that already exists in the notebook. If that's the case, things will run in the notebook, but not in the model. To guard against that, you will test the model locally before deploying to the cloud.**" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "f7cbec0adc49" + }, + "outputs": [], + "source": [ + "# Install the same dependencies used in the serving container in the notebook\n", + "# environment.\n", + "%pip install -U --user -r requirements.txt" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "hhq5zEbGg0XX" + }, + "source": [ + "### Restart the kernel\n", + "\n", + "After you install the additional packages, you need to restart the notebook kernel so it can find the packages." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "c1464805870e" + }, + "outputs": [], + "source": [ + "# Automatically restart kernel after installs\n", + "import os\n", + "\n", + "if not os.getenv(\"IS_TESTING\"):\n", + " # Automatically restart kernel after installs\n", + " import IPython\n", + "\n", + " app = IPython.Application.instance()\n", + " app.kernel.do_shutdown(True)" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "lWEdiXsJg0XY" + }, + "source": [ + "## Before you begin" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "64a0a87c5b66" + }, + "source": [ + "### Set up logging in this sample" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "348ca5392459" + }, + "outputs": [], + "source": [ + "import logging\n", + "\n", + "logging.basicConfig(level=logging.INFO)" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "BF1j6f9HApxa" + }, + "source": [ + "### Set up your Google Cloud project\n", + "\n", + "**The following steps are required, regardless of your notebook environment.**\n", + "\n", + "1. [Select or create a Google Cloud project](https://console.cloud.google.com/cloud-resource-manager). When you first create an account, you get a $300 free credit towards your compute/storage costs.\n", + "\n", + "1. [Make sure that billing is enabled for your project](https://cloud.google.com/billing/docs/how-to/modify-project).\n", + "\n", + "1. [Enable the Vertex AI API and Compute Engine API](https://console.cloud.google.com/flows/enableapi?apiid=aiplatform.googleapis.com,compute_component).\n", + "\n", + "1. If you are running this notebook locally, you will need to install the [Cloud SDK](https://cloud.google.com/sdk).\n", + "\n", + "1. Enter your project ID in the cell below. Then run the cell to make sure the\n", + "Cloud SDK uses the right project for all the commands in this notebook.\n", + "\n", + "**Note**: Jupyter runs lines prefixed with `!` or `%` as shell commands, and it interpolates Python variables with `$` or `{}` into these commands." + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "WReHDGG5g0XY" + }, + "source": [ + "#### Set your project ID\n", + "\n", + "**If you don't know your project ID**, you may be able to get your project ID using `gcloud`." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "oM1iC_MfAts1" + }, + "outputs": [], + "source": [ + "import os\n", + "\n", + "PROJECT_ID = \"\"\n", + "\n", + "# Get your Google Cloud project ID from gcloud\n", + "if not os.getenv(\"IS_TESTING\"):\n", + " shell_output = !gcloud config list --format 'value(core.project)' 2>/dev/null\n", + " PROJECT_ID = shell_output[0]\n", + " print(\"Project ID: \", PROJECT_ID)" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "qJYoRfYng0XZ" + }, + "source": [ + "Otherwise, set your project ID here." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "riG_qUokg0XZ" + }, + "outputs": [], + "source": [ + "if PROJECT_ID == \"\" or PROJECT_ID is None:\n", + " PROJECT_ID = \"[your-project-id]\" # @param {type:\"string\"}" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "set_gcloud_project_id" + }, + "outputs": [], + "source": [ + "! gcloud config set project $PROJECT_ID" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "region" + }, + "source": [ + "#### Region\n", + "\n", + "You can also change the `REGION` variable, which is used for operations\n", + "throughout the rest of this notebook. Below are regions supported for Vertex AI. We recommend that you choose the region closest to you.\n", + "\n", + "- Americas: `us-central1`\n", + "- Europe: `europe-west4`\n", + "- Asia Pacific: `asia-east1`\n", + "\n", + "You may not use a multi-regional bucket for training with Vertex AI. Not all regions provide support for all Vertex AI services.\n", + "\n", + "Learn more about [Vertex AI regions](https://cloud.google.com/vertex-ai/docs/general/locations)." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "region" + }, + "outputs": [], + "source": [ + "REGION = \"[your-region]\" # @param {type: \"string\"}\n", + "\n", + "if REGION == \"[your-region]\":\n", + " REGION = \"us-central1\"" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "XoEqT2Y4DJmf" + }, + "source": [ + "### Configure project and resource names" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "MzGDU7TWdts_" + }, + "outputs": [], + "source": [ + "MODEL_ARTIFACT_DIR = \"triton-pytorch\" # @param {type:\"string\"}\n", + "REPOSITORY = \"custom-container-prediction-sdk\" # @param {type:\"string\"}\n", + "IMAGE = \"triton-pytorch\" # @param {type:\"string\"}\n", + "MODEL_DISPLAY_NAME = \"triton-pytorch\" # @param {type:\"string\"}" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "ca1a915d641d" + }, + "source": [ + "`MODEL_ARTIFACT_DIR` - Folder directory path to your model artifacts within a Cloud Storage bucket, for example: \"my-models/fraud-detection/trial-4\"\n", + "\n", + "`REPOSITORY` - Name of the Artifact Repository to create or use.\n", + "\n", + "`IMAGE` - Name of the container image that will be pushed.\n", + "\n", + "`MODEL_DISPLAY_NAME` - Display name of Vertex AI Model resource." + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "zgPO1eR3CYjk" + }, + "source": [ + "### Create a Cloud Storage bucket\n", + "\n", + "**The following steps are required, regardless of your notebook environment.**\n", + "\n", + "To update your model artifacts without re-building the container, you must upload your model\n", + "artifacts and any custom code to Cloud Storage.\n", + "\n", + "Set the name of your Cloud Storage bucket below. It must be unique across all\n", + "Cloud Storage buckets." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "MzGDU7TWdts_" + }, + "outputs": [], + "source": [ + "BUCKET_NAME = \"[your-bucket-name]\" # @param {type:\"string\"}\n", + "BUCKET_URI = f\"gs://{BUCKET_NAME}\"" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "cf221059d072" + }, + "outputs": [], + "source": [ + "if BUCKET_NAME == \"\" or BUCKET_NAME is None or BUCKET_NAME == \"[your-bucket-name]\":\n", + " BUCKET_NAME = PROJECT_ID + \"aip-\" + TIMESTAMP\n", + " BUCKET_URI = f\"gs://{BUCKET_NAME}\"" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "-EcIXiGsCePi" + }, + "source": [ + "**Only if your bucket doesn't already exist**: Run the following cell to create your Cloud Storage bucket." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "NIq7R4HZCfIc" + }, + "outputs": [], + "source": [ + "! gsutil mb -l $REGION -p $PROJECT_ID $BUCKET_URI" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "ucvCsknMCims" + }, + "source": [ + "Finally, validate access to your Cloud Storage bucket by examining its contents:" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "vhOb7YnwClBb" + }, + "outputs": [], + "source": [ + "! gsutil ls -al $BUCKET_URI" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "3c2d091d9e73" + }, + "source": [ + "### Set up directories" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "3c2d091d9e73" + }, + "source": [ + "Decide the directory to put your model artifacts." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "3c2d091d9e73" + }, + "outputs": [], + "source": [ + "MODEL_ARTIFACTS_DIRECTORY = \"model_artifacts\" # @param {type:\"string\"}" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "6e74556ea0b4" + }, + "outputs": [], + "source": [ + "%mkdir $MODEL_ARTIFACTS_DIRECTORY" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "a8b6a08d0d97" + }, + "source": [ + "## Training a PyTorch model\n", + "\n", + "### Download iris data\n", + "In this example, you want to build a classifier for the simple [iris dataset](https://archive.ics.uci.edu/ml/datasets/iris). So first, you download the data csv file locally." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "c2d231f9e522" + }, + "outputs": [], + "source": [ + "DATA_DIR = \"data_iris\"\n", + "\n", + "%mkdir $DATA_DIR\n", + "\n", + "LOCAL_DATA_FILE = f\"{DATA_DIR}/iris.csv\"" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "86cda068907b" + }, + "outputs": [], + "source": [ + "from urllib.request import urlretrieve\n", + "\n", + "urlretrieve(\n", + " \"https://archive.ics.uci.edu/ml/machine-learning-databases/iris/iris.data\",\n", + " LOCAL_DATA_FILE,\n", + ")" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "059236726bb9" + }, + "source": [ + "### Build a PyTorch NN Classifier" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "4b816cd52f4b" + }, + "source": [ + "Make sure that pytorch package is [installed](https://pytorch.org/get-started/locally/)." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "43e47249f736" + }, + "outputs": [], + "source": [ + "import torch\n", + "from torch.autograd import Variable\n", + "\n", + "print(\"PyTorch Version: {}\".format(torch.__version__))" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "6732994d86ef" + }, + "source": [ + "#### Step 1. Load data\n", + "\n", + "In this step, you are going to:\n", + "1. Load the data to Pandas Dataframe.\n", + "1. Convert the class feature (species) from string to a numeric indicator.\n", + "1. Split the Dataframe into input feature (xtrain) and target feature (ytrain)." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "6259129f102c" + }, + "outputs": [], + "source": [ + "import pandas as pd\n", + "\n", + "CLASS_VOCAB = [\"setosa\", \"versicolor\", \"virginica\"]\n", + "\n", + "datatrain = pd.read_csv(\n", + " LOCAL_DATA_FILE,\n", + " names=[\"sepal_length\", \"sepal_width\", \"petal_length\", \"petal_width\", \"species\"],\n", + ")\n", + "\n", + "# change string value to numeric\n", + "datatrain.loc[datatrain[\"species\"] == \"Iris-setosa\", \"species\"] = 0\n", + "datatrain.loc[datatrain[\"species\"] == \"Iris-versicolor\", \"species\"] = 1\n", + "datatrain.loc[datatrain[\"species\"] == \"Iris-virginica\", \"species\"] = 2\n", + "datatrain = datatrain.apply(pd.to_numeric)\n", + "\n", + "# change dataframe to array\n", + "datatrain_array = datatrain.values\n", + "\n", + "# split x and y (feature and target)\n", + "xtrain = datatrain_array[:, :4]\n", + "ytrain = datatrain_array[:, 4]\n", + "\n", + "input_features = xtrain.shape[1]\n", + "num_classes = len(CLASS_VOCAB)\n", + "\n", + "print(\"Records loaded: {}\".format(len(xtrain)))\n", + "print(\"Number of input features: {}\".format(input_features))\n", + "print(\"Number of classes: {}\".format(num_classes))" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "f1fb3cfa67a2" + }, + "source": [ + "#### Step 2. Set model parameters\n", + "You can try different values for **hidden_units** or **learning_rate**." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "b964ab7b0826" + }, + "outputs": [], + "source": [ + "HIDDEN_UNITS = 10\n", + "LEARNING_RATE = 0.1" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "83767e4aa8ec" + }, + "source": [ + "#### Step 3. Define the PyTorch NN model\n", + "Here, you build a a neural network with one hidden layer, and a Softmax output layer for classification." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "665b1a79e813" + }, + "outputs": [], + "source": [ + "model = torch.nn.Sequential(\n", + " torch.nn.Linear(input_features, HIDDEN_UNITS),\n", + " torch.nn.Sigmoid(),\n", + " torch.nn.Linear(HIDDEN_UNITS, num_classes),\n", + " torch.nn.Softmax(),\n", + ")\n", + "\n", + "loss_metric = torch.nn.CrossEntropyLoss()\n", + "optimizer = torch.optim.SGD(model.parameters(), lr=LEARNING_RATE)" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "e74dd9676f18" + }, + "source": [ + "#### Step 4. Train the model\n", + "You are going to train the model for **num_epoch** epochs." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "8ff929fcd864" + }, + "outputs": [], + "source": [ + "NUM_EPOCHS = 10000\n", + "\n", + "for epoch in range(NUM_EPOCHS):\n", + "\n", + " x = Variable(torch.Tensor(xtrain).float())\n", + " y = Variable(torch.Tensor(ytrain).long())\n", + " optimizer.zero_grad()\n", + " y_pred = model(x)\n", + " loss = loss_metric(y_pred, y)\n", + " loss.backward()\n", + " optimizer.step()\n", + " if (epoch) % 1000 == 0:\n", + " print(\n", + " \"Epoch [{}/{}] Loss: {}\".format(\n", + " epoch + 1, NUM_EPOCHS, round(loss.item(), 3)\n", + " )\n", + " )\n", + "\n", + "print(\"Epoch [{}/{}] Loss: {}\".format(epoch + 1, NUM_EPOCHS, round(loss.item(), 3)))" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "bb6f316f99ea" + }, + "source": [ + "#### Step 5. Save the model\n", + "\n", + "Triton inference server requires PyTorch models to be saved as [the TorchScript format](https://pytorch.org/tutorials/beginner/Intro_to_TorchScript_tutorial.html)." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "18db4fa0de42" + }, + "outputs": [], + "source": [ + "# An example input you would normally provide to your model's forward() method.\n", + "example = torch.rand(1, 4)" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "504874630adf" + }, + "outputs": [], + "source": [ + "MODEL_NAME = \"pytorch\"\n", + "LOCAL_MODEL_DIRECTORY = f\"{MODEL_ARTIFACTS_DIRECTORY}/{MODEL_NAME}\"\n", + "LOCAL_MODEL_VERSION_1 = f\"{LOCAL_MODEL_DIRECTORY}/1\"\n", + "\n", + "%mkdir $LOCAL_MODEL_DIRECTORY\n", + "%mkdir $LOCAL_MODEL_VERSION_1" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "7f95dcdd29d9" + }, + "outputs": [], + "source": [ + "LOCAL_MODEL_FILE = f\"{LOCAL_MODEL_VERSION_1}/model.pt\"\n", + "\n", + "# Use torch.jit.trace to generate a torch.jit.ScriptModule via tracing.\n", + "traced_script_module = torch.jit.trace(model, example)\n", + "\n", + "# Save the TorchScript model\n", + "traced_script_module.save(LOCAL_MODEL_FILE)" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "40d0d3e86a2f" + }, + "source": [ + "#### Step 6. Test the loaded TorchScript model for predictions" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "c260e0606bcd" + }, + "outputs": [], + "source": [ + "iris_classifier = torch.jit.load(LOCAL_MODEL_FILE)\n", + "\n", + "\n", + "def predict_class(instances):\n", + " instances = torch.Tensor(instances)\n", + " output = iris_classifier(instances)\n", + " _, predicted = torch.max(output, 1)\n", + " return predicted\n", + "\n", + "\n", + "predicted = predict_class(xtrain[0:5])\n", + "print([CLASS_VOCAB[class_index] for class_index in predicted])" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "a7781d51a090" + }, + "source": [ + "## Prepare for model configuration for Triton\n", + "\n", + "Each model in a [model repository](https://github.com/triton-inference-server/server/blob/main/docs/model_repository.md) must include a [model configuration](https://github.com/triton-inference-server/server/blob/main/docs/model_configuration.md) that provides required and optional information about the model. A Triton backend is the implementation that executes a model, which can be a wrapper around a deep-learning framework. A backend can also implement any functionality you want as long as it adheres to the [backend API](https://github.com/triton-inference-server/backend/blob/main/README.md#triton-backend-api). Triton uses this API to send requests to the backend for execution and the backend uses the API to communicate with Triton. Every model must be associated with a backend. A model's backend is specified in the model's configuration using the `backend` and `platform` settings.\n", + "\n", + "[To load PyTorch models](https://github.com/triton-inference-server/backend/blob/main/README.md#backends), the model configuration needs to set `backend` and `platform` to `pytorch` and `pytorch_libtorch` respectively. There are also [special conventions for PyTorch backend](https://github.com/triton-inference-server/server/blob/main/docs/model_configuration.md)." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "05da1e885e08" + }, + "outputs": [], + "source": [ + "%%writefile $LOCAL_MODEL_DIRECTORY/config.pbtxt\n", + "backend: \"pytorch\"\n", + "platform: \"pytorch_libtorch\"\n", + "max_batch_size: 8\n", + "input {\n", + " name: \"INPUT__0\"\n", + " data_type: TYPE_FP32\n", + " dims: [ 4 ]\n", + "}\n", + "output {\n", + " name: \"OUTPUT__0\"\n", + " data_type: TYPE_FP32\n", + " dims: [ 1 ]\n", + "}" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "3849066a33bd" + }, + "source": [ + "## Upload model artifacts to Cloud Storage\n", + "\n", + "Before you can deploy your model for serving, Vertex AI needs access to the following files in Cloud Storage:\n", + "\n", + "* `model.pt` (model artifact)\n", + "* `config.pbtxt` (model configuration)\n", + "\n", + "Run the following commands to upload your files:" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "ca67ee52d4d9" + }, + "outputs": [], + "source": [ + "!gsutil cp -r {MODEL_ARTIFACTS_DIRECTORY}/* {BUCKET_URI}/{MODEL_ARTIFACT_DIR}/\n", + "!gsutil ls {BUCKET_URI}/{MODEL_ARTIFACT_DIR}/" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "490ee68c86c4" + }, + "source": [ + "## Use NVIDIA Triton inference server\n", + "\n", + "### Initiate Local Model with NVIDIA Triton inference server\n", + "\n", + "In this tutorial, you use [NVIDIA Triton inference server](https://developer.nvidia.com/nvidia-triton-inference-server) to serve the PyTorch model. The images are hosted on `nvcr.io` and all released tags can be found in [this link](https://catalog.ngc.nvidia.com/orgs/nvidia/containers/tritonserver/tags)." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "63e14c94ed75" + }, + "outputs": [], + "source": [ + "TRITON_VERSION = \"21.08\"" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "a039b5368233" + }, + "source": [ + "With the Custom Prediction Routine features, Vertex AI SDK allows you to load the existing images to test them locally and then deploy them to Vertex AI Prediction easily.\n", + "\n", + "To use Triton inference server on Vertex AI, you need to set up needed parameters such as predict route, health route, ports, and args. By default, Triton listens for HTTP requests on port `8000`. A quickstart of Triton can be found [here](https://github.com/triton-inference-server/server/blob/main/docs/quickstart.md#run-triton)." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "697b4a2bb11e" + }, + "outputs": [], + "source": [ + "from google.cloud.aiplatform.prediction import LocalModel\n", + "\n", + "local_model = LocalModel.create(\n", + " serving_container_image_uri=f\"nvcr.io/nvidia/tritonserver:{TRITON_VERSION}-py3\",\n", + " serving_container_predict_route=f\"/v2/models/{MODEL_NAME}/infer\",\n", + " serving_container_health_route=f\"/v2/models/{MODEL_NAME}\",\n", + " serving_container_ports=[8000],\n", + " serving_container_args=[\"tritonserver\", \"--model-repository=$(AIP_STORAGE_URI)\"],\n", + ")" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "155e6838aa70" + }, + "source": [ + "You can check out the serving container spec of the image." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "6c427921a9ce" + }, + "outputs": [], + "source": [ + "local_model.get_serving_container_spec()" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "8b62ddf1def3" + }, + "source": [ + "### Store test instances\n", + "\n", + "To learn more about formatting input instances in JSON, [read the documentation.](https://cloud.google.com/vertex-ai/docs/predictions/online-predictions-custom-models#request-body-details)" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "7d93f7e01e04" + }, + "outputs": [], + "source": [ + "INPUT_FILE = \"instances.json\"" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "b6605e9e6186" + }, + "outputs": [], + "source": [ + "%%writefile $INPUT_FILE\n", + "{\n", + " \"id\": \"0\",\n", + " \"inputs\": [\n", + " {\n", + " \"name\": \"INPUT__0\",\n", + " \"shape\": [2, 4],\n", + " \"datatype\": \"FP32\",\n", + " \"data\": [[6.7, 3.1, 4.7, 1.5], [4.6, 3.1, 1.5, 0.2]]\n", + " }\n", + " ]\n", + "}" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "87f8d4d7a9f6" + }, + "source": [ + "### Run and test the container locally" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "9a213b08fcf4" + }, + "source": [ + "Custom Prediction Routines provide two ways to download model artifacts for testing images locally.\n", + "1. A local path.\n", + "2. A GCS path.\n", + "\n", + "To use a local path, your `Predictor` needs to support both ways to load models so it can be tested locally and deployed to Vertex AI Prediction service. SDK provides a method `download_model_artifacts` to support these two ways in [prediction_utils.py](https://github.com/googleapis/python-aiplatform/blob/custom-prediction-routine/google/cloud/aiplatform/utils/prediction_utils.py) which you can call in the `load` function in your `Predictor`.\n", + "\n", + "If you want to test images locally with a GCS path, you need to follow the instructions to set up the credentials." + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "8phJWBhe7u9u" + }, + "source": [ + "#### Set up credentials\n", + "\n", + "Setting up credentials is only required to run the custom serving container locally with GCS paths. Credentials set up is required to execute the `Predictor`'s `load` function, which downloads the model artifacts from Google Cloud Storage.\n", + "\n", + "To access Google Cloud Storage in your project, you'll need to set up credentials by using one of the following:\n", + "\n", + "1. User account\n", + "\n", + "1. Service account\n", + "\n", + "You can learn more about each of the above [here](https://cloud.google.com/docs/authentication#principals)." + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "5261966581d2" + }, + "source": [ + "**Option 1. Use Google user credentials**\n", + "\n", + "First authorize Google auth library to access the Cloud Platform with Google user credentials. This step involves an interactive prompt which requires user inputs. If you are using a non-interactive shell, you should open a terminal to run this command. See [here](https://jupyterlab.readthedocs.io/en/stable/user/terminal.html) for how to open a terminal in a Vertex Workbench notebook environment.\n", + "\n", + "Note that if you are running the command below on a machine without a window manager/browser, it will prompt you to run the command `gcloud auth application-default login --remote-bootstrap=\"...\"`. Only machines that can launch web browsers, e.g. laptops or workstations, are allowed to run this command. After completing the authentication steps using your browser, you'll need to copy and paste the verification code back into the original terminal to continue the login flow. If you are running this command on a machine with web browser support, there is no need to use a separate machine.\n", + "\n", + "This command will print the credential file as `Credentials saved to file: [CREDENTIALS_FILE]`." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "2f9d26108e4c" + }, + "outputs": [], + "source": [ + "!gcloud auth application-default login" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "90808ec3bb54" + }, + "source": [ + "Specify the credential file, which is a json file, as the path." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "816b9864614c" + }, + "outputs": [], + "source": [ + "CREDENTIALS_FILE = \"[CREDENTIALS_FILE]\" # @param {type:\"string\"}" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "dfba3a7a8907" + }, + "source": [ + "Next, authorize gcloud to access the Cloud Platform with Google user credentials. The user credentials need to have `setIamPolicy` permission in the project. See [here](https://cloud.google.com/resource-manager/docs/access-control-proj) for more details.\n", + "\n", + "Similar to the previous step, this also requires user inputs. Instructions for the previous step apply here as well." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "dfe6dae6464c" + }, + "outputs": [], + "source": [ + "!gcloud auth login" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "ff5fc2640fff" + }, + "source": [ + "Grant roles to the user account. Use the email address that you used in the gcloud auth step, e.g., `myemail@gmail.com`. Since you will be using this user account to download the model artifacts from Google Cloud Storage, you grant the user account **Storage Object Viewer** role. See [IAM roles for Cloud Storage](https://cloud.google.com/storage/docs/access-control/iam-roles) for more details." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "15e6a2d55290" + }, + "outputs": [], + "source": [ + "USER_ACCOUNT = \"[USER_ACCOUNT]\" # @param {type:\"string\"}\n", + "\n", + "!gcloud projects add-iam-policy-binding $PROJECT_ID \\\n", + " --member=user:$USER_ACCOUNT --role=roles/storage.objectViewer" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "49516700e7fd" + }, + "source": [ + "Initialize the Vertex AI SDK with the project ID, which is required for using user credentials." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "a919f3d1a0e7" + }, + "outputs": [], + "source": [ + "from google.cloud import aiplatform\n", + "\n", + "aiplatform.init(project=PROJECT_ID, location=REGION)" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "23907a99baa7" + }, + "source": [ + "**Option 2. Use Google Service Account credentials**\n", + "\n", + "Skip this if you've already completed Option 1 above. First enable the IAM API if it's not already enabled." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "mERTlLb6dluB" + }, + "outputs": [], + "source": [ + "!gcloud services enable iam.googleapis.com" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "set_service_account" + }, + "source": [ + "#### Service Account\n", + "\n", + "You use a service account to download model artifacts while you run containers locally. If you do not want to use your project's Compute Engine service account, set `SERVICE_ACCOUNT` to another service account ID." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "set_service_account" + }, + "outputs": [], + "source": [ + "SERVICE_ACCOUNT = \"[your-service-account]\" # @param {type:\"string\"}" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "autoset_service_account" + }, + "outputs": [], + "source": [ + "if (\n", + " SERVICE_ACCOUNT == \"\"\n", + " or SERVICE_ACCOUNT is None\n", + " or SERVICE_ACCOUNT == \"[your-service-account]\"\n", + "):\n", + " # Get your service account from gcloud\n", + " if not IS_COLAB:\n", + " shell_output = !gcloud auth list 2>/dev/null\n", + " SERVICE_ACCOUNT = shell_output[2].replace(\"*\", \"\").strip()\n", + "\n", + " else: # IS_COLAB:\n", + " shell_output = ! gcloud projects describe $PROJECT_ID\n", + " project_number = shell_output[-1].split(\":\")[1].strip().replace(\"'\", \"\")\n", + " SERVICE_ACCOUNT = f\"{project_number}-compute@developer.gserviceaccount.com\"\n", + "\n", + " print(\"Service Account:\", SERVICE_ACCOUNT)" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "c57086a991ef" + }, + "source": [ + "Create the service account." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "aec4edb3bb32" + }, + "outputs": [], + "source": [ + "!gcloud iam service-accounts create $SERVICE_ACCOUNT" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "9676dfef3d3b" + }, + "source": [ + "Authorize gcloud to access the Cloud Platform with Google user credentials. The user credentials need to have `setIamPolicy` permission in the project. See [here](https://cloud.google.com/resource-manager/docs/access-control-proj) for more details.\n", + "\n", + "This step involves an interactive prompt which requires user inputs. If you are using a non-interactive shell, you should open a terminal to run this command. See [here](https://jupyterlab.readthedocs.io/en/stable/user/terminal.html) for how to open a terminal in a Vertex Workbench notebook environment.\n", + "\n", + "Note that if you are running the command below on a machine without a window manager/browser, it will prompt you to run the command `gcloud auth application-default login --remote-bootstrap=\"...\"`. Only machines that can launch web browsers, e.g. laptops or workstations, are allowed to run this command. After completing the authentication steps using your browser, you'll need to copy and paste the verification code back into the original terminal to continue the login flow. If you are running this command on a machine with web browser support, there is no need to use a separate machine." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "afca052d7d07" + }, + "outputs": [], + "source": [ + "!gcloud auth login" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "3017a89b1e05" + }, + "source": [ + "Grant roles to the service account. Since you will be using this service account to download the model artifacts from Google Cloud Storage, you grant the service account **Storage Object Viewer** role. See [IAM roles for Cloud Storage](https://cloud.google.com/storage/docs/access-control/iam-roles) for more details." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "11ee24d335bf" + }, + "outputs": [], + "source": [ + "!gcloud projects add-iam-policy-binding $PROJECT_ID \\\n", + " --member=serviceAccount:{SERVICE_ACCOUNT}@{PROJECT_ID}.iam.gserviceaccount.com \\\n", + " --role=roles/storage.objectViewer" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "1d65aba7673b" + }, + "source": [ + "Specify the service account key file name." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "4e4dea3da627" + }, + "outputs": [], + "source": [ + "CREDENTIALS_FILE = \"./credentials.json\"" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "62b7259652b3" + }, + "source": [ + "Generate the service account key file. This will be used while running the custom serving container locally." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "9bb3770c1ec1" + }, + "outputs": [], + "source": [ + "!gcloud iam service-accounts keys create $CREDENTIALS_FILE \\\n", + " --iam-account=\"{SERVICE_ACCOUNT}@{PROJECT_ID}.iam.gserviceaccount.com\"" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "pJe8TCr4ec4F" + }, + "source": [ + "#### Run and send requests to the container locally\n", + "\n", + "With the Custom Prediction Routine feature, it is easy to test the container locally.\n", + "\n", + "In this example, the container executes a prediction request and a health check.\n", + "\n", + "Note: It will be a bit slow for the first time you run the following command because you need to pull the image from `nvcr.io`." + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "84d432d67e2b" + }, + "source": [ + "**Option 1. Pass local paths**\n", + "\n", + "This option requires the `load` function in the `Predictor` supports loading model artifacts from both of GCS paths and local paths." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "62ed2d334d0f" + }, + "outputs": [], + "source": [ + "with local_model.deploy_to_local_endpoint(\n", + " artifact_uri=f\"{MODEL_ARTIFACTS_DIRECTORY}\",\n", + ") as local_endpoint:\n", + " predict_response = local_endpoint.predict(\n", + " request_file=INPUT_FILE,\n", + " headers={\"Content-Type\": \"application/json\"},\n", + " )\n", + "\n", + " health_check_response = local_endpoint.run_health_check()" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "236e1704871d" + }, + "source": [ + "Print out the predict response and its content." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "ce629eea32fd" + }, + "outputs": [], + "source": [ + "print(predict_response, predict_response.content)" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "1bf6c7d044bc" + }, + "source": [ + "Print out the health check response and its content." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "56986f93438e" + }, + "outputs": [], + "source": [ + "print(health_check_response, health_check_response.content)" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "a29fcbbe0188" + }, + "source": [ + "Also print out all the container logs. You will see the logs of container startup, serving requests, and container teardown." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "1cd47d55bbd0" + }, + "outputs": [], + "source": [ + "local_endpoint.print_container_logs(show_all=True)" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "e16789f2025b" + }, + "source": [ + "**Option 2. Pass GCS paths**\n", + "\n", + "If you want to pass GCS paths, you need to have the credentials set up in the previous step and pass the path to the credentials while running the container. The service account should have the **Storage Object Viewer** permission." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "b5f3af366697" + }, + "outputs": [], + "source": [ + "with local_model.deploy_to_local_endpoint(\n", + " artifact_uri=f\"{BUCKET_URI}/{MODEL_ARTIFACT_DIR}\",\n", + " credential_path=CREDENTIALS_FILE,\n", + ") as local_endpoint:\n", + " predict_response = local_endpoint.predict(\n", + " request_file=INPUT_FILE,\n", + " headers={\"Content-Type\": \"application/json\"},\n", + " )\n", + "\n", + " health_check_response = local_endpoint.run_health_check()" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "120d9382c22f" + }, + "source": [ + "Print out the predict response and its content." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "ce629eea32fd" + }, + "outputs": [], + "source": [ + "print(predict_response, predict_response.content)" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "cfe9c701b0e8" + }, + "source": [ + "Print out the health check response and its content." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "56986f93438e" + }, + "outputs": [], + "source": [ + "print(health_check_response, health_check_response.content)" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "a29fcbbe0188" + }, + "source": [ + "Also print out all the container logs. You will see the logs of container startup, serving requests, and container teardown." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "d312a6136b56" + }, + "outputs": [], + "source": [ + "local_endpoint.print_container_logs(show_all=True)" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "d713c4cc90b4" + }, + "source": [ + "### Change image uris to Artifact Registry\n", + "\n", + "Because Vertex AI Prediction currently does not support `nvcr.io` repositories, you need to copy the images to Artifact Registry. You can easily initiate another Local Model instance with the copied images." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "0e7e847b15b5" + }, + "outputs": [], + "source": [ + "local_model_ar = local_model.copy_image(\n", + " f\"{REGION}-docker.pkg.dev/{PROJECT_ID}/{REPOSITORY}/{IMAGE}\"\n", + ")" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "60ea52c3bc5b" + }, + "source": [ + "Print out the serving container spec of the new Local Model instance. The image uri should be changed to Artifact Registry." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "c46cc423b8ee" + }, + "outputs": [], + "source": [ + "local_model_ar.get_serving_container_spec()" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "212b2935ea12" + }, + "source": [ + "### Push the container to artifact registry\n", + "\n", + "Configure Docker to access Artifact Registry. Then push your container image to your Artifact Registry repository.\n", + "\n", + "Print out all enabled services in your project." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "wSFXCj3LdluJ" + }, + "outputs": [], + "source": [ + "!gcloud services list" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "ABE9UpwSdluK" + }, + "source": [ + "If `artifactregistry.googleapis.com` is not enabled in your project, enable the API before proceeding." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "qDhLoQMydluK" + }, + "outputs": [], + "source": [ + "!gcloud services enable artifactregistry.googleapis.com" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "09ffe2434e3d" + }, + "outputs": [], + "source": [ + "!gcloud beta artifacts repositories create {REPOSITORY} \\\n", + " --repository-format=docker \\\n", + " --location=$REGION" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "293437024749" + }, + "outputs": [], + "source": [ + "!gcloud auth configure-docker {REGION}-docker.pkg.dev --quiet" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "873892f3dbc7" + }, + "source": [ + "Use SDK to push the image." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "1dd7448f4703" + }, + "outputs": [], + "source": [ + "local_model_ar.push_image()" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "b438bfa2129f" + }, + "source": [ + "## Deploy to Vertex AI" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "4ae19df6a33e" + }, + "source": [ + "### Upload the NVIDIA Triton inference server model" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "8d682d8388ec" + }, + "outputs": [], + "source": [ + "from google.cloud import aiplatform" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "574fb82d3eed" + }, + "outputs": [], + "source": [ + "aiplatform.init(project=PROJECT_ID, location=REGION)" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "4879e61a1978" + }, + "source": [ + "Use the LocalModel instance to upload the model. It will populate the container spec automatically for you." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "2738154345d5" + }, + "outputs": [], + "source": [ + "model = local_model_ar.upload(\n", + " display_name=MODEL_DISPLAY_NAME,\n", + " artifact_uri=f\"{BUCKET_URI}/{MODEL_ARTIFACT_DIR}\",\n", + ")" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "bd1b85afc7df" + }, + "source": [ + "### Deploy the model on Vertex AI\n", + "After this step completes, the model is deployed and ready for online prediction." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "62cf66498a28" + }, + "outputs": [], + "source": [ + "endpoint = model.deploy(machine_type=\"n1-standard-4\")" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "6883e7b07143" + }, + "source": [ + "## Send predictions\n", + "\n", + "Need to use [raw predict](https://cloud.google.com/sdk/gcloud/reference/ai/endpoints/raw-predict) to send arbitrary payloads." + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "356d9ee4f53d" + }, + "source": [ + "### Using REST" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "4ddd5e816996" + }, + "outputs": [], + "source": [ + "ENDPOINT_ID = endpoint.name" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "2ad52431aecc" + }, + "outputs": [], + "source": [ + "! curl \\\n", + "-H \"Authorization: Bearer $(gcloud auth print-access-token)\" \\\n", + "-H \"Content-Type: application/json\" \\\n", + "--data-binary @$INPUT_FILE \\\n", + "https://{REGION}-aiplatform.googleapis.com/v1/projects/{PROJECT_ID}/locations/{REGION}/endpoints/{ENDPOINT_ID}:rawPredict" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "TpV-iwP9qw9c" + }, + "source": [ + "## Cleaning up\n", + "\n", + "To clean up all Google Cloud resources used in this project, you can [delete the Google Cloud\n", + "project](https://cloud.google.com/resource-manager/docs/creating-managing-projects#shutting_down_projects) you used for the tutorial.\n", + "\n", + "Otherwise, you can delete the individual resources you created in this tutorial:" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "sx_vKniMq9ZX" + }, + "outputs": [], + "source": [ + "# Undeploy model and delete endpoint\n", + "endpoint.delete(force=True)\n", + "\n", + "# Delete the model resource\n", + "model.delete()\n", + "\n", + "# Delete the container image from Artifact Registry\n", + "!gcloud artifacts docker images delete \\\n", + " --quiet \\\n", + " --delete-tags \\\n", + " {REGION}-docker.pkg.dev/{PROJECT_ID}/{REPOSITORY}/{IMAGE}\n", + "\n", + "delete_bucket = False\n", + "\n", + "if delete_bucket or os.getenv(\"IS_TESTING\"):\n", + " ! gsutil rm -r $BUCKET_URI" + ] + } + ], + "metadata": { + "colab": { + "name": "SDK_Triton_PyTorch_Local_Prediction.ipynb", + "toc_visible": true + }, + "kernelspec": { + "display_name": "Python 3", + "name": "python3" + } + }, + "nbformat": 4, + "nbformat_minor": 0 +}