diff --git a/notebooks/official/explainable_ai/gapic-custom_image_classification_batch_explain.ipynb b/notebooks/official/explainable_ai/gapic-custom_image_classification_batch_explain.ipynb new file mode 100644 index 000000000..52e09a35e --- /dev/null +++ b/notebooks/official/explainable_ai/gapic-custom_image_classification_batch_explain.ipynb @@ -0,0 +1,2594 @@ +{ + "cells": [ + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "copyright", + "repo": "snippets_housekeeping.ipynb" + }, + "outputs": [], + "source": [ + "# Copyright 2020 Google LLC\n", + "#\n", + "# Licensed under the Apache License, Version 2.0 (the \"License\");\n", + "# you may not use this file except in compliance with the License.\n", + "# You may obtain a copy of the License at\n", + "#\n", + "# https://www.apache.org/licenses/LICENSE-2.0\n", + "#\n", + "# Unless required by applicable law or agreed to in writing, software\n", + "# distributed under the License is distributed on an \"AS IS\" BASIS,\n", + "# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n", + "# See the License for the specific language governing permissions and\n", + "# limitations under the License." + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "title", + "repo": "snippets_housekeeping.ipynb" + }, + "source": [ + "# Vertex client library for Python: Custom training image classification model for batch prediction with explanation\n", + "\n", + "\n", + " \n", + " \n", + "
\n", + " \n", + " \"Colab Run in Colab\n", + " \n", + " \n", + " \n", + " \"GitHub\n", + " View on GitHub\n", + " \n", + "
\n", + "


" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "overview:custom,xai", + "repo": "snippets_housekeeping.ipynb" + }, + "source": [ + "## Overview\n", + "\n", + "\n", + "This tutorial demonstrates how to use the Vertex client library for Python to train and deploy a custom image classification model for batch prediction with explanation." + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "dataset:custom,cifar10,icn", + "repo": "snippets_housekeeping.ipynb" + }, + "source": [ + "### Dataset\n", + "\n", + "The dataset used for this tutorial is the [CIFAR10 dataset](https://www.tensorflow.org/datasets/catalog/cifar10) from [TensorFlow Datasets](https://www.tensorflow.org/datasets/catalog/overview). The version of the dataset you will use is built into TensorFlow. The trained model predicts which type of class an image is from ten classes: airplane, automobile, bird, cat, deer, dog, frog, horse, ship, truck." + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "objective:custom,training,batch_prediction,xai", + "repo": "snippets_housekeeping.ipynb" + }, + "source": [ + "### Objective\n", + "\n", + "In this tutorial, you create a custom trained model, with a training pipeline, from a Python script in a Google prebuilt Docker container using the Vertex client library for Python, and then do a batch prediction with explanations on the uploaded model. Alternatively, you can create custom trained models using `gcloud` command-line tool or online using Cloud Console.\n", + "\n", + "The steps performed include:\n", + "\n", + "- Create a Vertex custom job for training a model.\n", + "- Train the TensorFlow model.\n", + "- Retrieve and load the model artifacts.\n", + "- View the model evaluation.\n", + "- Set explanation parameters.\n", + "- Upload the model as a Vertex `Model` resource.\n", + "- Make a batch prediction with explanations." + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "costs", + "repo": "snippets_housekeeping.ipynb" + }, + "source": [ + "### Costs\n", + "\n", + "This tutorial uses billable components of Google Cloud (GCP):\n", + "\n", + "* Vertex AI\n", + "* Cloud Storage\n", + "\n", + "Learn about [Vertex AI\n", + "pricing](https://cloud.google.com/vertex-ai/pricing) and [Cloud Storage\n", + "pricing](https://cloud.google.com/storage/pricing), and use the [Pricing\n", + "Calculator](https://cloud.google.com/products/calculator/)\n", + "to generate a cost estimate based on your projected usage." + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "install_aip", + "repo": "snippets_housekeeping.ipynb" + }, + "source": [ + "## Installation\n", + "\n", + "Install the latest version of Vertex client library for Python." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "install_aip", + "repo": "snippets_housekeeping.ipynb" + }, + "outputs": [], + "source": [ + "import sys\n", + "import os\n", + "\n", + "\n", + "# Google Cloud Notebook\n", + "if os.path.exists(\"/opt/deeplearning/metadata/env_version\"):\n", + " USER_FLAG = '--user'\n", + "else:\n", + " USER_FLAG = ''\n", + "\n", + "! pip3 install --upgrade google-cloud-aiplatform $USER_FLAG" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "install_storage", + "repo": "snippets_housekeeping.ipynb" + }, + "source": [ + "Install the latest GA version of *google-cloud-storage* library as well." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "install_storage", + "repo": "snippets_housekeeping.ipynb" + }, + "outputs": [], + "source": [ + "! pip3 install -U google-cloud-storage $USER_FLAG" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "install_tensorflow", + "repo": "snippets_housekeeping.ipynb" + }, + "outputs": [], + "source": [ + "if os.environ[\"IS_TESTING\"]:\n", + " ! pip3 install --upgrade tensorflow $USER_FLAG" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "install_cv2", + "repo": "snippets_housekeeping.ipynb" + }, + "outputs": [], + "source": [ + "if os.environ[\"IS_TESTING\"]:\n", + " ! apt-get update && apt-get install -y python3-opencv\n", + " ! pip3 install --upgrade opencv-python $USER_FLAG" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "restart", + "repo": "snippets_housekeeping.ipynb" + }, + "source": [ + "### Restart the kernel\n", + "\n", + "Once you've installed the Vertex client library for Python and *google-cloud-storage* library, you need to restart the notebook kernel so it can find the packages." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "restart", + "repo": "snippets_housekeeping.ipynb" + }, + "outputs": [], + "source": [ + "if not os.getenv(\"IS_TESTING\"):\n", + " # Automatically restart kernel after installs\n", + " import IPython\n", + " app = IPython.Application.instance()\n", + " app.kernel.do_shutdown(True)" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "before_you_begin", + "repo": "snippets_housekeeping.ipynb" + }, + "source": [ + "## Before you begin\n", + "\n", + "### GPU runtime\n", + "\n", + "*Make sure you're running this notebook in a GPU runtime if you have that option. In Colab, select* **Runtime > Change Runtime Type > GPU**\n", + "\n", + "### Set up your Google Cloud project\n", + "\n", + "**The following steps are required, regardless of your notebook environment.**\n", + "\n", + "1. [Select or create a Google Cloud project](https://console.cloud.google.com/cloud-resource-manager). When you first create an account, you get a $300 free credit towards your compute/storage costs.\n", + "\n", + "2. [Make sure that billing is enabled for your project.](https://cloud.google.com/billing/docs/how-to/modify-project)\n", + "\n", + "3. [Enable the Vertex AI APIs and Compute Engine APIs.](https://console.cloud.google.com/flows/enableapi?apiid=ml.googleapis.com,compute_component)\n", + "\n", + "4. [The Cloud SDK](https://cloud.google.com/sdk) is already installed in Google Cloud Notebook.\n", + "\n", + "5. Enter your project ID in the cell below. Then run the cell to make sure the\n", + "Cloud SDK uses the right project for all the commands in this notebook.\n", + "\n", + "**Note**: Jupyter runs lines prefixed with `!` as shell commands, and it interpolates Python variables prefixed with `$`." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "set_project_id", + "repo": "snippets_housekeeping.ipynb" + }, + "outputs": [], + "source": [ + "PROJECT_ID = \"[your-project-id]\" #@param {type:\"string\"}" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "autoset_project_id", + "repo": "snippets_housekeeping.ipynb" + }, + "outputs": [], + "source": [ + "if PROJECT_ID == \"\" or PROJECT_ID is None or PROJECT_ID == \"[your-project-id]\":\n", + " # Get your GCP project id from gcloud\n", + " shell_output = !gcloud config list --format 'value(core.project)' 2>/dev/null\n", + " PROJECT_ID = shell_output[0]\n", + " print(\"Project ID:\", PROJECT_ID)" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "set_gcloud_project_id", + "repo": "snippets_housekeeping.ipynb" + }, + "outputs": [], + "source": [ + "! gcloud config set project $PROJECT_ID" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "region", + "repo": "snippets_housekeeping.ipynb" + }, + "source": [ + "#### Region\n", + "\n", + "You can also change the `REGION` variable, which is used for operations\n", + "throughout the rest of this notebook. Below are regions supported for Vertex AI. We recommend that you choose the region closest to you.\n", + "\n", + "- Americas: `us-central1`\n", + "- Europe: `europe-west4`\n", + "- Asia Pacific: `asia-east1`\n", + "\n", + "You may not use a multi-regional bucket for training with Vertex AI. Not all regions provide support for all Vertex AI services.\n", + "\n", + "Learn more about [Vertex AI regions](https://cloud.google.com/vertex-ai/docs/general/locations)" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "region", + "repo": "snippets_housekeeping.ipynb" + }, + "outputs": [], + "source": [ + "REGION = 'us-central1' #@param {type: \"string\"}" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "timestamp", + "repo": "snippets_housekeeping.ipynb" + }, + "source": [ + "#### Timestamp\n", + "\n", + "If you are in a live tutorial session, you might be using a shared test account or project. To avoid name collisions between users on resources created, you create a timestamp for each instance session, and append the timestamp onto the name of resources you create in this tutorial." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "timestamp", + "repo": "snippets_housekeeping.ipynb" + }, + "outputs": [], + "source": [ + "from datetime import datetime\n", + "\n", + "TIMESTAMP = datetime.now().strftime(\"%Y%m%d%H%M%S\")" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "gcp_authenticate", + "repo": "snippets_housekeeping.ipynb" + }, + "source": [ + "### Authenticate your Google Cloud account\n", + "\n", + "**If you are using Google Cloud Notebook**, your environment is already authenticated. Skip this step.\n", + "\n", + "**If you are using Colab**, run the cell below and follow the instructions when prompted to authenticate your account via oAuth.\n", + "\n", + "**Otherwise**, follow these steps:\n", + "\n", + "In the Cloud Console, go to the [Create service account key](https://console.cloud.google.com/apis/credentials/serviceaccountkey) page.\n", + "\n", + "**Click Create service account**.\n", + "\n", + "In the **Service account name** field, enter a name, and click **Create**.\n", + "\n", + "In the **Grant this service account access to project** section, click the Role drop-down list. Type \"Vertex\" into the filter box, and select **Vertex AI Administrator**. Type \"Storage Object Admin\" into the filter box, and select **Storage Object Admin**.\n", + "\n", + "Click Create. A JSON file that contains your key downloads to your local environment.\n", + "\n", + "Enter the path to your service account key as the `GOOGLE_APPLICATION_CREDENTIALS` variable in the cell below and run the cell." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "gcp_authenticate", + "repo": "snippets_housekeeping.ipynb" + }, + "outputs": [], + "source": [ + "# If you are running this notebook in Colab, run this cell and follow the\n", + "# instructions to authenticate your Google Cloud account. This provides access to your\n", + "# Cloud Storage bucket and lets you submit training jobs and prediction\n", + "# requests.\n", + "\n", + "# If on Google Cloud Notebook, then don't execute this code\n", + "if not os.path.exists(\"/opt/deeplearning/metadata/env_version\"):\n", + " if \"google.colab\" in sys.modules:\n", + " from google.colab import auth as google_auth\n", + "\n", + " google_auth.authenticate_user()\n", + "\n", + " # If you are running this notebook locally, replace the string below with the\n", + " # path to your service account key and run this cell to authenticate your Google Cloud\n", + " # account.\n", + " elif not os.getenv(\"IS_TESTING\"):\n", + " %env GOOGLE_APPLICATION_CREDENTIALS ''" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "bucket:custom", + "repo": "snippets_housekeeping.ipynb" + }, + "source": [ + "### Create a Cloud Storage bucket\n", + "\n", + "**The following steps are required, regardless of your notebook environment.**\n", + "\n", + "When you submit a custom training job using the Vertex client library for Python, you upload a Python package\n", + "containing your training code to a Cloud Storage bucket. Vertex runs\n", + "the code from this package. In this tutorial, Vertex also saves the\n", + "trained model that results from your job in the same bucket. You can then\n", + "create an `Endpoint` resource based on this output in order to serve\n", + "online predictions.\n", + "\n", + "Set the name of your Cloud Storage bucket below. Bucket names must be globally unique across all Google Cloud projects, including those outside of your organization." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "bucket", + "repo": "snippets_housekeeping.ipynb" + }, + "outputs": [], + "source": [ + "BUCKET_NAME = \"gs://[your-bucket-name]\" #@param {type:\"string\"}" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "autoset_bucket", + "repo": "snippets_housekeeping.ipynb" + }, + "outputs": [], + "source": [ + "if BUCKET_NAME == \"\" or BUCKET_NAME is None or BUCKET_NAME == \"gs://[your-bucket-name]\":\n", + " BUCKET_NAME = \"gs://\" + PROJECT_ID + \"aip-\" + TIMESTAMP" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "create_bucket", + "repo": "snippets_housekeeping.ipynb" + }, + "source": [ + "**Only if your bucket doesn't already exist**: Run the following cell to create your Cloud Storage bucket." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "create_bucket", + "repo": "snippets_housekeeping.ipynb" + }, + "outputs": [], + "source": [ + "! gsutil mb -l $REGION $BUCKET_NAME" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "validate_bucket", + "repo": "snippets_housekeeping.ipynb" + }, + "source": [ + "Finally, validate access to your Cloud Storage bucket by examining its contents:" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "validate_bucket", + "repo": "snippets_housekeeping.ipynb" + }, + "outputs": [], + "source": [ + "! gsutil ls -al $BUCKET_NAME" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "setup_vars", + "repo": "snippets_housekeeping.ipynb" + }, + "source": [ + "### Set up variables\n", + "\n", + "Next, set up some variables used throughout the tutorial.\n", + "### Import libraries and define constants" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "import_aip:v1beta1,protobuf", + "repo": "snippets_housekeeping.ipynb" + }, + "source": [ + "#### Import Vertex client library for Python\n", + "\n", + "Import the Vertex client library for Python into your Python environment." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "import_aip:v1beta1,protobuf", + "repo": "snippets_housekeeping.ipynb" + }, + "outputs": [], + "source": [ + "import time\n", + "\n", + "import google.cloud.aiplatform_v1beta1 as aip\n", + "\n", + "from google.protobuf import json_format\n", + "from google.protobuf.struct_pb2 import Value\n", + "from google.protobuf.struct_pb2 import Struct\n", + "from google.protobuf.json_format import MessageToJson\n", + "from google.protobuf.json_format import ParseDict" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "aip_constants", + "repo": "snippets_housekeeping.ipynb" + }, + "source": [ + "#### Vertex AI constants\n", + "\n", + "Setup up the following constants for Vertex AI:\n", + "\n", + "- `API_ENDPOINT`: The Vertex AI service endpoint for dataset, model, job, pipeline and endpoint services.\n", + "- `PARENT`: The Vertex AI location root path for `Dataset`, `Model`, `Job`, `Pipeline` and `Endpoint` resources." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "aip_constants", + "repo": "snippets_housekeeping.ipynb" + }, + "outputs": [], + "source": [ + "# API service endpoint\n", + "API_ENDPOINT = \"{0}-aiplatform.googleapis.com\".format(REGION)\n", + "\n", + "# Vertex AI location root path for your dataset, model and endpoint resources\n", + "PARENT = \"projects/\" + PROJECT_ID + \"/locations/\" + REGION" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "accelerators:training,prediction,cpu", + "repo": "snippets_housekeeping.ipynb" + }, + "source": [ + "#### Set hardware accelerators\n", + "\n", + "You can set hardware accelerators for training and prediction.\n", + "\n", + "Set the variables `TRAIN_GPU/TRAIN_NGPU` and `DEPLOY_GPU/DEPLOY_NGPU` to use a container image supporting a GPU and the number of GPUs allocated to the virtual machine (VM) instance. For example, to use a GPU container image with 4 Nvidia Telsa K80 GPUs allocated to each VM, you would specify:\n", + "\n", + " (aip.AcceleratorType.NVIDIA_TESLA_K80, 4)\n", + "\n", + "\n", + "Otherwise specify `(None, None)` to use a container image to run on a CPU.\n", + "\n", + "Learn [which accelerators are available in your region](https://cloud.google.com/vertex-ai/docs/general/locations#accelerators).\n", + "\n", + "*Note*: TF releases before 2.3 for GPU support will fail to load the custom trained model in this tutorial. It is a known issue and fixed in TF 2.3 -- which is caused by static graph ops that are generated in the serving function. If you encounter this issue on your own custom trained models, use a container image for TF 2.3 with GPU support." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "accelerators:training,prediction,cpu", + "repo": "snippets_housekeeping.ipynb" + }, + "outputs": [], + "source": [ + "if os.getenv(\"IS_TESTING_TRAIN_GPU\"):\n", + " TRAIN_GPU, TRAIN_NGPU = (aip.AcceleratorType.NVIDIA_TESLA_K80, int(os.getenv(\"IS_TESTING_TRAIN_GPU\")))\n", + "else:\n", + " TRAIN_GPU, TRAIN_NGPU = (aip.AcceleratorType.NVIDIA_TESLA_K80, 1)\n", + "\n", + "if os.getenv(\"IS_TESTING_DEPLOY_GPU\"):\n", + " DEPLOY_GPU, DEPLOY_NGPU = (aip.AcceleratorType.NVIDIA_TESLA_K80, int(os.getenv(\"IS_TESTING_DEPLOY_GPU\")))\n", + "else:\n", + " DEPLOY_GPU, DEPLOY_NGPU = (None, None)" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "container:training,prediction", + "repo": "snippets_housekeeping.ipynb" + }, + "source": [ + "#### Set pre-built containers\n", + "\n", + "Set the pre-built Docker container image for training and prediction.\n", + "\n", + "\n", + "For the latest list, see [Pre-built containers for training](https://cloud.google.com/ai-platform-unified/docs/training/pre-built-containers).\n", + "\n", + "\n", + "For the latest list, see [Pre-built containers for prediction](https://cloud.google.com/ai-platform-unified/docs/predictions/pre-built-containers)" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "container:training,prediction", + "repo": "snippets_housekeeping.ipynb" + }, + "outputs": [], + "source": [ + "if os.getenv(\"IS_TESTING_TF\"):\n", + " TF = os.getenv(\"IS_TESTING_TF\")\n", + "else:\n", + " TF = '2-1'\n", + "\n", + "if TF[0] == '2':\n", + " if TRAIN_GPU:\n", + " TRAIN_VERSION = 'tf-gpu.{}'.format(TF)\n", + " else:\n", + " TRAIN_VERSION = 'tf-cpu.{}'.format(TF)\n", + " if DEPLOY_GPU:\n", + " DEPLOY_VERSION = 'tf2-gpu.{}'.format(TF)\n", + " else:\n", + " DEPLOY_VERSION = 'tf2-cpu.{}'.format(TF)\n", + "else:\n", + " if TRAIN_GPU:\n", + " TRAIN_VERSION = 'tf-gpu.{}'.format(TF)\n", + " else:\n", + " TRAIN_VERSION = 'tf-cpu.{}'.format(TF)\n", + " if DEPLOY_GPU:\n", + " DEPLOY_VERSION = 'tf-gpu.{}'.format(TF)\n", + " else:\n", + " DEPLOY_VERSION = 'tf-cpu.{}'.format(TF)\n", + "\n", + "TRAIN_IMAGE = \"gcr.io/cloud-aiplatform/training/{}:latest\".format(TRAIN_VERSION)\n", + "DEPLOY_IMAGE = \"gcr.io/cloud-aiplatform/prediction/{}:latest\".format(DEPLOY_VERSION)\n", + "\n", + "print(\"Training:\", TRAIN_IMAGE, TRAIN_GPU, TRAIN_NGPU)\n", + "print(\"Deployment:\", DEPLOY_IMAGE, DEPLOY_GPU, DEPLOY_NGPU)" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "machine:training,prediction", + "repo": "snippets_housekeeping.ipynb" + }, + "source": [ + "#### Set machine type\n", + "\n", + "Next, set the machine type to use for training and prediction.\n", + "\n", + "- Set the variables `TRAIN_COMPUTE` and `DEPLOY_COMPUTE` to configure the compute resources for the VMs you will use for for training and prediction.\n", + " - `machine type`\n", + " - `n1-standard`: 3.75GB of memory per vCPU.\n", + " - `n1-highmem`: 6.5GB of memory per vCPU\n", + " - `n1-highcpu`: 0.9 GB of memory per vCPU\n", + " - `vCPUs`: number of \\[2, 4, 8, 16, 32, 64, 96 \\]\n", + "\n", + "*Note: The following is not supported for training:*\n", + "\n", + " - `standard`: 2 vCPUs\n", + " - `highcpu`: 2, 4 and 8 vCPUs\n", + "\n", + "*Note: You may also use n2 and e2 machine types for training and deployment, but they do not support GPUs*.\n", + "\n", + "Learn [which machine types are available for training](https://cloud.google.com/vertex-ai/docs/training/configure-compute) and [for prediction](https://cloud.google.com/vertex-ai/docs/predictions/configure-compute)" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "machine:training,prediction", + "repo": "snippets_housekeeping.ipynb" + }, + "outputs": [], + "source": [ + "if os.getenv(\"IS_TESTING_TRAIN_MACHINE\"):\n", + " MACHINE_TYPE = os.getenv(\"IS_TESTING_TRAIN_MACHINE\")\n", + "else:\n", + " MACHINE_TYPE = 'n1-standard'\n", + "\n", + "VCPU = '4'\n", + "TRAIN_COMPUTE = MACHINE_TYPE + '-' + VCPU\n", + "print('Train machine type', TRAIN_COMPUTE)\n", + "\n", + "if os.getenv(\"IS_TESTING_DEPLOY_MACHINE\"):\n", + " MACHINE_TYPE = os.getenv(\"IS_TESTING_DEPLOY_MACHINE\")\n", + "else:\n", + " MACHINE_TYPE = 'n1-standard'\n", + "\n", + "VCPU = '4'\n", + "DEPLOY_COMPUTE = MACHINE_TYPE + '-' + VCPU\n", + "print('Deploy machine type', DEPLOY_COMPUTE)" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "tutorial_start:custom", + "repo": "snippets_custom.ipynb" + }, + "source": [ + "# Tutorial\n", + "\n", + "Now you are ready to start creating your own custom model and training for CIFAR10." + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "clients:custom", + "repo": "snippets_housekeeping.ipynb" + }, + "source": [ + "## Set up clients\n", + "\n", + "The Vertex client library for Python works as a client/server model. On your side (the Python script) you will create a client that sends requests and receives responses from the Vertex AI server.\n", + "\n", + "You will use different clients in this tutorial for different steps in the workflow. So set them all up upfront.\n", + "\n", + "- Model Service for `Model` resources.\n", + "- Endpoint Service for deployment.\n", + "- Job Service for batch jobs and custom training.\n", + "- Prediction Service for serving." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "clients:custom", + "repo": "snippets_housekeeping.ipynb" + }, + "outputs": [], + "source": [ + "# client options same for all services\n", + "client_options = {\"api_endpoint\": API_ENDPOINT}\n", + "\n", + "\n", + "def create_job_client():\n", + " client = aip.JobServiceClient(\n", + " client_options=client_options\n", + " )\n", + " return client\n", + "\n", + "\n", + "def create_model_client():\n", + " client = aip.ModelServiceClient(\n", + " client_options=client_options\n", + " )\n", + " return client\n", + "\n", + "\n", + "def create_endpoint_client():\n", + " client = aip.EndpointServiceClient(\n", + " client_options=client_options\n", + " )\n", + " return client\n", + "\n", + "\n", + "def create_prediction_client():\n", + " client = aip.PredictionServiceClient(\n", + " client_options=client_options\n", + " )\n", + " return client\n", + "\n", + "\n", + "clients = {}\n", + "clients['job'] = create_job_client()\n", + "clients['model'] = create_model_client()\n", + "clients['endpoint'] = create_endpoint_client()\n", + "clients['prediction'] = create_prediction_client()\n", + "\n", + "for client in clients.items():\n", + " print(client)" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "train_custom_model", + "repo": "snippets_custom.ipynb" + }, + "source": [ + "## Train a model\n", + "\n", + "There are two ways you can train a custom model using a container image:\n", + "\n", + "- **Use a Google Cloud prebuilt container**. If you use a prebuilt container, you will additionally specify a Python package to install into the container image. This Python package contains your code for training a custom model.\n", + "\n", + "- **Use your own custom container image**. If you use your own container, the container needs to contain your code for training a custom model." + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "train_custom_job_specification:prebuilt_container", + "repo": "snippets_custom.ipynb" + }, + "source": [ + "## Prepare your custom job specification\n", + "\n", + "Now that your clients are ready, your first step is to create a Job Specification for your custom training job. The job specification will consist of the following:\n", + "\n", + "- `worker_pool_spec` : The specification of the type of machine(s) you will use for training and how many (single or distributed)\n", + "- `python_package_spec` : The specification of the Python package to be installed with the pre-built container." + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "train_custom_job_machine_specification", + "repo": "snippets_custom.ipynb" + }, + "source": [ + "### Prepare your machine specification\n", + "\n", + "Now define the machine specification for your custom training job. This tells Vertex AI what type of machine instance to provision for the training.\n", + " - `machine_type`: The type of Google Cloud instance to provision -- e.g., n1-standard-8.\n", + " - `accelerator_type`: The type, if any, of hardware accelerator. In this tutorial if you previously set the variable `TRAIN_GPU != None`, you are using a GPU; otherwise you will use a CPU.\n", + " - `accelerator_count`: The number of accelerators." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "train_custom_job_machine_specification", + "repo": "snippets_custom.ipynb" + }, + "outputs": [], + "source": [ + "if TRAIN_GPU:\n", + " machine_spec = {\n", + " \"machine_type\": TRAIN_COMPUTE,\n", + " \"accelerator_type\": TRAIN_GPU,\n", + " \"accelerator_count\": TRAIN_NGPU\n", + " }\n", + "else:\n", + " machine_spec = {\n", + " \"machine_type\": TRAIN_COMPUTE,\n", + " \"accelerator_count\": 0\n", + " }" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "train_custom_job_disk_specification", + "repo": "snippets_custom.ipynb" + }, + "source": [ + "### Prepare your disk specification\n", + "\n", + "(optional) Now define the disk specification for your custom training job. This tells Vertex AI what type and size of disk to provision in each machine instance for the training.\n", + "\n", + " - `boot_disk_type`: Either SSD or Standard. SSD is faster, and Standard is less expensive. Defaults to SSD.\n", + " - `boot_disk_size_gb`: Size of disk in GB." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "train_custom_job_disk_specification", + "repo": "snippets_custom.ipynb" + }, + "outputs": [], + "source": [ + "DISK_TYPE = \"pd-ssd\" # [ pd-ssd, pd-standard]\n", + "DISK_SIZE = 200 # GB\n", + "\n", + "disk_spec = {\n", + " \"boot_disk_type\": DISK_TYPE,\n", + " \"boot_disk_size_gb\": DISK_SIZE\n", + "}" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "train_custom_job_worker_pool_specification:prebuilt_container", + "repo": "snippets_custom.ipynb" + }, + "source": [ + "### Define the worker pool specification\n", + "\n", + "Next, you define the worker pool specification for your custom training job. The worker pool specification will consist of the following:\n", + "\n", + "- `replica_count`: The number of instances to provision of this machine type.\n", + "- `machine_spec`: The hardware specification.\n", + "- `disk_spec` : (optional) The disk storage specification.\n", + "\n", + "- `python_package`: The Python training package to install on the VM instance(s) and which Python module to invoke, along with command line arguments for the Python module.\n", + "\n", + "Let's dive deeper now into the python package specification:\n", + "\n", + "-`executor_image_spec`: This is the docker image which is configured for your custom training job.\n", + "\n", + "-`package_uris`: This is a list of the locations (URIs) of your python training packages to install on the provisioned instance. The locations need to be in a Cloud Storage bucket. These can be either individual python files or a zip (archive) of an entire package. In the later case, the job service will unzip (unarchive) the contents into the docker image.\n", + "\n", + "-`python_module`: The Python module (script) to invoke for running the custom training job. In this example, you will be invoking `trainer.task.py` -- note that it was not neccessary to append the `.py` suffix.\n", + "\n", + "-`args`: The command line arguments to pass to the corresponding Pythom module. In this example, you will be setting:\n", + " - `\"--model-dir=\" + MODEL_DIR` : The Cloud Storage location where to store the model artifacts. There are two ways to tell the training script where to save the model artifacts:\n", + " - direct: You pass the Cloud Storage location as a command line argument to your training script (set variable `DIRECT = True`), or\n", + " - indirect: The service passes the Cloud Storage location as the environment variable `AIP_MODEL_DIR` to your training script (set variable `DIRECT = False`). In this case, you tell the service the model artifact location in the job specification.\n", + " - `\"--epochs=\" + EPOCHS`: The number of epochs for training.\n", + " - `\"--steps=\" + STEPS`: The number of steps (batches) per epoch.\n", + " - `\"--distribute=\" + TRAIN_STRATEGY\"` : The training distribution strategy to use for single or distributed training.\n", + " - `\"single\"`: single device.\n", + " - `\"mirror\"`: all GPU devices on a single compute instance.\n", + " - `\"multi\"`: all GPU devices on all compute instances." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "train_custom_job_worker_pool_specification:prebuilt_container", + "repo": "snippets_custom.ipynb" + }, + "outputs": [], + "source": [ + "JOB_NAME = \"custom_job_\" + TIMESTAMP\n", + "MODEL_DIR = '{}/{}'.format(BUCKET_NAME, JOB_NAME)\n", + "\n", + "if not TRAIN_NGPU or TRAIN_NGPU < 2:\n", + " TRAIN_STRATEGY = \"single\"\n", + "else:\n", + " TRAIN_STRATEGY = \"mirror\"\n", + "\n", + "EPOCHS = 20\n", + "STEPS = 100\n", + "\n", + "DIRECT = True\n", + "if DIRECT:\n", + " CMDARGS = [\n", + " \"--model-dir=\" + MODEL_DIR,\n", + " \"--epochs=\" + str(EPOCHS),\n", + " \"--steps=\" + str(STEPS),\n", + " \"--distribute=\" + TRAIN_STRATEGY\n", + " ]\n", + "else:\n", + " CMDARGS = [\n", + " \"--epochs=\" + str(EPOCHS),\n", + " \"--steps=\" + str(STEPS),\n", + " \"--distribute=\" + TRAIN_STRATEGY\n", + " ]\n", + "\n", + "worker_pool_spec = [\n", + " {\n", + " \"replica_count\": 1,\n", + " \"machine_spec\": machine_spec,\n", + " \"disk_spec\": disk_spec,\n", + " \"python_package_spec\": {\n", + " \"executor_image_uri\": TRAIN_IMAGE,\n", + " \"package_uris\": [BUCKET_NAME + \"/trainer_cifar10.tar.gz\"],\n", + " \"python_module\": \"trainer.task\",\n", + " \"args\": CMDARGS,\n", + " }\n", + " }\n", + "]" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "assemble_custom_job_specification", + "repo": "snippets_custom.ipynb" + }, + "source": [ + "### Assemble a job specification\n", + "\n", + "Now assemble the complete description for the custom job specification:\n", + "\n", + "- `display_name`: The human readable name you assign to this custom job.\n", + "- `job_spec`: The specification for the custom job.\n", + " - `worker_pool_specs`: The specification for the machine VM instances.\n", + " - `base_output_directory`: This tells the service the Cloud Storage location where to save the model artifacts (when variable `DIRECT = False`). The service will then pass the location to the training script as the environment variable `AIP_MODEL_DIR`, and the path will be of the form:\n", + "\n", + " /model" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "assemble_custom_job_specification", + "repo": "snippets_custom.ipynb" + }, + "outputs": [], + "source": [ + "if DIRECT:\n", + " job_spec = {\n", + " \"worker_pool_specs\": worker_pool_spec\n", + " }\n", + "else:\n", + " job_spec = {\n", + " \"worker_pool_specs\": worker_pool_spec,\n", + " \"base_output_directory\": {\"output_uri_prefix\": MODEL_DIR}\n", + " }\n", + "\n", + "custom_job = {\n", + " \"display_name\": JOB_NAME,\n", + " \"job_spec\": job_spec\n", + "}" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "examine_training_package", + "repo": "snippets_custom.ipynb" + }, + "source": [ + "### Examine the training package\n", + "\n", + "#### Package layout\n", + "\n", + "Before you start the training, you will look at how a Python package is assembled for a custom training job. When unarchived, the package contains the following directory/file layout.\n", + "\n", + "- PKG-INFO\n", + "- README.md\n", + "- setup.cfg\n", + "- setup.py\n", + "- trainer\n", + " - \\_\\_init\\_\\_.py\n", + " - task.py\n", + "\n", + "The files `setup.cfg` and `setup.py` are the instructions for installing the package into the operating environment of the Docker image.\n", + "\n", + "The file `trainer/task.py` is the Python script for executing the custom training job. *Note*, when we referred to it in the worker pool specification, we replace the directory slash with a dot (`trainer.task`) and dropped the file suffix (`.py`).\n", + "\n", + "#### Package Assembly\n", + "\n", + "In the following cells, you will assemble the training package." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "examine_training_package", + "repo": "snippets_custom.ipynb" + }, + "outputs": [], + "source": [ + "# Make folder for Python training script\n", + "! rm -rf custom\n", + "! mkdir custom\n", + "\n", + "# Add package information\n", + "! touch custom/README.md\n", + "\n", + "setup_cfg = \"[egg_info]\\n\\ntag_build =\\n\\ntag_date = 0\"\n", + "! echo \"$setup_cfg\" > custom/setup.cfg\n", + "\n", + "setup_py = \"import setuptools\\n\\nsetuptools.setup(\\n\\n install_requires=[\\n\\n 'tensorflow_datasets==1.3.0',\\n\\n ],\\n\\n packages=setuptools.find_packages())\"\n", + "! echo \"$setup_py\" > custom/setup.py\n", + "\n", + "pkg_info = \"Metadata-Version: 1.0\\n\\nName: CIFAR10 image classification\\n\\nVersion: 0.0.0\\n\\nSummary: Demostration training script\\n\\nHome-page: www.google.com\\n\\nAuthor: Google\\n\\nAuthor-email: aferlitsch@google.com\\n\\nLicense: Public\\n\\nDescription: Demo\\n\\nPlatform: Vertex\"\n", + "! echo \"$pkg_info\" > custom/PKG-INFO\n", + "\n", + "# Make the training subfolder\n", + "! mkdir custom/trainer\n", + "! touch custom/trainer/__init__.py" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "taskpy_contents:cifar10", + "repo": "snippets_custom.ipynb" + }, + "source": [ + "#### Task.py contents\n", + "\n", + "In the next cell, you write the contents of the training script task.py. We won't go into detail, it's just there for you to browse. In summary:\n", + "\n", + "- Get the directory where to save the model artifacts from the command line (`--model_dir`), and if not specified, then from the environment variable `AIP_MODEL_DIR`.\n", + "- Loads CIFAR10 dataset from TF Datasets (tfds).\n", + "- Builds a model using TF.Keras model API.\n", + "- Compiles the model (`compile()`).\n", + "- Sets a training distribution strategy according to the argument `args.distribute`.\n", + "- Trains the model (`fit()`) with epochs and steps according to the arguments `args.epochs` and `args.steps`\n", + "- Saves the trained model (`save(args.model_dir)`) to the specified model directory." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "taskpy_contents:cifar10", + "repo": "snippets_custom.ipynb" + }, + "outputs": [], + "source": [ + "%%writefile custom/trainer/task.py\n", + "# Single, Mirror and Multi-Machine Distributed Training for CIFAR-10\n", + "\n", + "import tensorflow_datasets as tfds\n", + "import tensorflow as tf\n", + "from tensorflow.python.client import device_lib\n", + "import argparse\n", + "import os\n", + "import sys\n", + "tfds.disable_progress_bar()\n", + "\n", + "parser = argparse.ArgumentParser()\n", + "parser.add_argument('--model-dir', dest='model_dir',\n", + " default=os.getenv(\"AIP_MODEL_DIR\"), type=str, help='Model dir.')\n", + "parser.add_argument('--lr', dest='lr',\n", + " default=0.01, type=float,\n", + " help='Learning rate.')\n", + "parser.add_argument('--epochs', dest='epochs',\n", + " default=10, type=int,\n", + " help='Number of epochs.')\n", + "parser.add_argument('--steps', dest='steps',\n", + " default=200, type=int,\n", + " help='Number of steps per epoch.')\n", + "parser.add_argument('--distribute', dest='distribute', type=str, default='single',\n", + " help='distributed training strategy')\n", + "args = parser.parse_args()\n", + "\n", + "print('Python Version = {}'.format(sys.version))\n", + "print('TensorFlow Version = {}'.format(tf.__version__))\n", + "print('TF_CONFIG = {}'.format(os.environ.get('TF_CONFIG', 'Not found')))\n", + "print('DEVICES', device_lib.list_local_devices())\n", + "\n", + "# Single Machine, single compute device\n", + "if args.distribute == 'single':\n", + " if tf.test.is_gpu_available():\n", + " strategy = tf.distribute.OneDeviceStrategy(device=\"/gpu:0\")\n", + " else:\n", + " strategy = tf.distribute.OneDeviceStrategy(device=\"/cpu:0\")\n", + "# Single Machine, multiple compute device\n", + "elif args.distribute == 'mirror':\n", + " strategy = tf.distribute.MirroredStrategy()\n", + "# Multiple Machine, multiple compute device\n", + "elif args.distribute == 'multi':\n", + " strategy = tf.distribute.experimental.MultiWorkerMirroredStrategy()\n", + "\n", + "# Multi-worker configuration\n", + "print('num_replicas_in_sync = {}'.format(strategy.num_replicas_in_sync))\n", + "\n", + "# Preparing dataset\n", + "BUFFER_SIZE = 10000\n", + "BATCH_SIZE = 64\n", + "\n", + "\n", + "def make_datasets_unbatched():\n", + "\n", + " # Scaling CIFAR10 data from (0, 255] to (0., 1.]\n", + " def scale(image, label):\n", + " image = tf.cast(image, tf.float32)\n", + " image /= 255.0\n", + " return image, label\n", + "\n", + "\n", + " datasets, info = tfds.load(name='cifar10',\n", + " with_info=True,\n", + " as_supervised=True)\n", + " return datasets['train'].map(scale).cache().shuffle(BUFFER_SIZE).repeat()\n", + "\n", + "\n", + "# Build the Keras model\n", + "def build_and_compile_cnn_model():\n", + " model = tf.keras.Sequential([\n", + " tf.keras.layers.Conv2D(32, 3, activation='relu', input_shape=(32, 32, 3)),\n", + " tf.keras.layers.MaxPooling2D(),\n", + " tf.keras.layers.Conv2D(32, 3, activation='relu'),\n", + " tf.keras.layers.MaxPooling2D(),\n", + " tf.keras.layers.Flatten(),\n", + " tf.keras.layers.Dense(10, activation='softmax')\n", + " ])\n", + " model.compile(\n", + " loss=tf.keras.losses.sparse_categorical_crossentropy,\n", + " optimizer=tf.keras.optimizers.SGD(learning_rate=args.lr),\n", + " metrics=['accuracy'])\n", + " return model\n", + "\n", + "\n", + "# Train the model\n", + "NUM_WORKERS = strategy.num_replicas_in_sync\n", + "# Here the batch size scales up by number of workers since\n", + "# `tf.data.Dataset.batch` expects the global batch size.\n", + "GLOBAL_BATCH_SIZE = BATCH_SIZE * NUM_WORKERS\n", + "train_dataset = make_datasets_unbatched().batch(GLOBAL_BATCH_SIZE)\n", + "\n", + "with strategy.scope():\n", + " # Creation of dataset, and model building/compiling need to be within\n", + " # `strategy.scope()`.\n", + " model = build_and_compile_cnn_model()\n", + "\n", + "model.fit(x=train_dataset, epochs=args.epochs, steps_per_epoch=args.steps)\n", + "model.save(args.model_dir)" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "tarball_training_script", + "repo": "snippets_custom.ipynb" + }, + "source": [ + "#### Store training script on your Cloud Storage bucket\n", + "\n", + "Next, you package the training folder into a compressed tar ball, and then store it in your Cloud Storage bucket." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "tarball_training_script", + "repo": "snippets_custom.ipynb" + }, + "outputs": [], + "source": [ + "! rm -f custom.tar custom.tar.gz\n", + "! tar cvf custom.tar custom\n", + "! gzip custom.tar\n", + "! gsutil cp custom.tar.gz $BUCKET_NAME/trainer_cifar10.tar.gz" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "train_custom_job", + "repo": "snippets_custom.ipynb" + }, + "source": [ + "### Train the model\n", + "\n", + "\n", + "Now start the training of your custom training job on Vertex AI. Use this helper function `create_custom_job`, which takes the following parameter:\n", + "\n", + "-`custom_job`: The specification for the custom job.\n", + "\n", + "The helper function calls job client service's `create_custom_job` method, with the following parameters:\n", + "\n", + "-`parent`: The Vertex AI location path to `Dataset`, `Model` and `Endpoint` resources.\n", + "-`custom_job`: The specification for the custom job.\n", + "\n", + "You will display a handful of the fields returned in `response` object, with the two that are of most interest are:\n", + "\n", + "-`response.name`: The Vertex AI fully qualified identifier assigned to this custom training job. You save this identifier for using in subsequent steps.\n", + "\n", + "-`response.state`: The current state of the custom training job." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "train_custom_job", + "repo": "snippets_custom.ipynb" + }, + "outputs": [], + "source": [ + "def create_custom_job(custom_job):\n", + " response = clients['job'].create_custom_job(parent=PARENT, custom_job=custom_job)\n", + " print(\"name:\", response.name)\n", + " print(\"display_name:\", response.display_name)\n", + " print(\"state:\", response.state)\n", + " print(\"create_time:\", response.create_time)\n", + " print(\"update_time:\", response.update_time)\n", + " return response\n", + "\n", + "\n", + "response = create_custom_job(custom_job)" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "job_id:response", + "repo": "snippets_custom.ipynb" + }, + "source": [ + "Now get the unique identifier for the custom job you created." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "job_id:response", + "repo": "snippets_custom.ipynb" + }, + "outputs": [], + "source": [ + "# The full unique ID for the custom job\n", + "job_id = response.name\n", + "# The short numeric ID for the custom job\n", + "job_short_id = job_id.split('/')[-1]\n", + "\n", + "print(job_id)" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "get_custom_job", + "repo": "snippets_custom.ipynb" + }, + "source": [ + "### Get information on a custom job\n", + "\n", + "Next, use this helper function `get_custom_job`, which takes the following parameter:\n", + "\n", + "- `name`: The Vertex AI fully qualified identifier for the custom job.\n", + "\n", + "The helper function calls the job client service's `get_custom_job` method, with the following parameter:\n", + "\n", + "- `name`: The Vertex AI fully qualified identifier for the custom job.\n", + "\n", + "If you recall, you got the Vertex AI fully qualified identifier for the custom job in the `response.name` field when you called the `create_custom_job` method, and saved the identifier in the variable `job_id`." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "get_custom_job", + "repo": "snippets_custom.ipynb" + }, + "outputs": [], + "source": [ + "def get_custom_job(name, silent=False):\n", + " response = clients['job'].get_custom_job(name=name)\n", + " if silent:\n", + " return response\n", + "\n", + " print(\"name:\", response.name)\n", + " print(\"display_name:\", response.display_name)\n", + " print(\"state:\", response.state)\n", + " print(\"create_time:\", response.create_time)\n", + " print(\"update_time:\", response.update_time)\n", + " return response\n", + "\n", + "\n", + "response = get_custom_job(job_id)" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "wait_training_complete:custom", + "repo": "snippets_custom.ipynb" + }, + "source": [ + "# Deploy the model\n", + "\n", + "Training the above model may take upwards of 20 minutes time.\n", + "\n", + "Once your model is done training, you can calculate the actual time it took to train the model by subtracting `end_time` from `start_time`. For your model, we will need to know the location of the saved model, which the Python script saved in your local Cloud Storage bucket at `MODEL_DIR + '/saved_model.pb'`." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "wait_training_complete:custom", + "repo": "snippets_custom.ipynb" + }, + "outputs": [], + "source": [ + "while True:\n", + " response = get_custom_job(job_id, True)\n", + " if response.state != aip.JobState.JOB_STATE_SUCCEEDED:\n", + " print(\"Training job has not completed:\", response.state)\n", + " model_path_to_deploy = None\n", + " if response.state == aip.JobState.JOB_STATE_FAILED:\n", + " break\n", + " else:\n", + " if not DIRECT:\n", + " MODEL_DIR = MODEL_DIR + \"/model\"\n", + " model_path_to_deploy = MODEL_DIR\n", + " print(\"Training Job Time:\", response.end_time - response.start_time)\n", + " print(\"Training Elapsed Time:\", response.update_time - response.create_time)\n", + " break\n", + " time.sleep(60)\n", + "\n", + "print(\"model_to_deploy:\", model_path_to_deploy)" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "load_saved_model", + "repo": "snippets_custom.ipynb" + }, + "source": [ + "## Load the saved model\n", + "\n", + "Your model is stored in a TensorFlow SavedModel format in a Cloud Storage bucket. Now load it from the Cloud Storage bucket, and then you can do some things, like evaluate the model, and do a prediction.\n", + "\n", + "To load, you use the TF.Keras `model.load_model()` method passing it the Cloud Storage path where the model is saved -- specified by `MODEL_DIR`." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "load_saved_model", + "repo": "snippets_custom.ipynb" + }, + "outputs": [], + "source": [ + "import tensorflow as tf\n", + "\n", + "model = tf.keras.models.load_model(MODEL_DIR)" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "evaluate_custom_model:image", + "repo": "snippets_custom.ipynb" + }, + "source": [ + "## Evaluate the model\n", + "\n", + "Now find out how good the model is.\n", + "\n", + "### Load evaluation data\n", + "\n", + "You will load the CIFAR10 test (holdout) data from `tf.keras.datasets`, using the method `load_data()`. This returns the dataset as a tuple of two elements. The first element is the training data and the second is the test data. Each element is also a tuple of two elements: the image data, and the corresponding labels.\n", + "\n", + "You don't need the training data, and hence why we loaded it as `(_, _)`.\n", + "\n", + "Before you can run the data through evaluation, you need to preprocess it:\n", + "\n", + "`x_test`:\n", + "1. Normalize (rescale) the pixel data by dividing each pixel by 255. This replaces each single byte integer pixel with a 32-bit floating point number between 0 and 1.\n", + "\n", + "`y_test`:
\n", + "2. The labels are currently scalar (sparse). If you look back at the `compile()` step in the `trainer/task.py` script, you will find that it was compiled for sparse labels. So we don't need to do anything more." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "evaluate_custom_model:image,cifar10", + "repo": "snippets_custom.ipynb" + }, + "outputs": [], + "source": [ + "from tensorflow.keras.datasets import cifar10\n", + "import numpy as np\n", + "\n", + "(_, _), (x_test, y_test) = cifar10.load_data()\n", + "x_test = (x_test / 255.0).astype(np.float32)\n", + "\n", + "print(x_test.shape, y_test.shape)" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "perform_evaluation_custom", + "repo": "snippets_custom.ipynb" + }, + "source": [ + "### Perform the model evaluation\n", + "\n", + "Now evaluate how well the model in the custom job did." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "perform_evaluation_custom", + "repo": "snippets_custom.ipynb" + }, + "outputs": [], + "source": [ + "model.evaluate(x_test, y_test)" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "how_serving_function_works", + "repo": "snippets_custom.ipynb" + }, + "source": [ + "## Upload the model for serving\n", + "\n", + "Next, you will upload your TF.Keras model from the custom job to Vertex `Model` service, which will create a Vertex `Model` resource for your custom model. During upload, you need to define a serving function to convert data to the format your model expects. If you send encoded data to Vertex AI, your serving function ensures that the data is decoded on the model server before it is passed as input to your model.\n", + "\n", + "### How does the serving function work\n", + "\n", + "When you send a request to an online prediction server, the request is received by a HTTP server. The HTTP server extracts the prediction request from the HTTP request content body. The extracted prediction request is forwarded to the serving function. For Google pre-built prediction containers, the request content is passed to the serving function as a `tf.string`.\n", + "\n", + "The serving function consists of two parts:\n", + "\n", + "- `preprocessing function`:\n", + " - Converts the input (`tf.string`) to the input shape and data type of the underlying model (dynamic graph).\n", + " - Performs the same preprocessing of the data that was done during training the underlying model, including normalizing and scalingh.\n", + "- `post-processing function`:\n", + " - Converts the model output to format expected by the receiving application -- e.q., compresses the output.\n", + " - Packages the output for the the receiving application, including adding headings, and making JSON objects.\n", + "\n", + "Both the preprocessing and post-processing functions are converted to static graphs which are fused to the model. The output from the underlying model is passed to the post-processing function. The post-processing function passes the converted/packaged output back to the HTTP server. The HTTP server returns the output as the HTTP response content.\n", + "\n", + "One consideration you need to consider when building serving functions for TF.Keras models is that they run as static graphs. That means, you cannot use TF graph operations that require a dynamic graph. If you do, you will get an error during the compile of the serving function which will indicate that you are using an EagerTensor which is not supported." + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "serving_function_image:xai", + "repo": "snippets_custom.ipynb" + }, + "source": [ + "### Serving function for image data\n", + "\n", + "To pass images to the prediction service, you encode the compressed (e.g., JPEG) image bytes into base 64 -- which makes the content safe from modification while transmitting binary data over the network. Since this deployed model expects input data as raw (uncompressed) bytes, you need to ensure that the base 64 encoded data gets converted back to raw bytes before it is passed as input to the deployed model.\n", + "\n", + "To resolve this, define a serving function (`serving_fn`) and attach it to the model as a preprocessing step. Add a `@tf.function` decorator so the serving function is fused to the underlying model (instead of upstream on a CPU).\n", + "\n", + "When you send a prediction or explanation request, the content of the request is base 64 decoded into a TensorFlow string (`tf.string`), which is passed to the serving function (`serving_fn`). The serving function preprocesses the `tf.string` into raw (uncompressed) numpy bytes (`preprocess_fn`) to match the input requirements of the model:\n", + "- `io.decode_jpeg`- Decompresses the JPG image which is returned as a Tensorflow tensor with three channels (RGB).\n", + "- `image.convert_image_dtype` - Changes integer pixel values to float 32.\n", + "- `image.resize` - Resizes the image to match the input shape for the model.\n", + "- `resized / 255.0` - Rescales (normalization) the pixel data between 0 and 1.\n", + "\n", + "At this point, the data can be passed to the model (`m_call`).\n", + "\n", + "#### Vertex AI Explainability Signatures\n", + "\n", + "When the serving function is saved back with the underlying model (`tf.saved_model.save`), you specify the input layer of the serving function as the signature `serving_default`.\n", + "\n", + "For Vertex AI Explainability image models, you need to save two additional signatures from the serving function:\n", + "\n", + "- `xai_preprocess`: The preprocessing function in the serving function.\n", + "- `xai_model`: The concrete function for calling the model." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "serving_function_image:xai", + "repo": "snippets_custom.ipynb" + }, + "outputs": [], + "source": [ + "CONCRETE_INPUT = \"numpy_inputs\"\n", + "\n", + "def _preprocess(bytes_input):\n", + " decoded = tf.io.decode_jpeg(bytes_input, channels=3)\n", + " decoded = tf.image.convert_image_dtype(decoded, tf.float32)\n", + " resized = tf.image.resize(decoded, size=(32, 32))\n", + " rescale = tf.cast(resized / 255.0, tf.float32)\n", + " return rescale\n", + "\n", + "\n", + "@tf.function(input_signature=[tf.TensorSpec([None], tf.string)])\n", + "def preprocess_fn(bytes_inputs):\n", + " decoded_images = tf.map_fn(_preprocess, bytes_inputs, dtype=tf.float32, back_prop=False)\n", + " return {CONCRETE_INPUT: decoded_images} # User needs to make sure the key matches model's input\n", + "\n", + "\n", + "@tf.function(input_signature=[tf.TensorSpec([None], tf.string)])\n", + "def serving_fn(bytes_inputs):\n", + " images = preprocess_fn(bytes_inputs)\n", + " prob = m_call(**images)\n", + " return prob\n", + "\n", + "\n", + "m_call = tf.function(model.call).get_concrete_function([tf.TensorSpec(shape=[None, 32, 32, 3], dtype=tf.float32, name=CONCRETE_INPUT)])\n", + "\n", + "tf.saved_model.save(model, model_path_to_deploy, signatures={\n", + " 'serving_default': serving_fn,\n", + " # Required for XAI\n", + " 'xai_preprocess': preprocess_fn,\n", + " 'xai_model': m_call\n", + "})" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "serving_function_signature:xai", + "repo": "snippets_custom.ipynb" + }, + "source": [ + "## Get the serving function signature\n", + "\n", + "You can get the signatures of your model's input and output layers by reloading the model into memory, and querying it for the signatures corresponding to each layer.\n", + "\n", + "When making a prediction request, you need to route the request to the serving function instead of the model, so you need to know the input layer name of the serving function -- which you will use later when you make a prediction request.\n", + "\n", + "You also need to know the name of the serving function's input and output layer for constructing the explanation metadata -- which is discussed subsequently." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "serving_function_signature:xai", + "repo": "snippets_custom.ipynb" + }, + "outputs": [], + "source": [ + "loaded = tf.saved_model.load(model_path_to_deploy)\n", + "\n", + "serving_input = list(loaded.signatures['serving_default'].structured_input_signature[1].keys())[0]\n", + "print('Serving function input:', serving_input)\n", + "serving_output = list(loaded.signatures['serving_default'].structured_outputs.keys())[0]\n", + "print('Serving function output:', serving_output)\n", + "\n", + "input_name = model.input.name\n", + "print('Model input name:', input_name)\n", + "output_name = model.output.name\n", + "print('Model output name:', output_name)" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "explanation_spec", + "repo": "snippets_custom.ipynb" + }, + "source": [ + "### Explanation Specification\n", + "\n", + "To get explanations when doing a prediction, you must enable the explanation capability and set corresponding settings when you upload your custom model to a Vertex `Model` resource. These settings are referred to as the explanation metadata, which consists of:\n", + "\n", + "- `parameters`: This is the specification for the explainability algorithm to use for explanations on your model. You can choose between:\n", + " - Shapley - *Note*, not recommended for image data -- can be very long running\n", + " - XRAI\n", + " - Integrated Gradients\n", + "- `metadata`: This is the specification for how the algoithm is applied on your custom model.\n", + "\n", + "#### Explanation Parameters\n", + "\n", + "Let's first dive deeper into the settings for the explainability algorithm.\n", + "\n", + "#### Shapley\n", + "\n", + "Assigns credit for the outcome to each feature, and considers different permutations of the features. This method provides a sampling approximation of exact Shapley values.\n", + "\n", + "Use Cases:\n", + " - Classification and regression on tabular data.\n", + "\n", + "Parameters:\n", + "\n", + "- `path_count`: This is the number of paths over the features that will be processed by the algorithm. An exact approximation of the Shapley values requires M! paths, where M is the number of features. For the CIFAR10 dataset, this would be 784 (28*28).\n", + "\n", + "For any non-trival number of features, this is too compute expensive. You can reduce the number of paths over the features to M * `path_count`.\n", + "\n", + "#### Integrated Gradients\n", + "\n", + "A gradients-based method to efficiently compute feature attributions with the same axiomatic properties as the Shapley value.\n", + "\n", + "Use Cases:\n", + " - Classification and regression on tabular data.\n", + " - Classification on image data.\n", + "\n", + "Parameters:\n", + "\n", + "- `step_count`: This is the number of steps to approximate the remaining sum. The more steps, the more accurate the integral approximation. The general rule of thumb is 50 steps, but as you increase so does the compute time.\n", + "\n", + "#### XRAI\n", + "\n", + "Based on the integrated gradients method, XRAI assesses overlapping regions of the image to create a saliency map, which highlights relevant regions of the image rather than pixels.\n", + "\n", + "Use Cases:\n", + "\n", + " - Classification on image data.\n", + "\n", + "Parameters:\n", + "\n", + "- `step_count`: This is the number of steps to approximate the remaining sum. The more steps, the more accurate the integral approximation. The general rule of thumb is 50 steps, but as you increase so does the compute time.\n", + "\n", + "In the next code cell, set the variable `XAI` to which explainabilty algorithm you will use on your custom model." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "explanation_spec", + "repo": "snippets_custom.ipynb" + }, + "outputs": [], + "source": [ + "XAI = \"ig\" # [ shapley, ig, xrai ]\n", + "\n", + "if XAI == \"shapley\":\n", + " PARAMETERS = {\n", + " \"sampled_shapley_attribution\": {'path_count': 10}\n", + " }\n", + "elif XAI == \"ig\":\n", + " PARAMETERS = {\n", + " \"integrated_gradients_attribution\": {\"step_count\": 50}\n", + " }\n", + "elif XAI == \"xrai\":\n", + " PARAMETERS = {\n", + " \"xrai_attribution\": {\"step_count\": 50}\n", + " }\n", + "\n", + "parameters = aip.ExplanationParameters(PARAMETERS)" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "explanation_metadata:image", + "repo": "snippets_custom.ipynb" + }, + "source": [ + "#### Explanation Metadata\n", + "\n", + "Let's first dive deeper into the explanation metadata, which consists of:\n", + "\n", + "- `outputs`: A scalar value in the output to attribute -- what to explain. For example, in a probability output \\[0.1, 0.2, 0.7\\] for classification, one wants an explanation for 0.7. Consider the following formulae, where the output is `y` and that is what we want to explain.\n", + "\n", + " y = f(x)\n", + "\n", + "Consider the following formulae, where the outputs are `y` and `z`. Since we can only do attribution for one scalar value, we have to pick whether we want to explain the output `y` or `z`. Assume in this example the model is object detection and y and z are the bounding box and the object classification. You would want to pick which of the two outputs to explain.\n", + "\n", + " y, z = f(x)\n", + "\n", + "The dictionary format for `outputs` is:\n", + "\n", + " { \"outputs\": { \"[your_display_name]\":\n", + " \"output_tensor_name\": [layer]\n", + " }\n", + " }\n", + "\n", + "
\n", + " \n", + "
\n", + "\n", + "\n", + "- `inputs`: The features for attribution -- how they contributed to the output. Consider the following formulae, where `a` and `b` are the features. We have to pick which features to explain how the contributed. Assume that this model is deployed for A/B testing, where `a` are the data_items for the prediction and `b` identifies whether the model instance is A or B. You would want to pick `a` (or some subset of) for the features, and not `b` since it does not contribute to the prediction.\n", + "\n", + " y = f(a,b)\n", + "\n", + "The minimum dictionary format for `inputs` is:\n", + "\n", + " { \"inputs\": { \"[your_display_name]\":\n", + " \"input_tensor_name\": [layer]\n", + " }\n", + " }\n", + "\n", + "
\n", + " \n", + "
\n", + "\n", + "Since the inputs to the model are images, you can specify the following additional fields as reporting/visualization aids:\n", + "\n", + "
\n", + " \n", + "
" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "explanation_metadata:image", + "repo": "snippets_custom.ipynb" + }, + "outputs": [], + "source": [ + "random_baseline = np.random.rand(32, 32, 3)\n", + "input_baselines = [{'number_vaue': x} for x in random_baseline]\n", + "\n", + "INPUT_METADATA = {'input_tensor_name': CONCRETE_INPUT,\n", + " \"modality\": \"image\"\n", + "}\n", + "\n", + "OUTPUT_METADATA = {'output_tensor_name': serving_output}\n", + "\n", + "input_metadata = aip.ExplanationMetadata.InputMetadata(INPUT_METADATA)\n", + "output_metadata = aip.ExplanationMetadata.OutputMetadata(OUTPUT_METADATA)\n", + "\n", + "metadata=aip.ExplanationMetadata(\n", + " inputs={'image': input_metadata},\n", + " outputs={'class' : output_metadata}\n", + ")\n", + "\n", + "explanation_spec = aip.ExplanationSpec(\n", + " metadata=metadata,\n", + " parameters=parameters\n", + ")" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "upload_the_model:explanation", + "repo": "snippets_custom.ipynb" + }, + "source": [ + "### Upload the model\n", + "\n", + "Use this helper function `upload_model` to upload your model, stored in SavedModel format, up to the `Model` service, which will instantiate a Vertex `Model` resource instance for your model. Once you've done that, you can use the `Model` resource instance in the same way as any other Vertex `Model` resource instance, such as deploying to an `Endpoint` resource for serving predictions.\n", + "\n", + "The helper function takes the following parameters:\n", + "\n", + "- `display_name`: A human readable name for the `Endpoint` service.\n", + "- `image_uri`: The container image for the model deployment.\n", + "- `model_uri`: The Cloud Storage path to our SavedModel artificat. For this tutorial, this is the Cloud Storage location where the `trainer/task.py` saved the model artifacts, which we specified in the variable `MODEL_DIR`.\n", + "\n", + "The helper function calls the `Model` client service's method `upload_model`, which takes the following parameters:\n", + "\n", + "- `parent`: The Vertex AI location root path for `Dataset`, `Model` and `Endpoint` resources.\n", + "- `model`: The specification for the Vertex `Model` resource instance.\n", + "\n", + "Let's now dive deeper into the Vertex model specification `model`. This is a dictionary object that consists of the following fields:\n", + "\n", + "- `display_name`: A human readable name for the `Model` resource.\n", + "- `metadata_schema_uri`: Since your model was built without an Vertex `Dataset` resource, you will leave this blank (`''`).\n", + "- `artifact_uri`: The Cloud Storage path where the model is stored in SavedModel format.\n", + "- `container_spec`: This is the specification for the Docker container that will be installed on the `Endpoint` resource, from which the `Model` resource will serve predictions. Use the variable you set earlier `DEPLOY_GPU != None` to use a GPU; otherwise only a CPU is allocated.\n", + "- `explanation_spec`: This is the specification for enabling explainability for your model.\n", + "\n", + "Uploading a model into a Vertex `Model` resource returns a long running operation, since it may take a few moments. You call response.result(), which is a synchronous call and will return when the Vertex Model resource is ready.\n", + "\n", + "The helper function returns the Vertex AI fully qualified identifier for the corresponding Vertex `Model` instance upload_model_response.model. You will save the identifier for subsequent steps in the variable model_to_deploy_id." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "upload_the_model:explanation", + "repo": "snippets_custom.ipynb" + }, + "outputs": [], + "source": [ + "IMAGE_URI = DEPLOY_IMAGE\n", + "\n", + "\n", + "def upload_model(display_name, image_uri, model_uri):\n", + "\n", + " model = aip.Model(display_name=display_name,\n", + " artifact_uri=model_uri,\n", + " metadata_schema_uri=\"\",\n", + " explanation_spec=explanation_spec,\n", + " container_spec={\n", + " \"image_uri\": image_uri\n", + " })\n", + "\n", + " response = clients['model'].upload_model(parent=PARENT, model=model)\n", + " print(\"Long running operation:\", response.operation.name)\n", + " upload_model_response = response.result(timeout=180)\n", + " print(\"upload_model_response\")\n", + " print(\" model:\", upload_model_response.model)\n", + " return upload_model_response.model\n", + "\n", + "\n", + "model_to_deploy_id = upload_model(\"cifar10-\" + TIMESTAMP, IMAGE_URI, model_path_to_deploy)" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "get_model", + "repo": "snippets_common.ipynb" + }, + "source": [ + "### Get `Model` resource information\n", + "\n", + "Now let's get the model information for just your model. Use this helper function `get_model`, with the following parameter:\n", + "\n", + "- `name`: The Vertex AI unique identifier for the `Model` resource.\n", + "\n", + "This helper function calls the Vertex AI `Model` client service's method `get_model`, with the following parameter:\n", + "\n", + "- `name`: The Vertex AI unique identifier for the `Model` resource." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "get_model", + "repo": "snippets_common.ipynb" + }, + "outputs": [], + "source": [ + "def get_model(name):\n", + " response = clients['model'].get_model(name=name)\n", + " print(response)\n", + "\n", + "\n", + "get_model(model_to_deploy_id)" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "deploy:batch_prediction", + "repo": "snippets_common.ipynb" + }, + "source": [ + "## Model deployment for batch prediction\n", + "\n", + "Now deploy the trained Vertex `Model` resource you created for batch prediction. This differs from deploying a `Model` resource for online prediction.\n", + "\n", + "For online prediction, you:\n", + "\n", + "1. Create an `Endpoint` resource for deploying the `Model` resource to.\n", + "\n", + "2. Deploy the `Model` resource to the `Endpoint` resource.\n", + "\n", + "3. Make online prediction requests to the `Endpoint` resource.\n", + "\n", + "For batch-prediction, you:\n", + "\n", + "1. Create a batch prediction job.\n", + "\n", + "2. The job service will provision resources for the batch prediction request.\n", + "\n", + "3. The results of the batch prediction request are returned to the caller.\n", + "\n", + "4. The job service will unprovision the resoures for the batch prediction request." + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "make_prediction", + "repo": "snippets_common.ipynb" + }, + "source": [ + "## Send a batch prediction request\n", + "\n", + "Send a batch prediction to your deployed model." + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "get_test_items:test", + "repo": "snippets_common.ipynb" + }, + "source": [ + "### Get test items\n", + "\n", + "You will use examples out of the test (holdout) portion of the dataset as a test items." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "get_test_items:test", + "repo": "snippets_common.ipynb" + }, + "outputs": [], + "source": [ + "test_image_1 = x_test[0]\n", + "test_label_1 = y_test[0]\n", + "test_image_2 = x_test[1]\n", + "test_label_2 = y_test[1]\n", + "print(test_image_1.shape)" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "prepare_test_items:test,image", + "repo": "snippets_custom.ipynb" + }, + "source": [ + "### Prepare the request content\n", + "You are going to send the CIFAR10 images as compressed JPG image, instead of the raw uncompressed bytes:\n", + "\n", + "- `cv2.imwrite`: Use openCV to write the uncompressed image to disk as a compressed JPEG image.\n", + " - Denormalize the image data from \\[0,1) range back to [0,255).\n", + " - Convert the 32-bit floating point values to 8-bit unsigned integers." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "prepare_test_items:test,image", + "repo": "snippets_custom.ipynb" + }, + "outputs": [], + "source": [ + "import cv2\n", + "cv2.imwrite('tmp1.jpg', (test_image_1 * 255).astype(np.uint8))\n", + "cv2.imwrite('tmp2.jpg', (test_image_2 * 255).astype(np.uint8))" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "copy_test_items:test", + "repo": "snippets_common.ipynb" + }, + "source": [ + "### Copy test item(s)\n", + "\n", + "For the batch prediction, you will copy the test items over to your Cloud Storage bucket." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "copy_test_items:test", + "repo": "snippets_common.ipynb" + }, + "outputs": [], + "source": [ + "! gsutil cp tmp1.jpg $BUCKET_NAME/tmp1.jpg\n", + "! gsutil cp tmp2.jpg $BUCKET_NAME/tmp2.jpg\n", + "\n", + "test_item_1 = BUCKET_NAME + \"/tmp1.jpg\"\n", + "test_item_2 = BUCKET_NAME + \"/tmp2.jpg\"" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "make_batch_file:custom,image", + "repo": "snippets_custom.ipynb" + }, + "source": [ + "### Make the batch input file\n", + "\n", + "Now make a batch input file, which you will store in your local Cloud Storage bucket. The batch input file can only be in JSONL format. For JSONL file, you make one dictionary entry per line for each data item (instance). The dictionary contains the key/value pairs:\n", + "\n", + "- `input_name`: the name of the input layer of the underlying model.\n", + "- `'b64'`: A key that indicates the content is base64 encoded.\n", + "- `content`: The compressed JPG image bytes as a base64 encoded string.\n", + "\n", + "Each instance in the prediction request is a dictionary entry of the form:\n", + "\n", + " {serving_input: {'b64': content}}\n", + "\n", + "To pass the image data to the prediction service you encode the bytes into base64 -- which makes the content safe from modification when transmitting binary data over the network.\n", + "\n", + "- `tf.io.read_file`: Read the compressed JPG images into memory as raw bytes.\n", + "- `base64.b64encode`: Encode the raw bytes into a base64 encoded string." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "make_batch_file:custom,image", + "repo": "snippets_custom.ipynb" + }, + "outputs": [], + "source": [ + "import base64\n", + "import json\n", + "\n", + "gcs_input_uri = BUCKET_NAME + \"/\" + \"test.jsonl\"\n", + "with tf.io.gfile.GFile(gcs_input_uri, 'w') as f:\n", + " bytes = tf.io.read_file(test_item_1)\n", + " b64str = base64.b64encode(bytes.numpy()).decode('utf-8')\n", + " data = {serving_input: {'b64': b64str}}\n", + " f.write(json.dumps(data) + '\\n')\n", + " bytes = tf.io.read_file(test_item_2)\n", + " b64str = base64.b64encode(bytes.numpy()).decode('utf-8')\n", + " data = {serving_input: {'b64': b64str}}\n", + " f.write(json.dumps(data) + '\\n')" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "instance_scaling", + "repo": "snippets_common.ipynb" + }, + "source": [ + "### Compute instance scaling\n", + "\n", + "You have several choices on scaling the compute instances for handling your batch prediction requests:\n", + "\n", + "- Single Instance: The batch prediction requests are processed on a single compute instance.\n", + " - Set the minimum (`MIN_NODES`) and maximum (`MAX_NODES`) number of compute instances to one.\n", + "\n", + "- Manual Scaling: The batch prediction requests are split across a fixed number of compute instances that you manually specified.\n", + " - Set the minimum (`MIN_NODES`) and maximum (`MAX_NODES`) number of compute instances to the same number of nodes. When a model is first deployed to the instance, the fixed number of compute instances are provisioned and batch prediction requests are evenly distributed across them.\n", + "\n", + "- Auto Scaling: The batch prediction requests are split across a scaleable number of compute instances.\n", + " - Set the minimum (`MIN_NODES`) number of compute instances to provision when a model is first deployed and to de-provision, and set the maximum (`MAX_NODES) number of compute instances to provision, depending on load conditions.\n", + "\n", + "The minimum number of compute instances corresponds to the field `min_replica_count` and the maximum number of compute instances corresponds to the field `max_replica_count`, in your subsequent deployment request." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "instance_scaling", + "repo": "snippets_common.ipynb" + }, + "outputs": [], + "source": [ + "MIN_NODES = 1\n", + "MAX_NODES = 1" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "make_batch_request:custom", + "repo": "snippets_custom.ipynb" + }, + "source": [ + "### Make batch prediction request\n", + "\n", + "Now that your batch of two test items is ready, let's do the batch request. Use this helper function `create_batch_prediction_job`, with the following parameters:\n", + "\n", + "- `display_name`: The human readable name for the prediction job.\n", + "- `model_name`: The Vertex AI fully qualified identifier for the `Model` resource.\n", + "- `gcs_source_uri`: The Cloud Storage path to the input file -- which you created above.\n", + "- `gcs_destination_output_uri_prefix`: The Cloud Storage path that the service will write the predictions to.\n", + "- `parameters`: Additional filtering parameters for serving prediction results.\n", + "\n", + "The helper function calls the job client service's `create_batch_prediction_job` metho, with the following parameters:\n", + "\n", + "- `parent`: The Vertex AI location root path for Dataset, Model and Pipeline resources.\n", + "- `batch_prediction_job`: The specification for the batch prediction job.\n", + "\n", + "Let's now dive into the specification for the `batch_prediction_job`:\n", + "\n", + "- `display_name`: The human readable name for the prediction batch job.\n", + "- `model`: The Vertex AI fully qualified identifier for the `Model` resource.\n", + "- `dedicated_resources`: The compute resources to provision for the batch prediction job.\n", + " - `machine_spec`: The compute instance to provision. Use the variable you set earlier `DEPLOY_GPU != None` to use a GPU; otherwise only a CPU is allocated.\n", + " - `starting_replica_count`: The number of compute instances to initially provision, which you set earlier as the variable `MIN_NODES`.\n", + " - `max_replica_count`: The maximum number of compute instances to scale to, which you set earlier as the variable `MAX_NODES`.\n", + "- `model_parameters`: Additional filtering parameters for serving prediction results. No Additional parameters are supported for custom models.\n", + "- `input_config`: The input source and format type for the instances to predict.\n", + " - `instances_format`: The format of the batch prediction request file: `csv` or `jsonl`.\n", + " - `gcs_source`: A list of one or more Cloud Storage paths to your batch prediction requests.\n", + "- `output_config`: The output destination and format for the predictions.\n", + " - `prediction_format`: The format of the batch prediction response file: `csv` or `jsonl`.\n", + " - `gcs_destination`: The output destination for the predictions.\n", + "\n", + "This call is an asychronous operation. You will print from the response object a few select fields, including:\n", + "\n", + "- `name`: The Vertex AI fully qualified identifier assigned to the batch prediction job.\n", + "- `display_name`: The human readable name for the prediction batch job.\n", + "- `model`: The Vertex AI fully qualified identifier for the Model resource.\n", + "- `generate_explanations`: Whether True/False explanations were provided with the predictions (explainability).\n", + "- `state`: The state of the prediction job (pending, running, etc).\n", + "\n", + "Since this call will take a few moments to execute, you will likely get `JobState.JOB_STATE_PENDING` for `state`." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "make_batch_request:custom", + "repo": "snippets_custom.ipynb" + }, + "outputs": [], + "source": [ + "BATCH_MODEL = \"cifar10_batch-\" + TIMESTAMP\n", + "\n", + "\n", + "def create_batch_prediction_job(display_name, model_name, gcs_source_uri, gcs_destination_output_uri_prefix, parameters=None):\n", + "\n", + " if DEPLOY_GPU:\n", + " machine_spec = {\n", + " \"machine_type\": DEPLOY_COMPUTE,\n", + " \"accelerator_type\": DEPLOY_GPU,\n", + " \"accelerator_count\": DEPLOY_NGPU,\n", + " }\n", + " else:\n", + " machine_spec = {\n", + " \"machine_type\": DEPLOY_COMPUTE,\n", + " \"accelerator_count\": 0,\n", + " }\n", + "\n", + " batch_prediction_job = {\n", + " \"display_name\": display_name,\n", + " # Format: 'projects/{project}/locations/{location}/models/{model_id}'\n", + " \"model\": model_name,\n", + " \"model_parameters\": json_format.ParseDict(parameters, Value()),\n", + " \"input_config\": {\n", + " \"instances_format\": IN_FORMAT,\n", + " \"gcs_source\": {\"uris\": [gcs_source_uri]},\n", + " },\n", + " \"output_config\": {\n", + " \"predictions_format\": OUT_FORMAT,\n", + " \"gcs_destination\": {\"output_uri_prefix\": gcs_destination_output_uri_prefix},\n", + " },\n", + " \"dedicated_resources\": {\n", + " \"machine_spec\": machine_spec,\n", + " \"starting_replica_count\": MIN_NODES,\n", + " \"max_replica_count\": MAX_NODES\n", + " }\n", + " , 'generate_explanation': True\n", + " }\n", + " response = clients['job'].create_batch_prediction_job(\n", + " parent=PARENT, batch_prediction_job=batch_prediction_job\n", + " )\n", + " print(\"response\")\n", + " print(\" name:\", response.name)\n", + " print(\" display_name:\", response.display_name)\n", + " print(\" model:\", response.model)\n", + " try:\n", + " print(\" generate_explanation:\", response.generate_explanation)\n", + " except:\n", + " pass\n", + " print(\" state:\", response.state)\n", + " print(\" create_time:\", response.create_time)\n", + " print(\" start_time:\", response.start_time)\n", + " print(\" end_time:\", response.end_time)\n", + " print(\" update_time:\", response.update_time)\n", + " print(\" labels:\", response.labels)\n", + " return response\n", + "\n", + "\n", + "IN_FORMAT = 'jsonl'\n", + "OUT_FORMAT = 'jsonl'\n", + "\n", + "response = create_batch_prediction_job(BATCH_MODEL, model_to_deploy_id, gcs_input_uri, BUCKET_NAME)" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "batch_job_id:response", + "repo": "snippets_common.ipynb" + }, + "source": [ + "Now get the unique identifier for the batch prediction job you created." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "batch_job_id:response", + "repo": "snippets_common.ipynb" + }, + "outputs": [], + "source": [ + "# The full unique ID for the batch job\n", + "batch_job_id = response.name\n", + "# The short numeric ID for the batch job\n", + "batch_job_short_id = batch_job_id.split('/')[-1]\n", + "\n", + "print(batch_job_id)" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "get_batch_prediction_job", + "repo": "snippets_common.ipynb" + }, + "source": [ + "### Get information on a batch prediction job\n", + "\n", + "Use this helper function `get_batch_prediction_job`, with the following paramter:\n", + "\n", + "- `job_name`: The Vertex AI fully qualified identifier for the batch prediction job.\n", + "\n", + "The helper function calls the job client service's `get_batch_prediction_job` method, with the following paramter:\n", + "\n", + "- `name`: The Vertex AI fully qualified identifier for the batch prediction job. In this tutorial, you will pass it the Vertex fully qualified identifier for your batch prediction job -- `batch_job_id`\n", + "\n", + "The helper function will return the Cloud Storage path to where the predictions are stored -- `gcs_destination`." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "get_batch_prediction_job", + "repo": "snippets_common.ipynb" + }, + "outputs": [], + "source": [ + "def get_batch_prediction_job(job_name, silent=False):\n", + " response = clients['job'].get_batch_prediction_job(name=job_name)\n", + " if silent:\n", + " return response.output_config.gcs_destination.output_uri_prefix, response.state\n", + "\n", + " print(\"response\")\n", + " print(\" name:\", response.name)\n", + " print(\" display_name:\", response.display_name)\n", + " print(\" model:\", response.model)\n", + " try: # not all data types support explanations\n", + " print(\" generate_explanation:\", response.generate_explanation)\n", + " except:\n", + " pass\n", + " print(\" state:\", response.state)\n", + " print(\" error:\", response.error)\n", + " gcs_destination = response.output_config.gcs_destination\n", + " print(\" gcs_destination\")\n", + " print(\" output_uri_prefix:\", gcs_destination.output_uri_prefix)\n", + " return gcs_destination.output_uri_prefix, response.state\n", + "\n", + "\n", + "predictions, state = get_batch_prediction_job(batch_job_id)" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "get_the_explanations:custom,image", + "repo": "snippets_custom.ipynb" + }, + "source": [ + "### Get the predictions\n", + "\n", + "When the batch prediction is done processing, the job state will be `JOB_STATE_SUCCEEDED`.\n", + "\n", + "Finally you view the predictions stored at the Cloud Storage path you set as output. The predictions will be in a JSONL format, which you indicated at the time you made the batch prediction job. The predictions are in a subdirectory starting with the name `prediction`. Within that subdirectory there is a file named `prediction.results-xxxxx-of-xxxxx`.\n", + "\n", + "Now display (cat) the contents. You will see multiple JSON objects, one for each prediction.\n", + "\n", + "Finally you view the explanations stored at the Cloud Storage path you set as output. The explanations will be in a JSONL format, which you indicated at the time you made the batch explanation job. The explanations are in a subdirectory starting with the name `prediction`. Within that subdirectory there is a file named `explanations.results-xxxxx-of-xxxxx`.\n", + "\n", + "Let's display (cat) the contents. You will a row for each prediction -- in this case, there is just one row. The row is the softmax probability distribution for the corresponding CIFAR10 classes." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "get_the_explanations:custom,image", + "repo": "snippets_custom.ipynb" + }, + "outputs": [], + "source": [ + "def get_latest_predictions(gcs_out_dir):\n", + " ''' Get the latest prediction subfolder using the timestamp in the subfolder name'''\n", + " folders = !gsutil ls $gcs_out_dir\n", + " latest = \"\"\n", + " for folder in folders:\n", + " subfolder = folder.split('/')[-2]\n", + " if subfolder.startswith('prediction-'):\n", + " if subfolder > latest:\n", + " latest = folder[:-1]\n", + " return latest\n", + "\n", + "while True:\n", + " predictions, state = get_batch_prediction_job(batch_job_id, True)\n", + " if state != aip.JobState.JOB_STATE_SUCCEEDED:\n", + " print(\"The job has not completed:\", state)\n", + " if state == aip.JobState.JOB_STATE_FAILED:\n", + " break\n", + " else:\n", + " folder = get_latest_predictions(predictions)\n", + " ! gsutil ls $folder/explanation.results*\n", + "\n", + " print(\"Results:\")\n", + " ! gsutil cat $folder/explanation.results*\n", + "\n", + " print(\"Errors:\")\n", + " ! gsutil cat $folder/prediction.errors*\n", + " break\n", + " time.sleep(60)" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "cleanup", + "repo": "snippets_common.ipynb" + }, + "source": [ + "# Cleaning up\n", + "\n", + "To clean up all Google Cloud resources used in this project, you can [delete the GCP\n", + "project](https://cloud.google.com/resource-manager/docs/creating-managing-projects#shutting_down_projects) you used for the tutorial.\n", + "\n", + "Otherwise, you can delete the individual resources you created in this tutorial:\n", + "\n", + "- Dataset\n", + "- Pipeline\n", + "- Model\n", + "- Endpoint\n", + "- Batch Job\n", + "- Custom Job\n", + "- Hyperparameter Tuning Job\n", + "- Cloud Storage Bucket" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "cleanup", + "repo": "snippets_common.ipynb" + }, + "outputs": [], + "source": [ + "delete_dataset = True\n", + "delete_pipeline = True\n", + "delete_model = True\n", + "delete_endpoint = True\n", + "delete_batchjob = True\n", + "delete_customjob = True\n", + "delete_hptjob = True\n", + "delete_bucket = True\n", + "\n", + "# Delete the dataset using the Vertex fully qualified identifier for the dataset\n", + "try:\n", + " if delete_dataset and 'dataset_id' in globals():\n", + " clients['dataset'].delete_dataset(name=dataset_id)\n", + "except Exception as e:\n", + " print(e)\n", + "\n", + "# Delete the training pipeline using the Vertex fully qualified identifier for the pipeline\n", + "try:\n", + " if delete_pipeline and 'pipeline_id' in globals():\n", + " clients['pipeline'].delete_training_pipeline(name=pipeline_id)\n", + "except Exception as e:\n", + " print(e)\n", + "\n", + "# Delete the model using the Vertex fully qualified identifier for the model\n", + "try:\n", + " if delete_model and 'model_to_deploy_id' in globals():\n", + " clients['model'].delete_model(name=model_to_deploy_id)\n", + "except Exception as e:\n", + " print(e)\n", + "\n", + "# Delete the endpoint using the Vertex fully qualified identifier for the endpoint\n", + "try:\n", + " if delete_endpoint and 'endpoint_id' in globals():\n", + " clients['endpoint'].delete_endpoint(name=endpoint_id)\n", + "except Exception as e:\n", + " print(e)\n", + "\n", + "# Delete the batch job using the Vertex fully qualified identifier for the batch job\n", + "try:\n", + " if delete_batchjob and 'batch_job_id' in globals():\n", + " clients['job'].delete_batch_prediction_job(name=batch_job_id)\n", + "except Exception as e:\n", + " print(e)\n", + "\n", + "# Delete the custom job using the Vertex fully qualified identifier for the custom job\n", + "try:\n", + " if delete_customjob and 'job_id' in globals():\n", + " clients['job'].delete_custom_job(name=job_id)\n", + "except Exception as e:\n", + " print(e)\n", + "\n", + "# Delete the hyperparameter tuning job using the Vertex fully qualified identifier for the hyperparameter tuning job\n", + "try:\n", + " if delete_hptjob and 'hpt_job_id' in globals():\n", + " clients['job'].delete_hyperparameter_tuning_job(name=hpt_job_id)\n", + "except Exception as e:\n", + " print(e)\n", + "\n", + "if delete_bucket and 'BUCKET_NAME' in globals():\n", + " ! gsutil rm -r $BUCKET_NAME" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [] + } + ], + "metadata": { + "environment": { + "name": "common-cu110.m71", + "type": "gcloud", + "uri": "gcr.io/deeplearning-platform-release/base-cu110:m71" + }, + "kernelspec": { + "display_name": "Python 3", + "language": "python", + "name": "python3" + }, + "language_info": { + "codemirror_mode": { + "name": "ipython", + "version": 3 + }, + "file_extension": ".py", + "mimetype": "text/x-python", + "name": "python", + "nbconvert_exporter": "python", + "pygments_lexer": "ipython3", + "version": "3.7.10" + }, + "vars": { + "AIP": "AI Platform", + "AUTOML": "AutoML", + "AUTOML_URL": "https://cloud.google.com/vertex-ai/docs/start/automl-users", + "BATCHJOB_WAIT": "60", + "BATCH_OR_ONLINE": "batch", + "BQ": "BigQuery", + "CLOUD_SDK": "Cloud SDK", + "COLAB": "Colab", + "DATASET_ALIAS": "cifar10", + "DATASET_NAME": "CIFAR10", + "DOCKER": "Docker", + "EDGE": "Edge", + "EMAIL": "cdpe@gmail.com", + "EXPLAIN_BATCH": ", 'generate_explanation': True", + "GAPIC": "client library", + "GAPICP": "client library for Python", + "GCONSOLE": "Cloud Console", + "GCP": "Google Cloud", + "GCS": "Cloud Storage", + "GNOTEBOOK": "Google Cloud Notebook", + "HEIGHT": "32", + "MODEL_TYPE": "image classification", + "NOTEBOOK": "showcase_custom_image_classification_batch_explain.ipynb", + "REGION": "us-central1", + "REPO": "ai-platform-unified/notebooks/community/gapic/custom", + "TENSORFLOW": "TensorFlow", + "TFLite": "TFLite", + "TFServing": "TF Serving", + "TFV": "2-1", + "TITLE": "Custom training image classification model for batch prediction with explanation", + "TRAINING": "training", + "TRAINING_TIME": "20", + "TRAINING_WAIT": "60", + "VERTEX_AI": "Vertex AI", + "WIDTH": "32", + "YEAR": "2020", + "null": "null", + "uCAIP": "Vertex" + } + }, + "nbformat": 4, + "nbformat_minor": 4 +} diff --git a/notebooks/official/explainable_ai/gapic-custom_image_classification_online_explain.ipynb b/notebooks/official/explainable_ai/gapic-custom_image_classification_online_explain.ipynb new file mode 100644 index 000000000..3ee0c9c36 --- /dev/null +++ b/notebooks/official/explainable_ai/gapic-custom_image_classification_online_explain.ipynb @@ -0,0 +1,2421 @@ +{ + "cells": [ + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "copyright" + }, + "outputs": [], + "source": [ + "# Copyright 2020 Google LLC\n", + "#\n", + "# Licensed under the Apache License, Version 2.0 (the \"License\");\n", + "# you may not use this file except in compliance with the License.\n", + "# You may obtain a copy of the License at\n", + "#\n", + "# https://www.apache.org/licenses/LICENSE-2.0\n", + "#\n", + "# Unless required by applicable law or agreed to in writing, software\n", + "# distributed under the License is distributed on an \"AS IS\" BASIS,\n", + "# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n", + "# See the License for the specific language governing permissions and\n", + "# limitations under the License." + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "title" + }, + "source": [ + "# Vertex client library for Python: Custom training image classification model for online prediction with explanation\n", + "\n", + "\n", + " \n", + " \n", + "
\n", + " \n", + " \"Colab Run in Colab\n", + " \n", + " \n", + " \n", + " \"GitHub\n", + " View on GitHub\n", + " \n", + "
\n", + "


" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "overview:custom,xai" + }, + "source": [ + "## Overview\n", + "\n", + "\n", + "This tutorial demonstrates how to use the Vertex client library for Python to train and deploy a custom image classification model for online prediction with explanation." + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "dataset:custom,cifar10,icn" + }, + "source": [ + "### Dataset\n", + "\n", + "The dataset used for this tutorial is the [CIFAR10 dataset](https://www.tensorflow.org/datasets/catalog/cifar10) from [TensorFlow Datasets](https://www.tensorflow.org/datasets/catalog/overview). The version of the dataset you will use is built into TensorFlow. The trained model predicts which type of class an image is from ten classes: airplane, automobile, bird, cat, deer, dog, frog, horse, ship, truck." + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "objective:custom,training,online_prediction,xai" + }, + "source": [ + "### Objective\n", + "\n", + "In this tutorial, you create a custom trained model from a Python script in a Google prebuilt Docker container using the Vertex client library for Python, and then do a prediction with explanations on the deployed model by sending data. Alternatively, you can create custom trained models using `gcloud` command-line tool or online using Cloud Console.\n", + "\n", + "The steps performed include:\n", + "\n", + "- Create a Vertex custom job for training a model.\n", + "- Train a TensorFlow model.\n", + "- Retrieve and load the model artifacts.\n", + "- View the model evaluation.\n", + "- Set explanation parameters.\n", + "- Upload the model as a Vertex `Model` resource.\n", + "- Deploy the `Model` resource to a serving `Endpoint` resource.\n", + "- Make a prediction with explanation.\n", + "- Undeploy the `Model` resource." + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "costs" + }, + "source": [ + "### Costs\n", + "\n", + "This tutorial uses billable components of Google Cloud (GCP):\n", + "\n", + "* Vertex AI\n", + "* Cloud Storage\n", + "\n", + "Learn about [Vertex AI\n", + "pricing](https://cloud.google.com/vertex-ai/pricing) and [Cloud Storage\n", + "pricing](https://cloud.google.com/storage/pricing), and use the [Pricing\n", + "Calculator](https://cloud.google.com/products/calculator/)\n", + "to generate a cost estimate based on your projected usage." + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "install_aip" + }, + "source": [ + "## Installation\n", + "\n", + "Install the latest version of Vertex client library for Python." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "install_aip" + }, + "outputs": [], + "source": [ + "import os\n", + "import sys\n", + "\n", + "# Google Cloud Notebook\n", + "if os.path.exists(\"/opt/deeplearning/metadata/env_version\"):\n", + " USER_FLAG = \"--user\"\n", + "else:\n", + " USER_FLAG = \"\"\n", + "\n", + "! pip3 install --upgrade google-cloud-aiplatform $USER_FLAG" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "install_storage" + }, + "source": [ + "Install the latest GA version of *google-cloud-storage* library as well." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "install_storage" + }, + "outputs": [], + "source": [ + "! pip3 install -U google-cloud-storage $USER_FLAG" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "install_tensorflow" + }, + "outputs": [], + "source": [ + "if os.environ[\"IS_TESTING\"]:\n", + " ! pip3 install --upgrade tensorflow $USER_FLAG" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "install_cv2" + }, + "outputs": [], + "source": [ + "if os.environ[\"IS_TESTING\"]:\n", + " ! apt-get update && apt-get install -y python3-opencv\n", + " ! pip3 install --upgrade opencv-python $USER_FLAG" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "install_matplotlib" + }, + "outputs": [], + "source": [ + "if os.environ[\"IS_TESTING\"]:\n", + " ! pip3 install --upgrade matplotlib $USER_FLAG" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "restart" + }, + "source": [ + "### Restart the kernel\n", + "\n", + "Once you've installed the Vertex client library for Python and *google-cloud-storage* library, you need to restart the notebook kernel so it can find the packages." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "restart" + }, + "outputs": [], + "source": [ + "if not os.getenv(\"IS_TESTING\"):\n", + " # Automatically restart kernel after installs\n", + " import IPython\n", + "\n", + " app = IPython.Application.instance()\n", + " app.kernel.do_shutdown(True)" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "before_you_begin" + }, + "source": [ + "## Before you begin\n", + "\n", + "### GPU runtime\n", + "\n", + "*Make sure you're running this notebook in a GPU runtime if you have that option. In Colab, select* **Runtime > Change Runtime Type > GPU**\n", + "\n", + "### Set up your Google Cloud project\n", + "\n", + "**The following steps are required, regardless of your notebook environment.**\n", + "\n", + "1. [Select or create a Google Cloud project](https://console.cloud.google.com/cloud-resource-manager). When you first create an account, you get a $300 free credit towards your compute/storage costs.\n", + "\n", + "2. [Make sure that billing is enabled for your project.](https://cloud.google.com/billing/docs/how-to/modify-project)\n", + "\n", + "3. [Enable the Vertex AI APIs and Compute Engine APIs.](https://console.cloud.google.com/flows/enableapi?apiid=ml.googleapis.com,compute_component)\n", + "\n", + "4. [The Google Cloud SDK](https://cloud.google.com/sdk) is already installed in Google Cloud Notebook.\n", + "\n", + "5. Enter your project ID in the cell below. Then run the cell to make sure the\n", + "Cloud SDK uses the right project for all the commands in this notebook.\n", + "\n", + "**Note**: Jupyter runs lines prefixed with `!` as shell commands, and it interpolates Python variables prefixed with `$`." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "set_project_id" + }, + "outputs": [], + "source": [ + "PROJECT_ID = \"[your-project-id]\" # @param {type:\"string\"}" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "autoset_project_id" + }, + "outputs": [], + "source": [ + "if PROJECT_ID == \"\" or PROJECT_ID is None or PROJECT_ID == \"[your-project-id]\":\n", + " # Get your GCP project id from gcloud\n", + " shell_output = !gcloud config list --format 'value(core.project)' 2>/dev/null\n", + " PROJECT_ID = shell_output[0]\n", + " print(\"Project ID:\", PROJECT_ID)" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "set_gcloud_project_id" + }, + "outputs": [], + "source": [ + "! gcloud config set project $PROJECT_ID" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "region" + }, + "source": [ + "#### Region\n", + "\n", + "You can also change the `REGION` variable, which is used for operations\n", + "throughout the rest of this notebook. Below are regions supported for Vertex. We recommend that you choose the region closest to you.\n", + "\n", + "- Americas: `us-central1`\n", + "- Europe: `europe-west4`\n", + "- Asia Pacific: `asia-east1`\n", + "\n", + "You may not use a multi-regional bucket for training with Vertex. Not all regions provide support for all Vertex services.\n", + "\n", + "Learn more about [Vertex regions](https://cloud.google.com/vertex-ai/docs/general/locations)" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "region" + }, + "outputs": [], + "source": [ + "REGION = \"us-central1\" # @param {type: \"string\"}" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "timestamp" + }, + "source": [ + "#### Timestamp\n", + "\n", + "If you are in a live tutorial session, you might be using a shared test account or project. To avoid name collisions between users on resources created, you create a timestamp for each instance session, and append the timestamp onto the name of resources you create in this tutorial." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "timestamp" + }, + "outputs": [], + "source": [ + "from datetime import datetime\n", + "\n", + "TIMESTAMP = datetime.now().strftime(\"%Y%m%d%H%M%S\")" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "gcp_authenticate" + }, + "source": [ + "### Authenticate your Google Cloud account\n", + "\n", + "**If you are using Google Cloud Notebook**, your environment is already authenticated. Skip this step.\n", + "\n", + "**If you are using Colab**, run the cell below and follow the instructions when prompted to authenticate your account via oAuth.\n", + "\n", + "**Otherwise**, follow these steps:\n", + "\n", + "In the Cloud Console, go to the [Create service account key](https://console.cloud.google.com/apis/credentials/serviceaccountkey) page.\n", + "\n", + "**Click Create service account**.\n", + "\n", + "In the **Service account name** field, enter a name, and click **Create**.\n", + "\n", + "In the **Grant this service account access to project** section, click the Role drop-down list. Type \"Vertex\" into the filter box, and select **Vertex AI Administrator**. Type \"Storage Object Admin\" into the filter box, and select **Storage Object Admin**.\n", + "\n", + "Click Create. A JSON file that contains your key downloads to your local environment.\n", + "\n", + "Enter the path to your service account key as the `GOOGLE_APPLICATION_CREDENTIALS` variable in the cell below and run the cell." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "gcp_authenticate" + }, + "outputs": [], + "source": [ + "# If you are running this notebook in Colab, run this cell and follow the\n", + "# instructions to authenticate your GCP account. This provides access to your\n", + "# Cloud Storage bucket and lets you submit training jobs and prediction\n", + "# requests.\n", + "\n", + "# If on Google Cloud Notebook, then don't execute this code\n", + "if not os.path.exists(\"/opt/deeplearning/metadata/env_version\"):\n", + " if \"google.colab\" in sys.modules:\n", + " from google.colab import auth as google_auth\n", + "\n", + " google_auth.authenticate_user()\n", + "\n", + " # If you are running this notebook locally, replace the string below with the\n", + " # path to your service account key and run this cell to authenticate your GCP\n", + " # account.\n", + " elif not os.getenv(\"IS_TESTING\"):\n", + " %env GOOGLE_APPLICATION_CREDENTIALS ''" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "bucket:custom" + }, + "source": [ + "### Create a Cloud Storage bucket\n", + "\n", + "**The following steps are required, regardless of your notebook environment.**\n", + "\n", + "When you submit a custom training job using the Vertex client library for Python, you upload a Python package\n", + "containing your training code to a Cloud Storage bucket. Vertex runs\n", + "the code from this package. In this tutorial, Vertex also saves the\n", + "trained model that results from your job in the same bucket. You can then\n", + "create an `Endpoint` resource based on this output in order to serve\n", + "online predictions.\n", + "\n", + "Set the name of your Cloud Storage bucket below. Bucket names must be globally unique across all Google Cloud projects, including those outside of your organization." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "bucket" + }, + "outputs": [], + "source": [ + "BUCKET_NAME = \"gs://[your-bucket-name]\" # @param {type:\"string\"}" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "autoset_bucket" + }, + "outputs": [], + "source": [ + "if BUCKET_NAME == \"\" or BUCKET_NAME is None or BUCKET_NAME == \"gs://[your-bucket-name]\":\n", + " BUCKET_NAME = \"gs://\" + PROJECT_ID + \"aip-\" + TIMESTAMP" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "create_bucket" + }, + "source": [ + "**Only if your bucket doesn't already exist**: Run the following cell to create your Cloud Storage bucket." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "create_bucket" + }, + "outputs": [], + "source": [ + "! gsutil mb -l $REGION $BUCKET_NAME" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "validate_bucket" + }, + "source": [ + "Finally, validate access to your Cloud Storage bucket by examining its contents:" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "validate_bucket" + }, + "outputs": [], + "source": [ + "! gsutil ls -al $BUCKET_NAME" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "setup_vars" + }, + "source": [ + "### Set up variables\n", + "\n", + "Next, set up some variables used throughout the tutorial.\n", + "### Import libraries and define constants" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "import_aip:v1beta1,protobuf" + }, + "source": [ + "#### Import Vertex client library for Python\n", + "\n", + "Import the Vertex client library for Python into your Python environment." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "import_aip:v1beta1,protobuf" + }, + "outputs": [], + "source": [ + "import time\n", + "\n", + "import google.cloud.aiplatform_v1beta1 as aip\n", + "from google.protobuf import json_format\n", + "from google.protobuf.json_format import MessageToJson, ParseDict\n", + "from google.protobuf.struct_pb2 import Struct, Value" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "aip_constants" + }, + "source": [ + "#### Vertex AI constants\n", + "\n", + "Setup up the following constants for Vertex AI:\n", + "\n", + "- `API_ENDPOINT`: The Vertex AI API service endpoint for dataset, model, job, pipeline and endpoint services.\n", + "- `PARENT`: The Vertex AI location root path for dataset, model, job, pipeline and endpoint resources." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "aip_constants" + }, + "outputs": [], + "source": [ + "# API service endpoint\n", + "API_ENDPOINT = \"{}-aiplatform.googleapis.com\".format(REGION)\n", + "\n", + "# Vertex location root path for your dataset, model and endpoint resources\n", + "PARENT = \"projects/\" + PROJECT_ID + \"/locations/\" + REGION" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "accelerators:training,prediction,cpu" + }, + "source": [ + "#### Set hardware accelerators\n", + "\n", + "You can set hardware accelerators for training and prediction.\n", + "\n", + "Set the variables `TRAIN_GPU/TRAIN_NGPU` and `DEPLOY_GPU/DEPLOY_NGPU` to use a container image supporting a GPU and the number of GPUs allocated to the virtual machine (VM) instance. For example, to use a GPU container image with 4 Nvidia Telsa K80 GPUs allocated to each VM, you would specify:\n", + "\n", + " (aip.AcceleratorType.NVIDIA_TESLA_K80, 4)\n", + "\n", + "\n", + "Otherwise specify `(None, None)` to use a container image to run on a CPU.\n", + "\n", + "Learn [which accelerators are available in your region](https://cloud.google.com/vertex-ai/docs/general/locations#accelerators).\n", + "\n", + "*Note*: TF releases before 2.3 for GPU support will fail to load the custom trained model in this tutorial. It is a known issue and fixed in TF 2.3 -- which is caused by static graph ops that are generated in the serving function. If you encounter this issue on your own custom trained models, use a container image for TF 2.3 with GPU support." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "accelerators:training,prediction,cpu" + }, + "outputs": [], + "source": [ + "if os.getenv(\"IS_TESTING_TRAIN_GPU\"):\n", + " TRAIN_GPU, TRAIN_NGPU = (\n", + " aip.AcceleratorType.NVIDIA_TESLA_K80,\n", + " int(os.getenv(\"IS_TESTING_TRAIN_GPU\")),\n", + " )\n", + "else:\n", + " TRAIN_GPU, TRAIN_NGPU = (aip.AcceleratorType.NVIDIA_TESLA_K80, 1)\n", + "\n", + "if os.getenv(\"IS_TESTING_DEPLOY_GPU\"):\n", + " DEPLOY_GPU, DEPLOY_NGPU = (\n", + " aip.AcceleratorType.NVIDIA_TESLA_K80,\n", + " int(os.getenv(\"IS_TESTING_DEPLOY_GPU\")),\n", + " )\n", + "else:\n", + " DEPLOY_GPU, DEPLOY_NGPU = (None, None)" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "container:training,prediction" + }, + "source": [ + "#### Set pre-built containers\n", + "\n", + "Set the pre-built Docker container image for training and prediction.\n", + "\n", + "\n", + "For the latest list, see [Pre-built containers for training](https://cloud.google.com/ai-platform-unified/docs/training/pre-built-containers).\n", + "\n", + "\n", + "For the latest list, see [Pre-built containers for prediction](https://cloud.google.com/ai-platform-unified/docs/predictions/pre-built-containers)" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "container:training,prediction" + }, + "outputs": [], + "source": [ + "if os.getenv(\"IS_TESTING_TF\"):\n", + " TF = os.getenv(\"IS_TESTING_TF\")\n", + "else:\n", + " TF = \"2-1\"\n", + "\n", + "if TF[0] == \"2\":\n", + " if TRAIN_GPU:\n", + " TRAIN_VERSION = \"tf-gpu.{}\".format(TF)\n", + " else:\n", + " TRAIN_VERSION = \"tf-cpu.{}\".format(TF)\n", + " if DEPLOY_GPU:\n", + " DEPLOY_VERSION = \"tf2-gpu.{}\".format(TF)\n", + " else:\n", + " DEPLOY_VERSION = \"tf2-cpu.{}\".format(TF)\n", + "else:\n", + " if TRAIN_GPU:\n", + " TRAIN_VERSION = \"tf-gpu.{}\".format(TF)\n", + " else:\n", + " TRAIN_VERSION = \"tf-cpu.{}\".format(TF)\n", + " if DEPLOY_GPU:\n", + " DEPLOY_VERSION = \"tf-gpu.{}\".format(TF)\n", + " else:\n", + " DEPLOY_VERSION = \"tf-cpu.{}\".format(TF)\n", + "\n", + "TRAIN_IMAGE = \"gcr.io/cloud-aiplatform/training/{}:latest\".format(TRAIN_VERSION)\n", + "DEPLOY_IMAGE = \"gcr.io/cloud-aiplatform/prediction/{}:latest\".format(DEPLOY_VERSION)\n", + "\n", + "print(\"Training:\", TRAIN_IMAGE, TRAIN_GPU, TRAIN_NGPU)\n", + "print(\"Deployment:\", DEPLOY_IMAGE, DEPLOY_GPU, DEPLOY_NGPU)" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "machine:training,prediction" + }, + "source": [ + "#### Set machine type\n", + "\n", + "Next, set the machine type to use for training and prediction.\n", + "\n", + "- Set the variables `TRAIN_COMPUTE` and `DEPLOY_COMPUTE` to configure the compute resources for the VMs you will use for for training and prediction.\n", + " - `machine type`\n", + " - `n1-standard`: 3.75GB of memory per vCPU.\n", + " - `n1-highmem`: 6.5GB of memory per vCPU\n", + " - `n1-highcpu`: 0.9 GB of memory per vCPU\n", + " - `vCPUs`: number of \\[2, 4, 8, 16, 32, 64, 96 \\]\n", + "\n", + "*Note: The following is not supported for training:*\n", + "\n", + " - `standard`: 2 vCPUs\n", + " - `highcpu`: 2, 4 and 8 vCPUs\n", + "\n", + "*Note: You may also use n2 and e2 machine types for training and deployment, but they do not support GPUs*.\n", + "\n", + "Learn [which machine types are available for training](https://cloud.google.com/vertex-ai/docs/training/configure-compute) and [for prediction](https://cloud.google.com/vertex-ai/docs/predictions/configure-compute)" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "machine:training,prediction" + }, + "outputs": [], + "source": [ + "if os.getenv(\"IS_TESTING_TRAIN_MACHINE\"):\n", + " MACHINE_TYPE = os.getenv(\"IS_TESTING_TRAIN_MACHINE\")\n", + "else:\n", + " MACHINE_TYPE = \"n1-standard\"\n", + "\n", + "VCPU = \"4\"\n", + "TRAIN_COMPUTE = MACHINE_TYPE + \"-\" + VCPU\n", + "print(\"Train machine type\", TRAIN_COMPUTE)\n", + "\n", + "if os.getenv(\"IS_TESTING_DEPLOY_MACHINE\"):\n", + " MACHINE_TYPE = os.getenv(\"IS_TESTING_DEPLOY_MACHINE\")\n", + "else:\n", + " MACHINE_TYPE = \"n1-standard\"\n", + "\n", + "VCPU = \"4\"\n", + "DEPLOY_COMPUTE = MACHINE_TYPE + \"-\" + VCPU\n", + "print(\"Deploy machine type\", DEPLOY_COMPUTE)" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "tutorial_start:custom" + }, + "source": [ + "# Tutorial\n", + "\n", + "Now you are ready to start creating your own custom model and training for CIFAR10." + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "clients:custom" + }, + "source": [ + "## Set up clients\n", + "\n", + "The Vertex client library for Python works as a client/server model. On your side (the Python script) you will create a client that sends requests and receives responses from the Vertex server.\n", + "\n", + "You will use different clients in this tutorial for different steps in the workflow. So set them all up upfront.\n", + "\n", + "- Model Service for `Model` resources.\n", + "- Endpoint Service for deployment.\n", + "- Job Service for batch jobs and custom training.\n", + "- Prediction Service for serving." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "clients:custom" + }, + "outputs": [], + "source": [ + "# client options same for all services\n", + "client_options = {\"api_endpoint\": API_ENDPOINT}\n", + "\n", + "\n", + "def create_job_client():\n", + " client = aip.JobServiceClient(client_options=client_options)\n", + " return client\n", + "\n", + "\n", + "def create_model_client():\n", + " client = aip.ModelServiceClient(client_options=client_options)\n", + " return client\n", + "\n", + "\n", + "def create_endpoint_client():\n", + " client = aip.EndpointServiceClient(client_options=client_options)\n", + " return client\n", + "\n", + "\n", + "def create_prediction_client():\n", + " client = aip.PredictionServiceClient(client_options=client_options)\n", + " return client\n", + "\n", + "\n", + "clients = {}\n", + "clients[\"job\"] = create_job_client()\n", + "clients[\"model\"] = create_model_client()\n", + "clients[\"endpoint\"] = create_endpoint_client()\n", + "clients[\"prediction\"] = create_prediction_client()\n", + "\n", + "for client in clients.items():\n", + " print(client)" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "train_custom_model" + }, + "source": [ + "## Train a model\n", + "\n", + "There are two ways you can train a custom model using a container image:\n", + "\n", + "- **Use a Google Cloud prebuilt container**. If you use a prebuilt container, you will additionally specify a Python package to install into the container image. This Python package contains your code for training a custom model.\n", + "\n", + "- **Use your own custom container image**. If you use your own container, the container needs to contain your code for training a custom model." + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "train_custom_job_specification:prebuilt_container" + }, + "source": [ + "## Prepare your custom job specification\n", + "\n", + "Now that your clients are ready, your first step is to create a Job Specification for your custom training job. The job specification will consist of the following:\n", + "\n", + "- `worker_pool_spec` : The specification of the type of machine(s) you will use for training and how many (single or distributed)\n", + "- `python_package_spec` : The specification of the Python package to be installed with the pre-built container." + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "train_custom_job_machine_specification" + }, + "source": [ + "### Prepare your machine specification\n", + "\n", + "Now define the machine specification for your custom training job. This tells Vertex AI what type of machine instance to provision for the training.\n", + " - `machine_type`: The type of GCP instance to provision -- e.g., n1-standard-8.\n", + " - `accelerator_type`: The type, if any, of hardware accelerator. In this tutorial if you previously set the variable `TRAIN_GPU != None`, you are using a GPU; otherwise you will use a CPU.\n", + " - `accelerator_count`: The number of accelerators." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "train_custom_job_machine_specification" + }, + "outputs": [], + "source": [ + "if TRAIN_GPU:\n", + " machine_spec = {\n", + " \"machine_type\": TRAIN_COMPUTE,\n", + " \"accelerator_type\": TRAIN_GPU,\n", + " \"accelerator_count\": TRAIN_NGPU,\n", + " }\n", + "else:\n", + " machine_spec = {\"machine_type\": TRAIN_COMPUTE, \"accelerator_count\": 0}" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "train_custom_job_disk_specification" + }, + "outputs": [], + "source": [ + "DISK_TYPE = \"pd-ssd\" # [ pd-ssd, pd-standard]\n", + "DISK_SIZE = 200 # GB\n", + "\n", + "disk_spec = {\"boot_disk_type\": DISK_TYPE, \"boot_disk_size_gb\": DISK_SIZE}" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "train_custom_job_worker_pool_specification:prebuilt_container" + }, + "source": [ + "### Define the worker pool specification\n", + "\n", + "Next, you define the worker pool specification for your custom training job. The worker pool specification will consist of the following:\n", + "\n", + "- `replica_count`: The number of instances to provision of this machine type.\n", + "- `machine_spec`: The hardware specification.\n", + "- `disk_spec` : (optional) The disk storage specification.\n", + "\n", + "- `python_package`: The Python training package to install on the VM instance(s) and which Python module to invoke, along with command line arguments for the Python module.\n", + "\n", + "Let's dive deeper now into the python package specification:\n", + "\n", + "-`executor_image_spec`: This is the docker image which is configured for your custom training job.\n", + "\n", + "-`package_uris`: This is a list of the locations (URIs) of your python training packages to install on the provisioned instance. The locations need to be in a Cloud Storage bucket. These can be either individual python files or a zip (archive) of an entire package. In the later case, the job service will unzip (unarchive) the contents into the docker image.\n", + "\n", + "-`python_module`: The Python module (script) to invoke for running the custom training job. In this example, you will be invoking `trainer.task.py` -- note that it was not neccessary to append the `.py` suffix.\n", + "\n", + "-`args`: The command line arguments to pass to the corresponding Pythom module. In this example, you will be setting:\n", + " - `\"--model-dir=\" + MODEL_DIR` : The Cloud Storage location where to store the model artifacts. There are two ways to tell the training script where to save the model artifacts:\n", + " - direct: You pass the Cloud Storage location as a command line argument to your training script (set variable `DIRECT = True`), or\n", + " - indirect: The service passes the Cloud Storage location as the environment variable `AIP_MODEL_DIR` to your training script (set variable `DIRECT = False`). In this case, you tell the service the model artifact location in the job specification.\n", + " - `\"--epochs=\" + EPOCHS`: The number of epochs for training.\n", + " - `\"--steps=\" + STEPS`: The number of steps (batches) per epoch.\n", + " - `\"--distribute=\" + TRAIN_STRATEGY\"` : The training distribution strategy to use for single or distributed training.\n", + " - `\"single\"`: single device.\n", + " - `\"mirror\"`: all GPU devices on a single compute instance.\n", + " - `\"multi\"`: all GPU devices on all compute instances." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "train_custom_job_worker_pool_specification:prebuilt_container" + }, + "outputs": [], + "source": [ + "JOB_NAME = \"custom_job_\" + TIMESTAMP\n", + "MODEL_DIR = \"{}/{}\".format(BUCKET_NAME, JOB_NAME)\n", + "\n", + "if not TRAIN_NGPU or TRAIN_NGPU < 2:\n", + " TRAIN_STRATEGY = \"single\"\n", + "else:\n", + " TRAIN_STRATEGY = \"mirror\"\n", + "\n", + "EPOCHS = 20\n", + "STEPS = 100\n", + "\n", + "DIRECT = True\n", + "if DIRECT:\n", + " CMDARGS = [\n", + " \"--model-dir=\" + MODEL_DIR,\n", + " \"--epochs=\" + str(EPOCHS),\n", + " \"--steps=\" + str(STEPS),\n", + " \"--distribute=\" + TRAIN_STRATEGY,\n", + " ]\n", + "else:\n", + " CMDARGS = [\n", + " \"--epochs=\" + str(EPOCHS),\n", + " \"--steps=\" + str(STEPS),\n", + " \"--distribute=\" + TRAIN_STRATEGY,\n", + " ]\n", + "\n", + "worker_pool_spec = [\n", + " {\n", + " \"replica_count\": 1,\n", + " \"machine_spec\": machine_spec,\n", + " \"disk_spec\": disk_spec,\n", + " \"python_package_spec\": {\n", + " \"executor_image_uri\": TRAIN_IMAGE,\n", + " \"package_uris\": [BUCKET_NAME + \"/trainer_cifar10.tar.gz\"],\n", + " \"python_module\": \"trainer.task\",\n", + " \"args\": CMDARGS,\n", + " },\n", + " }\n", + "]" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "assemble_custom_job_specification" + }, + "source": [ + "### Assemble a job specification\n", + "\n", + "Now assemble the complete description for the custom job specification:\n", + "\n", + "- `display_name`: The human readable name you assign to this custom job.\n", + "- `job_spec`: The specification for the custom job.\n", + " - `worker_pool_specs`: The specification for the machine VM instances.\n", + " - `base_output_directory`: This tells the service the Cloud Storage location where to save the model artifacts (when variable `DIRECT = False`). The service will then pass the location to the training script as the environment variable `AIP_MODEL_DIR`, and the path will be of the form:\n", + "\n", + " /model" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "assemble_custom_job_specification" + }, + "outputs": [], + "source": [ + "if DIRECT:\n", + " job_spec = {\"worker_pool_specs\": worker_pool_spec}\n", + "else:\n", + " job_spec = {\n", + " \"worker_pool_specs\": worker_pool_spec,\n", + " \"base_output_directory\": {\"output_uri_prefix\": MODEL_DIR},\n", + " }\n", + "\n", + "custom_job = {\"display_name\": JOB_NAME, \"job_spec\": job_spec}" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "examine_training_package" + }, + "source": [ + "### Examine the training package\n", + "\n", + "#### Package layout\n", + "\n", + "Before you start the training, you will look at how a Python package is assembled for a custom training job. When unarchived, the package contains the following directory/file layout.\n", + "\n", + "- PKG-INFO\n", + "- README.md\n", + "- setup.cfg\n", + "- setup.py\n", + "- trainer\n", + " - \\_\\_init\\_\\_.py\n", + " - task.py\n", + "\n", + "The files `setup.cfg` and `setup.py` are the instructions for installing the package into the operating environment of the Docker image.\n", + "\n", + "The file `trainer/task.py` is the Python script for executing the custom training job. *Note*, when we referred to it in the worker pool specification, we replace the directory slash with a dot (`trainer.task`) and dropped the file suffix (`.py`).\n", + "\n", + "#### Package Assembly\n", + "\n", + "In the following cells, you will assemble the training package." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "examine_training_package" + }, + "outputs": [], + "source": [ + "# Make folder for Python training script\n", + "! rm -rf custom\n", + "! mkdir custom\n", + "\n", + "# Add package information\n", + "! touch custom/README.md\n", + "\n", + "setup_cfg = \"[egg_info]\\n\\ntag_build =\\n\\ntag_date = 0\"\n", + "! echo \"$setup_cfg\" > custom/setup.cfg\n", + "\n", + "setup_py = \"import setuptools\\n\\nsetuptools.setup(\\n\\n install_requires=[\\n\\n 'tensorflow_datasets==1.3.0',\\n\\n ],\\n\\n packages=setuptools.find_packages())\"\n", + "! echo \"$setup_py\" > custom/setup.py\n", + "\n", + "pkg_info = \"Metadata-Version: 1.0\\n\\nName: CIFAR10 image classification\\n\\nVersion: 0.0.0\\n\\nSummary: Demostration training script\\n\\nHome-page: www.google.com\\n\\nAuthor: Google\\n\\nAuthor-email: aferlitsch@google.com\\n\\nLicense: Public\\n\\nDescription: Demo\\n\\nPlatform: Vertex\"\n", + "! echo \"$pkg_info\" > custom/PKG-INFO\n", + "\n", + "# Make the training subfolder\n", + "! mkdir custom/trainer\n", + "! touch custom/trainer/__init__.py" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "taskpy_contents:cifar10" + }, + "source": [ + "#### Task.py contents\n", + "\n", + "In the next cell, you write the contents of the training script task.py. We won't go into detail, it's just there for you to browse. In summary:\n", + "\n", + "- Get the directory where to save the model artifacts from the command line (`--model_dir`), and if not specified, then from the environment variable `AIP_MODEL_DIR`.\n", + "- Loads CIFAR10 dataset from TF Datasets (tfds).\n", + "- Builds a model using TF.Keras model API.\n", + "- Compiles the model (`compile()`).\n", + "- Sets a training distribution strategy according to the argument `args.distribute`.\n", + "- Trains the model (`fit()`) with epochs and steps according to the arguments `args.epochs` and `args.steps`\n", + "- Saves the trained model (`save(args.model_dir)`) to the specified model directory." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "taskpy_contents:cifar10" + }, + "outputs": [], + "source": [ + "%%writefile custom/trainer/task.py\n", + "# Single, Mirror and Multi-Machine Distributed Training for CIFAR-10\n", + "\n", + "import tensorflow_datasets as tfds\n", + "import tensorflow as tf\n", + "from tensorflow.python.client import device_lib\n", + "import argparse\n", + "import os\n", + "import sys\n", + "tfds.disable_progress_bar()\n", + "\n", + "parser = argparse.ArgumentParser()\n", + "parser.add_argument('--model-dir', dest='model_dir',\n", + " default=os.getenv(\"AIP_MODEL_DIR\"), type=str, help='Model dir.')\n", + "parser.add_argument('--lr', dest='lr',\n", + " default=0.01, type=float,\n", + " help='Learning rate.')\n", + "parser.add_argument('--epochs', dest='epochs',\n", + " default=10, type=int,\n", + " help='Number of epochs.')\n", + "parser.add_argument('--steps', dest='steps',\n", + " default=200, type=int,\n", + " help='Number of steps per epoch.')\n", + "parser.add_argument('--distribute', dest='distribute', type=str, default='single',\n", + " help='distributed training strategy')\n", + "args = parser.parse_args()\n", + "\n", + "print('Python Version = {}'.format(sys.version))\n", + "print('TensorFlow Version = {}'.format(tf.__version__))\n", + "print('TF_CONFIG = {}'.format(os.environ.get('TF_CONFIG', 'Not found')))\n", + "print('DEVICES', device_lib.list_local_devices())\n", + "\n", + "# Single Machine, single compute device\n", + "if args.distribute == 'single':\n", + " if tf.test.is_gpu_available():\n", + " strategy = tf.distribute.OneDeviceStrategy(device=\"/gpu:0\")\n", + " else:\n", + " strategy = tf.distribute.OneDeviceStrategy(device=\"/cpu:0\")\n", + "# Single Machine, multiple compute device\n", + "elif args.distribute == 'mirror':\n", + " strategy = tf.distribute.MirroredStrategy()\n", + "# Multiple Machine, multiple compute device\n", + "elif args.distribute == 'multi':\n", + " strategy = tf.distribute.experimental.MultiWorkerMirroredStrategy()\n", + "\n", + "# Multi-worker configuration\n", + "print('num_replicas_in_sync = {}'.format(strategy.num_replicas_in_sync))\n", + "\n", + "# Preparing dataset\n", + "BUFFER_SIZE = 10000\n", + "BATCH_SIZE = 64\n", + "\n", + "\n", + "def make_datasets_unbatched():\n", + "\n", + " # Scaling CIFAR10 data from (0, 255] to (0., 1.]\n", + " def scale(image, label):\n", + " image = tf.cast(image, tf.float32)\n", + " image /= 255.0\n", + " return image, label\n", + "\n", + "\n", + " datasets, info = tfds.load(name='cifar10',\n", + " with_info=True,\n", + " as_supervised=True)\n", + " return datasets['train'].map(scale).cache().shuffle(BUFFER_SIZE).repeat()\n", + "\n", + "\n", + "# Build the Keras model\n", + "def build_and_compile_cnn_model():\n", + " model = tf.keras.Sequential([\n", + " tf.keras.layers.Conv2D(32, 3, activation='relu', input_shape=(32, 32, 3)),\n", + " tf.keras.layers.MaxPooling2D(),\n", + " tf.keras.layers.Conv2D(32, 3, activation='relu'),\n", + " tf.keras.layers.MaxPooling2D(),\n", + " tf.keras.layers.Flatten(),\n", + " tf.keras.layers.Dense(10, activation='softmax')\n", + " ])\n", + " model.compile(\n", + " loss=tf.keras.losses.sparse_categorical_crossentropy,\n", + " optimizer=tf.keras.optimizers.SGD(learning_rate=args.lr),\n", + " metrics=['accuracy'])\n", + " return model\n", + "\n", + "\n", + "# Train the model\n", + "NUM_WORKERS = strategy.num_replicas_in_sync\n", + "# Here the batch size scales up by number of workers since\n", + "# `tf.data.Dataset.batch` expects the global batch size.\n", + "GLOBAL_BATCH_SIZE = BATCH_SIZE * NUM_WORKERS\n", + "train_dataset = make_datasets_unbatched().batch(GLOBAL_BATCH_SIZE)\n", + "\n", + "with strategy.scope():\n", + " # Creation of dataset, and model building/compiling need to be within\n", + " # `strategy.scope()`.\n", + " model = build_and_compile_cnn_model()\n", + "\n", + "model.fit(x=train_dataset, epochs=args.epochs, steps_per_epoch=args.steps)\n", + "model.save(args.model_dir)" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "tarball_training_script" + }, + "source": [ + "#### Store training script on your Cloud Storage bucket\n", + "\n", + "Next, you package the training folder into a compressed tar ball, and then store it in your Cloud Storage bucket." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "tarball_training_script" + }, + "outputs": [], + "source": [ + "! rm -f custom.tar custom.tar.gz\n", + "! tar cvf custom.tar custom\n", + "! gzip custom.tar\n", + "! gsutil cp custom.tar.gz $BUCKET_NAME/trainer_cifar10.tar.gz" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "train_custom_job" + }, + "source": [ + "### Train the model\n", + "\n", + "\n", + "Now start the training of your custom training job on Vertex. Use this helper function `create_custom_job`, which takes the following parameter:\n", + "\n", + "-`custom_job`: The specification for the custom job.\n", + "\n", + "The helper function calls job client service's `create_custom_job` method, with the following parameters:\n", + "\n", + "-`parent`: The Vertex location path to `Dataset`, `Model` and `Endpoint` resources.\n", + "-`custom_job`: The specification for the custom job.\n", + "\n", + "You will display a handful of the fields returned in `response` object, with the two that are of most interest are:\n", + "\n", + "-`response.name`: The Vertex fully qualified identifier assigned to this custom training job. You save this identifier for using in subsequent steps.\n", + "\n", + "-`response.state`: The current state of the custom training job." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "train_custom_job" + }, + "outputs": [], + "source": [ + "def create_custom_job(custom_job):\n", + " response = clients[\"job\"].create_custom_job(parent=PARENT, custom_job=custom_job)\n", + " print(\"name:\", response.name)\n", + " print(\"display_name:\", response.display_name)\n", + " print(\"state:\", response.state)\n", + " print(\"create_time:\", response.create_time)\n", + " print(\"update_time:\", response.update_time)\n", + " return response\n", + "\n", + "\n", + "response = create_custom_job(custom_job)" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "job_id:response" + }, + "source": [ + "Now get the unique identifier for the custom job you created." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "job_id:response" + }, + "outputs": [], + "source": [ + "# The full unique ID for the custom job\n", + "job_id = response.name\n", + "# The short numeric ID for the custom job\n", + "job_short_id = job_id.split(\"/\")[-1]\n", + "\n", + "print(job_id)" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "get_custom_job" + }, + "source": [ + "### Get information on a custom job\n", + "\n", + "Next, use this helper function `get_custom_job`, which takes the following parameter:\n", + "\n", + "- `name`: The Vertex fully qualified identifier for the custom job.\n", + "\n", + "The helper function calls the job client service's `get_custom_job` method, with the following parameter:\n", + "\n", + "- `name`: The Vertex fully qualified identifier for the custom job.\n", + "\n", + "If you recall, you got the Vertex fully qualified identifier for the custom job in the `response.name` field when you called the `create_custom_job` method, and saved the identifier in the variable `job_id`." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "get_custom_job" + }, + "outputs": [], + "source": [ + "def get_custom_job(name, silent=False):\n", + " response = clients[\"job\"].get_custom_job(name=name)\n", + " if silent:\n", + " return response\n", + "\n", + " print(\"name:\", response.name)\n", + " print(\"display_name:\", response.display_name)\n", + " print(\"state:\", response.state)\n", + " print(\"create_time:\", response.create_time)\n", + " print(\"update_time:\", response.update_time)\n", + " return response\n", + "\n", + "\n", + "response = get_custom_job(job_id)" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "wait_training_complete:custom" + }, + "source": [ + "# Deploy the model\n", + "\n", + "Training the above model may take upwards of 20 minutes time.\n", + "\n", + "Once your model is done training, you can calculate the actual time it took to train the model by subtracting `end_time` from `start_time`. For your model, we will need to know the location of the saved model, which the Python script saved in your local Cloud Storage bucket at `MODEL_DIR + '/saved_model.pb'`." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "wait_training_complete:custom" + }, + "outputs": [], + "source": [ + "while True:\n", + " response = get_custom_job(job_id, True)\n", + " if response.state != aip.JobState.JOB_STATE_SUCCEEDED:\n", + " print(\"Training job has not completed:\", response.state)\n", + " model_path_to_deploy = None\n", + " if response.state == aip.JobState.JOB_STATE_FAILED:\n", + " break\n", + " else:\n", + " if not DIRECT:\n", + " MODEL_DIR = MODEL_DIR + \"/model\"\n", + " model_path_to_deploy = MODEL_DIR\n", + " print(\"Training Job Time:\", response.end_time - response.start_time)\n", + " print(\"Training Elapsed Time:\", response.update_time - response.create_time)\n", + " break\n", + " time.sleep(60)\n", + "\n", + "print(\"model_to_deploy:\", model_path_to_deploy)" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "load_saved_model" + }, + "source": [ + "## Load the saved model\n", + "\n", + "Your model is stored in a TensorFlow SavedModel format in a Cloud Storage bucket. Now load it from the Cloud Storage bucket, and then you can do some things, like evaluate the model, and do a prediction.\n", + "\n", + "To load, you use the TF.Keras `model.load_model()` method passing it the Cloud Storage path where the model is saved -- specified by `MODEL_DIR`." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "load_saved_model" + }, + "outputs": [], + "source": [ + "import tensorflow as tf\n", + "\n", + "model = tf.keras.models.load_model(MODEL_DIR)" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "evaluate_custom_model:image" + }, + "source": [ + "## Evaluate the model\n", + "\n", + "Now find out how good the model is.\n", + "\n", + "### Load evaluation data\n", + "\n", + "You will load the CIFAR10 test (holdout) data from `tf.keras.datasets`, using the method `load_data()`. This returns the dataset as a tuple of two elements. The first element is the training data and the second is the test data. Each element is also a tuple of two elements: the image data, and the corresponding labels.\n", + "\n", + "You don't need the training data, and hence why we loaded it as `(_, _)`.\n", + "\n", + "Before you can run the data through evaluation, you need to preprocess it:\n", + "\n", + "`x_test`:\n", + "1. Normalize (rescale) the pixel data by dividing each pixel by 255. This replaces each single byte integer pixel with a 32-bit floating point number between 0 and 1.\n", + "\n", + "`y_test`:
\n", + "2. The labels are currently scalar (sparse). If you look back at the `compile()` step in the `trainer/task.py` script, you will find that it was compiled for sparse labels. So we don't need to do anything more." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "evaluate_custom_model:image,cifar10" + }, + "outputs": [], + "source": [ + "import numpy as np\n", + "from tensorflow.keras.datasets import cifar10\n", + "\n", + "(_, _), (x_test, y_test) = cifar10.load_data()\n", + "x_test = (x_test / 255.0).astype(np.float32)\n", + "\n", + "print(x_test.shape, y_test.shape)" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "perform_evaluation_custom" + }, + "source": [ + "### Perform the model evaluation\n", + "\n", + "Now evaluate how well the model in the custom job did." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "perform_evaluation_custom" + }, + "outputs": [], + "source": [ + "model.evaluate(x_test, y_test)" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "how_serving_function_works" + }, + "source": [ + "## Upload the model for serving\n", + "\n", + "Next, you will upload your TF.Keras model from the custom job to Vertex `Model` service, which will create a Vertex `Model` resource for your custom model. During upload, you need to define a serving function to convert data to the format your model expects. If you send encoded data to Vertex, your serving function ensures that the data is decoded on the model server before it is passed as input to your model.\n", + "\n", + "### How does the serving function work\n", + "\n", + "When you send a request to an online prediction server, the request is received by a HTTP server. The HTTP server extracts the prediction request from the HTTP request content body. The extracted prediction request is forwarded to the serving function. For Google pre-built prediction containers, the request content is passed to the serving function as a `tf.string`.\n", + "\n", + "The serving function consists of two parts:\n", + "\n", + "- `preprocessing function`:\n", + " - Converts the input (`tf.string`) to the input shape and data type of the underlying model (dynamic graph).\n", + " - Performs the same preprocessing of the data that was done during training the underlying model, including normalizing and scalingh.\n", + "- `post-processing function`:\n", + " - Converts the model output to format expected by the receiving application -- e.q., compresses the output.\n", + " - Packages the output for the the receiving application, including adding headings, and making JSON objects.\n", + "\n", + "Both the preprocessing and post-processing functions are converted to static graphs which are fused to the model. The output from the underlying model is passed to the post-processing function. The post-processing function passes the converted/packaged output back to the HTTP server. The HTTP server returns the output as the HTTP response content.\n", + "\n", + "One consideration you need to consider when building serving functions for TF.Keras models is that they run as static graphs. That means, you cannot use TF graph operations that require a dynamic graph. If you do, you will get an error during the compile of the serving function which will indicate that you are using an EagerTensor which is not supported." + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "serving_function_image:xai" + }, + "source": [ + "### Serving function for image data\n", + "\n", + "To pass images to the prediction service, you encode the compressed (e.g., JPEG) image bytes into base 64 -- which makes the content safe from modification while transmitting binary data over the network. Since this deployed model expects input data as raw (uncompressed) bytes, you need to ensure that the base 64 encoded data gets converted back to raw bytes before it is passed as input to the deployed model.\n", + "\n", + "To resolve this, define a serving function (`serving_fn`) and attach it to the model as a preprocessing step. Add a `@tf.function` decorator so the serving function is fused to the underlying model (instead of upstream on a CPU).\n", + "\n", + "When you send a prediction or explanation request, the content of the request is base 64 decoded into a TensorFlow string (`tf.string`), which is passed to the serving function (`serving_fn`). The serving function preprocesses the `tf.string` into raw (uncompressed) numpy bytes (`preprocess_fn`) to match the input requirements of the model:\n", + "- `io.decode_jpeg`- Decompresses the JPG image which is returned as a Tensorflow tensor with three channels (RGB).\n", + "- `image.convert_image_dtype` - Changes integer pixel values to float 32.\n", + "- `image.resize` - Resizes the image to match the input shape for the model.\n", + "- `resized / 255.0` - Rescales (normalization) the pixel data between 0 and 1.\n", + "\n", + "At this point, the data can be passed to the model (`m_call`).\n", + "\n", + "#### Explainable AI Signatures\n", + "\n", + "When the serving function is saved back with the underlying model (`tf.saved_model.save`), you specify the input layer of the serving function as the signature `serving_default`.\n", + "\n", + "For Explainable AI image models, you need to save two additional signatures from the serving function:\n", + "\n", + "- `xai_preprocess`: The preprocessing function in the serving function.\n", + "- `xai_model`: The concrete function for calling the model." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "serving_function_image:xai" + }, + "outputs": [], + "source": [ + "CONCRETE_INPUT = \"numpy_inputs\"\n", + "\n", + "\n", + "def _preprocess(bytes_input):\n", + " decoded = tf.io.decode_jpeg(bytes_input, channels=3)\n", + " decoded = tf.image.convert_image_dtype(decoded, tf.float32)\n", + " resized = tf.image.resize(decoded, size=(32, 32))\n", + " rescale = tf.cast(resized / 255.0, tf.float32)\n", + " return rescale\n", + "\n", + "\n", + "@tf.function(input_signature=[tf.TensorSpec([None], tf.string)])\n", + "def preprocess_fn(bytes_inputs):\n", + " decoded_images = tf.map_fn(\n", + " _preprocess, bytes_inputs, dtype=tf.float32, back_prop=False\n", + " )\n", + " return {\n", + " CONCRETE_INPUT: decoded_images\n", + " } # User needs to make sure the key matches model's input\n", + "\n", + "\n", + "@tf.function(input_signature=[tf.TensorSpec([None], tf.string)])\n", + "def serving_fn(bytes_inputs):\n", + " images = preprocess_fn(bytes_inputs)\n", + " prob = m_call(**images)\n", + " return prob\n", + "\n", + "\n", + "m_call = tf.function(model.call).get_concrete_function(\n", + " [tf.TensorSpec(shape=[None, 32, 32, 3], dtype=tf.float32, name=CONCRETE_INPUT)]\n", + ")\n", + "\n", + "tf.saved_model.save(\n", + " model,\n", + " model_path_to_deploy,\n", + " signatures={\n", + " \"serving_default\": serving_fn,\n", + " # Required for XAI\n", + " \"xai_preprocess\": preprocess_fn,\n", + " \"xai_model\": m_call,\n", + " },\n", + ")" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "serving_function_signature:xai" + }, + "source": [ + "## Get the serving function signature\n", + "\n", + "You can get the signatures of your model's input and output layers by reloading the model into memory, and querying it for the signatures corresponding to each layer.\n", + "\n", + "When making a prediction request, you need to route the request to the serving function instead of the model, so you need to know the input layer name of the serving function -- which you will use later when you make a prediction request.\n", + "\n", + "You also need to know the name of the serving function's input and output layer for constructing the explanation metadata -- which is discussed subsequently." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "serving_function_signature:xai" + }, + "outputs": [], + "source": [ + "loaded = tf.saved_model.load(model_path_to_deploy)\n", + "\n", + "serving_input = list(\n", + " loaded.signatures[\"serving_default\"].structured_input_signature[1].keys()\n", + ")[0]\n", + "print(\"Serving function input:\", serving_input)\n", + "serving_output = list(loaded.signatures[\"serving_default\"].structured_outputs.keys())[0]\n", + "print(\"Serving function output:\", serving_output)\n", + "\n", + "input_name = model.input.name\n", + "print(\"Model input name:\", input_name)\n", + "output_name = model.output.name\n", + "print(\"Model output name:\", output_name)" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "explanation_spec:ig" + }, + "source": [ + "### Explanation Specification\n", + "\n", + "To get explanations when doing a prediction, you must enable the explanation capability and set corresponding settings when you upload your custom model to an Vertex `Model` resource. These settings are referred to as the explanation metadata, which consists of:\n", + "\n", + "- `parameters`: This is the specification for the explainability algorithm to use for explanations on your model.\n", + "- `metadata`: This is the specification for how the algoithm is applied on your custom model.\n", + "\n", + "See [Compare Methods](https://cloud.google.com/vertex-ai/docs/explainable-ai/overview#compare-method) for a comparision of supported algoritmes.\n", + "\n", + "In this tutorial, you use the `integrated gradients` algorithm, which has the following parameter:\n", + "\n", + "`step_count`: This is the number of steps to approximate the remaining sum. The more steps, the more accurate the integral approximation. The general rule of thumb is 50 steps, but as you increase so does the compute time." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "explanation_spec:ig" + }, + "outputs": [], + "source": [ + "XAI = \"ig\" # [ shapley, ig, xrai ]\n", + "\n", + "PARAMETERS = {\"integrated_gradients_attribution\": {\"step_count\": 50}}\n", + "\n", + "\n", + "parameters = aip.ExplanationParameters(PARAMETERS)" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "explanation_metadata:image" + }, + "source": [ + "#### Explanation Metadata\n", + "\n", + "Let's first dive deeper into the explanation metadata, which consists of:\n", + "\n", + "- `outputs`: A scalar value in the output to attribute -- what to explain. For example, in a probability output \\[0.1, 0.2, 0.7\\] for classification, one wants an explanation for 0.7. Consider the following formulae, where the output is `y` and that is what we want to explain.\n", + "\n", + " y = f(x)\n", + "\n", + "Consider the following formulae, where the outputs are `y` and `z`. Since we can only do attribution for one scalar value, we have to pick whether we want to explain the output `y` or `z`. Assume in this example the model is object detection and y and z are the bounding box and the object classification. You would want to pick which of the two outputs to explain.\n", + "\n", + " y, z = f(x)\n", + "\n", + "The dictionary format for `outputs` is:\n", + "\n", + " { \"outputs\": { \"[your_display_name]\":\n", + " \"output_tensor_name\": [layer]\n", + " }\n", + " }\n", + "\n", + "
\n", + "
    \n", + "
  • [your_display_name]: A human readable name you assign to the output to explain. A common example is \"probability\".
    • \n", + "
  • \"output_tensor_name\": The key/value field to identify the output layer to explain.
    • \n", + "
  • [layer]: The output layer to explain. In a single task model, like a tabular regressor, it is the last (topmost) layer in the model
    • .\n", + "
\n", + "
\n", + "\n", + "\n", + "- `inputs`: The features for attribution -- how they contributed to the output. Consider the following formulae, where `a` and `b` are the features. We have to pick which features to explain how the contributed. Assume that this model is deployed for A/B testing, where `a` are the data_items for the prediction and `b` identifies whether the model instance is A or B. You would want to pick `a` (or some subset of) for the features, and not `b` since it does not contribute to the prediction.\n", + "\n", + " y = f(a,b)\n", + "\n", + "The minimum dictionary format for `inputs` is:\n", + "\n", + " { \"inputs\": { \"[your_display_name]\":\n", + " \"input_tensor_name\": [layer]\n", + " }\n", + " }\n", + "\n", + "
\n", + "
    \n", + "
  • [your_display_name]: A human readable name you assign to the input to explain. A common example is \"features\".
    • \n", + "
  • \"input_tensor_name\": The key/value field to identify the input layer for the feature attribution.
    • \n", + "
  • [layer]: The input layer for feature attribution. In a single input tensor model, it is the first (bottom-most) layer in the model.
    • \n", + "
\n", + "
\n", + "\n", + "Since the inputs to the model are images, you can specify the following additional fields as reporting/visualization aids:\n", + "\n", + "
\n", + "
    \n", + "
  • \"modality\": \"image\": Indicates the field values are image data.
    • \n", + "
\n", + "
" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "explanation_metadata:image" + }, + "outputs": [], + "source": [ + "random_baseline = np.random.rand(32, 32, 3)\n", + "input_baselines = [{\"number_vaue\": x} for x in random_baseline]\n", + "\n", + "INPUT_METADATA = {\"input_tensor_name\": CONCRETE_INPUT, \"modality\": \"image\"}\n", + "\n", + "OUTPUT_METADATA = {\"output_tensor_name\": serving_output}\n", + "\n", + "input_metadata = aip.ExplanationMetadata.InputMetadata(INPUT_METADATA)\n", + "output_metadata = aip.ExplanationMetadata.OutputMetadata(OUTPUT_METADATA)\n", + "\n", + "metadata = aip.ExplanationMetadata(\n", + " inputs={\"image\": input_metadata}, outputs={\"class\": output_metadata}\n", + ")\n", + "\n", + "explanation_spec = aip.ExplanationSpec(metadata=metadata, parameters=parameters)" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "upload_the_model:explanation" + }, + "source": [ + "### Upload the model\n", + "\n", + "Use this helper function `upload_model` to upload your model, stored in SavedModel format, up to the `Model` service, which will instantiate a Vertex `Model` resource instance for your model. Once you've done that, you can use the `Model` resource instance in the same way as any other Vertex `Model` resource instance, such as deploying to an `Endpoint` resource for serving predictions.\n", + "\n", + "The helper function takes the following parameters:\n", + "\n", + "- `display_name`: A human readable name for the `Endpoint` service.\n", + "- `image_uri`: The container image for the model deployment.\n", + "- `model_uri`: The Cloud Storage path to our SavedModel artificat. For this tutorial, this is the Cloud Storage location where the `trainer/task.py` saved the model artifacts, which we specified in the variable `MODEL_DIR`.\n", + "\n", + "The helper function calls the `Model` client service's method `upload_model`, which takes the following parameters:\n", + "\n", + "- `parent`: The Vertex location root path for `Dataset`, `Model` and `Endpoint` resources.\n", + "- `model`: The specification for the Vertex `Model` resource instance.\n", + "\n", + "Let's now dive deeper into the Vertex model specification `model`. This is a dictionary object that consists of the following fields:\n", + "\n", + "- `display_name`: A human readable name for the `Model` resource.\n", + "- `metadata_schema_uri`: Since your model was built without an Vertex `Dataset` resource, you will leave this blank (`''`).\n", + "- `artifact_uri`: The Cloud Storage path where the model is stored in SavedModel format.\n", + "- `container_spec`: This is the specification for the Docker container that will be installed on the `Endpoint` resource, from which the `Model` resource will serve predictions. Use the variable you set earlier `DEPLOY_GPU != None` to use a GPU; otherwise only a CPU is allocated.\n", + "- `explanation_spec`: This is the specification for enabling explainability for your model.\n", + "\n", + "Uploading a model into a Vertex Model resource returns a long running operation, since it may take a few moments. You call response.result(), which is a synchronous call and will return when the Vertex Model resource is ready.\n", + "\n", + "The helper function returns the Vertex fully qualified identifier for the corresponding Vertex Model instance upload_model_response.model. You will save the identifier for subsequent steps in the variable model_to_deploy_id." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "upload_the_model:explanation" + }, + "outputs": [], + "source": [ + "IMAGE_URI = DEPLOY_IMAGE\n", + "\n", + "\n", + "def upload_model(display_name, image_uri, model_uri):\n", + "\n", + " model = aip.Model(\n", + " display_name=display_name,\n", + " artifact_uri=model_uri,\n", + " metadata_schema_uri=\"\",\n", + " explanation_spec=explanation_spec,\n", + " container_spec={\"image_uri\": image_uri},\n", + " )\n", + "\n", + " response = clients[\"model\"].upload_model(parent=PARENT, model=model)\n", + " print(\"Long running operation:\", response.operation.name)\n", + " upload_model_response = response.result(timeout=180)\n", + " print(\"upload_model_response\")\n", + " print(\" model:\", upload_model_response.model)\n", + " return upload_model_response.model\n", + "\n", + "\n", + "model_to_deploy_id = upload_model(\n", + " \"cifar10-\" + TIMESTAMP, IMAGE_URI, model_path_to_deploy\n", + ")" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "get_model" + }, + "source": [ + "### Get `Model` resource information\n", + "\n", + "Now let's get the model information for just your model. Use this helper function `get_model`, with the following parameter:\n", + "\n", + "- `name`: The Vertex unique identifier for the `Model` resource.\n", + "\n", + "This helper function calls the Vertex `Model` client service's method `get_model`, with the following parameter:\n", + "\n", + "- `name`: The Vertex unique identifier for the `Model` resource." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "get_model" + }, + "outputs": [], + "source": [ + "def get_model(name):\n", + " response = clients[\"model\"].get_model(name=name)\n", + " print(response)\n", + "\n", + "\n", + "get_model(model_to_deploy_id)" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "create_endpoint:custom" + }, + "source": [ + "## Deploy the `Model` resource\n", + "\n", + "Now deploy the trained Vertex custom `Model` resource. This requires two steps:\n", + "\n", + "1. Create an `Endpoint` resource for deploying the `Model` resource to.\n", + "\n", + "2. Deploy the `Model` resource to the `Endpoint` resource." + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "create_endpoint" + }, + "source": [ + "### Create an `Endpoint` resource\n", + "\n", + "Use this helper function `create_endpoint` to create an endpoint to deploy the model to for serving predictions, with the following parameter:\n", + "\n", + "- `display_name`: A human readable name for the `Endpoint` resource.\n", + "\n", + "The helper function uses the endpoint client service's `create_endpoint` method, which takes the following parameter:\n", + "\n", + "- `display_name`: A human readable name for the `Endpoint` resource.\n", + "\n", + "Creating an `Endpoint` resource returns a long running operation, since it may take a few moments to provision the `Endpoint` resource for serving. You call `response.result()`, which is a synchronous call and will return when the Endpoint resource is ready. The helper function returns the Vertex fully qualified identifier for the `Endpoint` resource: `response.name`." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "create_endpoint" + }, + "outputs": [], + "source": [ + "ENDPOINT_NAME = \"cifar10_endpoint-\" + TIMESTAMP\n", + "\n", + "\n", + "def create_endpoint(display_name):\n", + " endpoint = {\"display_name\": display_name}\n", + " response = clients[\"endpoint\"].create_endpoint(parent=PARENT, endpoint=endpoint)\n", + " print(\"Long running operation:\", response.operation.name)\n", + "\n", + " result = response.result(timeout=300)\n", + " print(\"result\")\n", + " print(\" name:\", result.name)\n", + " print(\" display_name:\", result.display_name)\n", + " print(\" description:\", result.description)\n", + " print(\" labels:\", result.labels)\n", + " print(\" create_time:\", result.create_time)\n", + " print(\" update_time:\", result.update_time)\n", + " return result\n", + "\n", + "\n", + "result = create_endpoint(ENDPOINT_NAME)" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "endpoint_id:result" + }, + "source": [ + "Now get the unique identifier for the `Endpoint` resource you created." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "endpoint_id:result" + }, + "outputs": [], + "source": [ + "# The full unique ID for the endpoint\n", + "endpoint_id = result.name\n", + "# The short numeric ID for the endpoint\n", + "endpoint_short_id = endpoint_id.split(\"/\")[-1]\n", + "\n", + "print(endpoint_id)" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "instance_scaling" + }, + "outputs": [], + "source": [ + "MIN_NODES = 1\n", + "MAX_NODES = 1" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "deploy_model:dedicated,v1beta1" + }, + "source": [ + "### Deploy `Model` resource to the `Endpoint` resource\n", + "\n", + "Use this helper function `deploy_model` to deploy the model to the endpoint you created for serving predictions, with the following parameters:\n", + "\n", + "- `model`: The Vertex fully qualified identifier of the `Model` resource to upload (deploy) from the training pipeline.\n", + "- `deploy_model_display_name`: A human readable name for the deployed model.\n", + "- `endpoint`: The Vertex fully qualified `Endpoint` resource identifier to deploy the `Model` resource to.\n", + "\n", + "The helper function calls the `Endpoint` client service's method `deploy_model`, which takes the following parameters:\n", + "\n", + "- `endpoint`: The Vertex fully qualified `Endpoint` resource identifier to deploy the `Model` resource to to.\n", + "- `deployed_model`: The requirements for deploying the model.\n", + "- `traffic_split`: Percent of traffic at endpoint that goes to this model, which is specified as a dictioney of one or more key/value pairs.\n", + " - If only one model, then specify as **{ \"0\": 100 }**, where \"0\" refers to this model being uploaded and 100 means 100% of the traffic.\n", + " - If there are existing models on the endpoint, for which the traffic will be split, then specify as, where `model_id` is the model id of an existing model to the deployed endpoint. The percents must add up to 100.\n", + "\n", + " { \"0\": percent, model_id: percent, ... }\n", + "\n", + "Let's now dive deeper into the `deployed_model` parameter. This parameter is specified as a Python dictionary with the minimum required fields:\n", + "\n", + "- `model`: The Vertex fully qualified identifier of the (upload) `Model` resource to deploy.\n", + "- `display_name`: A human readable name for the deployed model.\n", + "- `dedicated_resources`: This refers to how many compute instances (replicas) that are scaled for serving prediction requests.\n", + " - `machine_spec`: The compute instance to provision. Use the variable you set earlier `DEPLOY_GPU != None` to use a GPU; otherwise only a CPU is allocated.\n", + " - `min_replica_count`: The number of compute instances to initially provision, which you set earlier as the variable `MIN_NODES`.\n", + " - `max_replica_count`: The maximum number of compute instances to scale to, which you set earlier as the variable `MAX_NODES`.\n", + "- `enable_container_logging`: This enables logging of container events, such as execution failures (default is container logging is disabled). Container logging is typically enabled when debugging the deployment and then disabled when deployed for production.\n", + "\n", + "#### Traffic Split\n", + "\n", + "Let's now dive deeper into the `traffic_split` parameter. This parameter is specified as a Python dictionary. This might at first be a tad bit confusing. Let me explain, you can deploy more than one instance of your model to an endpoint, and then set how much (percent) goes to each instance.\n", + "\n", + "Why would you do that? Perhaps you already have a previous version deployed in production -- let's call that v1. You got better model evaluation on v2, but you don't know for certain that it is really better until you deploy to production. So in the case of traffic split, you might want to deploy v2 to the same endpoint as v1, but it only get's say 10% of the traffic. That way, you can monitor how well it does without disrupting the majority of users -- until you make a final decision.\n", + "\n", + "#### Response\n", + "\n", + "The method returns a long running operation `response`. We will wait sychronously for the operation to complete by calling the `response.result()`, which will block until the model is deployed. If this is the first time a model is deployed to the endpoint, it may take a few additional minutes to complete provisioning of resources." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "deploy_model:dedicated,v1beta1" + }, + "outputs": [], + "source": [ + "DEPLOYED_NAME = \"cifar10_deployed-\" + TIMESTAMP\n", + "\n", + "\n", + "def deploy_model(\n", + " model, deployed_model_display_name, endpoint, traffic_split={\"0\": 100}\n", + "):\n", + "\n", + " if DEPLOY_GPU:\n", + " machine_spec = {\n", + " \"machine_type\": DEPLOY_COMPUTE,\n", + " \"accelerator_type\": DEPLOY_GPU,\n", + " \"accelerator_count\": DEPLOY_NGPU,\n", + " }\n", + " else:\n", + " machine_spec = {\n", + " \"machine_type\": DEPLOY_COMPUTE,\n", + " \"accelerator_count\": 0,\n", + " }\n", + "\n", + " deployed_model = {\n", + " \"model\": model,\n", + " \"display_name\": deployed_model_display_name,\n", + " \"dedicated_resources\": {\n", + " \"min_replica_count\": MIN_NODES,\n", + " \"max_replica_count\": MAX_NODES,\n", + " \"machine_spec\": machine_spec,\n", + " },\n", + " \"enable_container_logging\": False,\n", + " }\n", + "\n", + " response = clients[\"endpoint\"].deploy_model(\n", + " endpoint=endpoint, deployed_model=deployed_model, traffic_split=traffic_split\n", + " )\n", + "\n", + " print(\"Long running operation:\", response.operation.name)\n", + " result = response.result()\n", + " print(\"result\")\n", + " deployed_model = result.deployed_model\n", + " print(\" deployed_model\")\n", + " print(\" id:\", deployed_model.id)\n", + " print(\" model:\", deployed_model.model)\n", + " print(\" display_name:\", deployed_model.display_name)\n", + " print(\" create_time:\", deployed_model.create_time)\n", + "\n", + " return deployed_model.id\n", + "\n", + "\n", + "deployed_model_id = deploy_model(model_to_deploy_id, DEPLOYED_NAME, endpoint_id)" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "make_prediction" + }, + "source": [ + "## Send a online prediction request\n", + "\n", + "Send a online prediction to your deployed model." + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "get_test_item:test" + }, + "source": [ + "### Get test item\n", + "\n", + "You will use an example out of the test (holdout) portion of the dataset as a test item." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "get_test_item:test" + }, + "outputs": [], + "source": [ + "test_image = x_test[0]\n", + "test_label = y_test[0]\n", + "print(test_image.shape)" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "prepare_test_item:test,image" + }, + "source": [ + "### Prepare the request content\n", + "You are going to send the CIFAR10 image as compressed JPG image, instead of the raw uncompressed bytes:\n", + "\n", + "- `cv2.imwrite`: Use openCV to write the uncompressed image to disk as a compressed JPEG image.\n", + " - Denormalize the image data from \\[0,1) range back to [0,255).\n", + " - Convert the 32-bit floating point values to 8-bit unsigned integers.\n", + "- `tf.io.read_file`: Read the compressed JPG images back into memory as raw bytes.\n", + "- `base64.b64encode`: Encode the raw bytes into a base 64 encoded string." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "prepare_test_item:test,image" + }, + "outputs": [], + "source": [ + "import base64\n", + "\n", + "import cv2\n", + "\n", + "cv2.imwrite(\"tmp.jpg\", (test_image * 255).astype(np.uint8))\n", + "\n", + "bytes = tf.io.read_file(\"tmp.jpg\")\n", + "b64str = base64.b64encode(bytes.numpy()).decode(\"utf-8\")" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "send_explain_request:image" + }, + "source": [ + "### Send the prediction with explanation request\n", + "\n", + "Ok, now you have a test image. Use this helper function `explain_image`, which takes the parameters:\n", + "\n", + "- `image`: A list of test image data as a numpy array.\n", + "- `endpoint`: The Vertex fully qualified identifier for the `Endpoint` resource where the `Model` resource was deployed.\n", + "- `parameters_dict`: Additional parameters for serving.\n", + "- `deployed_model_id`: The Vertex fully qualified identifier for the deployed model, when more than one model is deployed at the endpoint. Otherwise, if only one model deployed, can be set to `None`.\n", + "\n", + "This function uses the prediction client service and calls the `explain` method with the parameters:\n", + "\n", + "- `endpoint`: The Vertex fully qualified identifier for the `Endpoint` resource where the `Model` resource was deployed.\n", + "- `instances`: A list of instances (encoded images) to predict and explain.\n", + "- `parameters`: Additional parameters for serving.\n", + "- `deployed_model_id`: The Vertex fully qualified identifier for the deployed model, when more than one model is deployed at the endpoint. Otherwise, if only one model deployed, can be set to `None`.\n", + "\n", + "To pass the image data to the prediction service, in the previous step you encoded the bytes into base 64 -- which makes the content safe from modification when transmitting binary data over the network. You need to tell the serving binary where your model is deployed to, that the content has been base 64 encoded, so it will decode it on the other end in the serving binary.\n", + "\n", + "Each instance in the prediction request is a dictionary entry of the form:\n", + "\n", + " {input_name: {'b64': content}}\n", + "\n", + "- `input_name`: the name of the input layer of the underlying model.\n", + "- `'b64'`: A key that indicates the content is base 64 encoded.\n", + "- `content`: The compressed JPG image bytes as a base 64 encoded string.\n", + "\n", + "Since the `predict()` service can take multiple images (instances), you will send your single image as a list of one image. As a final step, you package the instances list into Google's protobuf format -- which is what we pass to the `explain()` service.\n", + "\n", + "The `response` object returns a list, where each element in the list corresponds to the corresponding image in the request. You will see in the output for each prediction:\n", + "\n", + "- `deployed_model_id` -- The Vertex fully qualified identifer for the model that did the prediction/explanation.\n", + "- `predictions` -- Confidence level for the prediction (`predictions`), between 0 and 1, for each of the ten classes.\n", + "- `explanations` -- How each feature contributed to the prediction." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "send_explain_request:image" + }, + "outputs": [], + "source": [ + "def explain_image(image, endpoint, parameters_dict, deployed_model_id):\n", + " parameters = json_format.ParseDict(parameters_dict, Value())\n", + "\n", + " # The format of each instance should conform to the deployed model's prediction input schema.\n", + " instances_list = [{serving_input: {\"b64\": image}}]\n", + " instances = [json_format.ParseDict(s, Value()) for s in instances_list]\n", + "\n", + " response = clients[\"prediction\"].explain(\n", + " endpoint=endpoint,\n", + " instances=instances,\n", + " parameters=parameters,\n", + " deployed_model_id=deployed_model_id,\n", + " )\n", + " print(\"response\")\n", + " print(\" deployed_model_id:\", response.deployed_model_id)\n", + " predictions = response.predictions\n", + " print(\"predictions\")\n", + " for prediction in predictions:\n", + " print(\" prediction:\", prediction)\n", + "\n", + " explanations = response.explanations\n", + " print(\"explanations\")\n", + " for explanation in explanations:\n", + " print(\" explanation:\", explanation)\n", + "\n", + " return response\n", + "\n", + "\n", + "response = explain_image(b64str, endpoint_id, None, None)" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "understanding_explanations:cifar10" + }, + "source": [ + "### Understanding the explanations response\n", + "\n", + "Preview the images and their predicted classes without the explanations. Why did the model predict these classes?" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "understanding_explanations:cifar10" + }, + "outputs": [], + "source": [ + "from io import BytesIO\n", + "\n", + "import matplotlib.image as mpimg\n", + "import matplotlib.pyplot as plt\n", + "\n", + "CLASSES = [\n", + " \"airplane\",\n", + " \"automobile\",\n", + " \"bird\",\n", + " \"cat\",\n", + " \"deer\",\n", + " \"dog\",\n", + " \"frog\",\n", + " \"horse\",\n", + " \"ship\",\n", + " \"truck\",\n", + "]\n", + "\n", + "# Note: change the `ig_response` variable below if you didn't deploy an IG model\n", + "for prediction in response.predictions:\n", + " label_index = np.argmax(prediction)\n", + " class_name = CLASSES[label_index]\n", + " confidence_score = prediction[label_index]\n", + " print(\n", + " \"Predicted class: \"\n", + " + class_name\n", + " + \"\\n\"\n", + " + \"Confidence score: \"\n", + " + str(confidence_score)\n", + " )\n", + "\n", + " image = base64.b64decode(b64str)\n", + " image = BytesIO(image)\n", + " img = mpimg.imread(image, format=\"JPG\")\n", + "\n", + " plt.imshow(img, interpolation=\"nearest\")\n", + " plt.show()" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "visualize_image_explanations" + }, + "source": [ + "### Visualize the images with AI Explanations\n", + "\n", + "The images returned show the explanations for only the top class predicted by the model. This means that if one of the model's predictions is incorrect, the pixels you see highlighted are for the *incorrect class*. For example, if the model predicted \"airplane\" when it should have predicted \"cat\", you can see explanations for why the model classified this image as an airplane.\n", + "\n", + "If you deployed an Integrated Gradients model, you can visualize its feature attributions. Currently, the highlighted pixels returned from AI Explanations show the top 60% of pixels that contributed to the model's prediction. The pixels you see after running the cell below show the pixels that most signaled the model's prediction." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "visualize_image_explanations" + }, + "outputs": [], + "source": [ + "import io\n", + "\n", + "for explanation in response.explanations:\n", + " attributions = dict(explanation.attributions[0].feature_attributions)\n", + " label_index = explanation.attributions[0].output_index[0]\n", + " class_name = CLASSES[label_index]\n", + " b64str = attributions[\"image\"][\"b64_jpeg\"]\n", + " image = base64.b64decode(b64str)\n", + " image = io.BytesIO(image)\n", + " img = mpimg.imread(image, format=\"JPG\")\n", + "\n", + " plt.imshow(img, interpolation=\"nearest\")\n", + " plt.show()" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "undeploy_model" + }, + "source": [ + "## Undeploy the `Model` resource\n", + "\n", + "To undeploy your `Model` resource from the serving `Endpoint` resoure, use this helper function `undeploy_model`, which takes the following parameters:\n", + "\n", + "- `deployed_model_id`: The model deployment identifier returned by the endpoint service when the `Model` resource was deployed to.\n", + "- `endpoint`: The Vertex fully qualified identifier for the `Endpoint` resource where the `Model` is deployed to.\n", + "\n", + "This function calls the endpoint client service's method `undeploy_model`, with the following parameters:\n", + "\n", + "- `deployed_model_id`: The model deployment identifier returned by the endpoint service when the `Model` resource was deployed.\n", + "- `endpoint`: The Vertex fully qualified identifier for the `Endpoint` resource where the `Model` resource is deployed.\n", + "- `traffic_split`: How to split traffic among the remaining deployed models on the `Endpoint` resource.\n", + "\n", + "Since this is the only deployed model on the `Endpoint` resource, you can leave `traffic_split` empty by setting it to {}." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "undeploy_model" + }, + "outputs": [], + "source": [ + "def undeploy_model(deployed_model_id, endpoint):\n", + " response = clients[\"endpoint\"].undeploy_model(\n", + " endpoint=endpoint, deployed_model_id=deployed_model_id, traffic_split={}\n", + " )\n", + " print(response)\n", + "\n", + "\n", + "undeploy_model(deployed_model_id, endpoint_id)" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "cleanup" + }, + "source": [ + "# Cleaning up\n", + "\n", + "To clean up all Google Cloud resources used in this project, you can [delete the GCP\n", + "project](https://cloud.google.com/resource-manager/docs/creating-managing-projects#shutting_down_projects) you used for the tutorial.\n", + "\n", + "Otherwise, you can delete the individual resources you created in this tutorial:\n", + "\n", + "- Dataset\n", + "- Pipeline\n", + "- Model\n", + "- Endpoint\n", + "- Batch Job\n", + "- Custom Job\n", + "- Hyperparameter Tuning Job\n", + "- Cloud Storage Bucket" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "cleanup" + }, + "outputs": [], + "source": [ + "delete_dataset = True\n", + "delete_pipeline = True\n", + "delete_model = True\n", + "delete_endpoint = True\n", + "delete_batchjob = True\n", + "delete_customjob = True\n", + "delete_hptjob = True\n", + "delete_bucket = True\n", + "\n", + "# Delete the dataset using the Vertex fully qualified identifier for the dataset\n", + "try:\n", + " if delete_dataset and \"dataset_id\" in globals():\n", + " clients[\"dataset\"].delete_dataset(name=dataset_id)\n", + "except Exception as e:\n", + " print(e)\n", + "\n", + "# Delete the training pipeline using the Vertex fully qualified identifier for the pipeline\n", + "try:\n", + " if delete_pipeline and \"pipeline_id\" in globals():\n", + " clients[\"pipeline\"].delete_training_pipeline(name=pipeline_id)\n", + "except Exception as e:\n", + " print(e)\n", + "\n", + "# Delete the model using the Vertex fully qualified identifier for the model\n", + "try:\n", + " if delete_model and \"model_to_deploy_id\" in globals():\n", + " clients[\"model\"].delete_model(name=model_to_deploy_id)\n", + "except Exception as e:\n", + " print(e)\n", + "\n", + "# Delete the endpoint using the Vertex fully qualified identifier for the endpoint\n", + "try:\n", + " if delete_endpoint and \"endpoint_id\" in globals():\n", + " clients[\"endpoint\"].delete_endpoint(name=endpoint_id)\n", + "except Exception as e:\n", + " print(e)\n", + "\n", + "# Delete the batch job using the Vertex fully qualified identifier for the batch job\n", + "try:\n", + " if delete_batchjob and \"batch_job_id\" in globals():\n", + " clients[\"job\"].delete_batch_prediction_job(name=batch_job_id)\n", + "except Exception as e:\n", + " print(e)\n", + "\n", + "# Delete the custom job using the Vertex fully qualified identifier for the custom job\n", + "try:\n", + " if delete_customjob and \"job_id\" in globals():\n", + " clients[\"job\"].delete_custom_job(name=job_id)\n", + "except Exception as e:\n", + " print(e)\n", + "\n", + "# Delete the hyperparameter tuning job using the Vertex fully qualified identifier for the hyperparameter tuning job\n", + "try:\n", + " if delete_hptjob and \"hpt_job_id\" in globals():\n", + " clients[\"job\"].delete_hyperparameter_tuning_job(name=hpt_job_id)\n", + "except Exception as e:\n", + " print(e)\n", + "\n", + "if delete_bucket and \"BUCKET_NAME\" in globals():\n", + " ! gsutil rm -r $BUCKET_NAME" + ] + } + ], + "metadata": { + "colab": { + "name": "gapic-custom_image_classification_online_explain.ipynb", + "toc_visible": true + }, + "environment": { + "name": "common-cu110.m71", + "type": "gcloud", + "uri": "gcr.io/deeplearning-platform-release/base-cu110:m71" + }, + "kernelspec": { + "display_name": "Python 3", + "language": "python", + "name": "python3" + }, + "language_info": { + "codemirror_mode": { + "name": "ipython", + "version": 3 + }, + "file_extension": ".py", + "mimetype": "text/x-python", + "name": "python", + "nbconvert_exporter": "python", + "pygments_lexer": "ipython3", + "version": "3.7.10" + } + }, + "nbformat": 4, + "nbformat_minor": 4 +} diff --git a/notebooks/official/explainable_ai/gapic-custom_tabular_regression_batch_explain.ipynb b/notebooks/official/explainable_ai/gapic-custom_tabular_regression_batch_explain.ipynb new file mode 100644 index 000000000..18ab396a3 --- /dev/null +++ b/notebooks/official/explainable_ai/gapic-custom_tabular_regression_batch_explain.ipynb @@ -0,0 +1,2455 @@ +{ + "cells": [ + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "copyright", + "repo": "snippets_housekeeping.ipynb" + }, + "outputs": [], + "source": [ + "# Copyright 2020 Google LLC\n", + "#\n", + "# Licensed under the Apache License, Version 2.0 (the \"License\");\n", + "# you may not use this file except in compliance with the License.\n", + "# You may obtain a copy of the License at\n", + "#\n", + "# https://www.apache.org/licenses/LICENSE-2.0\n", + "#\n", + "# Unless required by applicable law or agreed to in writing, software\n", + "# distributed under the License is distributed on an \"AS IS\" BASIS,\n", + "# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n", + "# See the License for the specific language governing permissions and\n", + "# limitations under the License." + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "title", + "repo": "snippets_housekeeping.ipynb" + }, + "source": [ + "# Vertex client library for Python: Custom training tabular regression model for batch prediction with explanation\n", + "\n", + "\n", + " \n", + " \n", + "
\n", + " \n", + " \"Colab Run in Colab\n", + " \n", + " \n", + " \n", + " \"GitHub\n", + " View on GitHub\n", + " \n", + "
\n", + "


" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "overview:custom,xai", + "repo": "snippets_housekeeping.ipynb" + }, + "source": [ + "## Overview\n", + "\n", + "\n", + "This tutorial demonstrates how to use the Vertex client library for Python to train and deploy a custom tabular regression model for batch prediction with explanation." + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "dataset:custom,boston,lrg", + "repo": "snippets_housekeeping.ipynb" + }, + "source": [ + "### Dataset\n", + "\n", + "The dataset used for this tutorial is the [Boston Housing Prices dataset](https://www.cs.toronto.edu/~delve/data/boston/bostonDetail.html). The version of the dataset you will use in this tutorial is built into TensorFlow. The trained model predicts the median price of a house in units of 1K USD." + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "objective:custom,training,batch_prediction,xai", + "repo": "snippets_housekeeping.ipynb" + }, + "source": [ + "### Objective\n", + "\n", + "In this tutorial, you create a custom trained model, with a training pipeline, from a Python script in a Google prebuilt Docker container using the Vertex client library for Python, and then do a batch prediction with explanations on the uploaded model. Alternatively, you can create custom trained models using `gcloud` command-line tool or online using Cloud Console.\n", + "\n", + "The steps performed include:\n", + "\n", + "- Create a Vertex custom job for training a model.\n", + "- Train the TensorFlow model.\n", + "- Retrieve and load the model artifacts.\n", + "- View the model evaluation.\n", + "- Set explanation parameters.\n", + "- Upload the model as a Vertex `Model` resource.\n", + "- Make a batch prediction with explanations." + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "costs", + "repo": "snippets_housekeeping.ipynb" + }, + "source": [ + "### Costs\n", + "\n", + "This tutorial uses billable components of Google Cloud (GCP):\n", + "\n", + "* Vertex AI\n", + "* Cloud Storage\n", + "\n", + "Learn about [Vertex AI\n", + "pricing](https://cloud.google.com/vertex-ai/pricing) and [Cloud Storage\n", + "pricing](https://cloud.google.com/storage/pricing), and use the [Pricing\n", + "Calculator](https://cloud.google.com/products/calculator/)\n", + "to generate a cost estimate based on your projected usage." + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "install_aip", + "repo": "snippets_housekeeping.ipynb" + }, + "source": [ + "## Installation\n", + "\n", + "Install the latest version of Vertex client library for Python." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "install_aip", + "repo": "snippets_housekeeping.ipynb" + }, + "outputs": [], + "source": [ + "import sys\n", + "import os\n", + "\n", + "\n", + "# Google Cloud Notebook\n", + "if os.path.exists(\"/opt/deeplearning/metadata/env_version\"):\n", + " USER_FLAG = '--user'\n", + "else:\n", + " USER_FLAG = ''\n", + "\n", + "! pip3 install --upgrade google-cloud-aiplatform $USER_FLAG" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "install_storage", + "repo": "snippets_housekeeping.ipynb" + }, + "source": [ + "Install the latest GA version of *google-cloud-storage* library as well." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "install_storage", + "repo": "snippets_housekeeping.ipynb" + }, + "outputs": [], + "source": [ + "! pip3 install -U google-cloud-storage $USER_FLAG" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "install_other_packages", + "repo": "snippets_common.ipynb" + }, + "source": [ + "### Install other packages\n", + "\n", + "Install other packages required for this tutorial." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "install_tabulate", + "repo": "snippets_common.ipynb" + }, + "outputs": [], + "source": [ + "! pip3 install -U tabulate $USER_FLAG" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "restart", + "repo": "snippets_housekeeping.ipynb" + }, + "source": [ + "### Restart the kernel\n", + "\n", + "Once you've installed the Vertex client library for Python and *google-cloud-storage* library, you need to restart the notebook kernel so it can find the packages." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "restart", + "repo": "snippets_housekeeping.ipynb" + }, + "outputs": [], + "source": [ + "if not os.getenv(\"IS_TESTING\"):\n", + " # Automatically restart kernel after installs\n", + " import IPython\n", + " app = IPython.Application.instance()\n", + " app.kernel.do_shutdown(True)" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "before_you_begin", + "repo": "snippets_housekeeping.ipynb" + }, + "source": [ + "## Before you begin\n", + "\n", + "### GPU runtime\n", + "\n", + "*Make sure you're running this notebook in a GPU runtime if you have that option. In Colab, select* **Runtime > Change Runtime Type > GPU**\n", + "\n", + "### Set up your Google Cloud project\n", + "\n", + "**The following steps are required, regardless of your notebook environment.**\n", + "\n", + "1. [Select or create a Google Cloud project](https://console.cloud.google.com/cloud-resource-manager). When you first create an account, you get a $300 free credit towards your compute/storage costs.\n", + "\n", + "2. [Make sure that billing is enabled for your project.](https://cloud.google.com/billing/docs/how-to/modify-project)\n", + "\n", + "3. [Enable the Vertex AI APIs and Compute Engine APIs.](https://console.cloud.google.com/flows/enableapi?apiid=ml.googleapis.com,compute_component)\n", + "\n", + "4. [The Cloud SDK](https://cloud.google.com/sdk) is already installed in Google Cloud Notebook.\n", + "\n", + "5. Enter your project ID in the cell below. Then run the cell to make sure the\n", + "Cloud SDK uses the right project for all the commands in this notebook.\n", + "\n", + "**Note**: Jupyter runs lines prefixed with `!` as shell commands, and it interpolates Python variables prefixed with `$`." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "set_project_id", + "repo": "snippets_housekeeping.ipynb" + }, + "outputs": [], + "source": [ + "PROJECT_ID = \"[your-project-id]\" #@param {type:\"string\"}" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "autoset_project_id", + "repo": "snippets_housekeeping.ipynb" + }, + "outputs": [], + "source": [ + "if PROJECT_ID == \"\" or PROJECT_ID is None or PROJECT_ID == \"[your-project-id]\":\n", + " # Get your GCP project id from gcloud\n", + " shell_output = !gcloud config list --format 'value(core.project)' 2>/dev/null\n", + " PROJECT_ID = shell_output[0]\n", + " print(\"Project ID:\", PROJECT_ID)" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "set_gcloud_project_id", + "repo": "snippets_housekeeping.ipynb" + }, + "outputs": [], + "source": [ + "! gcloud config set project $PROJECT_ID" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "region", + "repo": "snippets_housekeeping.ipynb" + }, + "source": [ + "#### Region\n", + "\n", + "You can also change the `REGION` variable, which is used for operations\n", + "throughout the rest of this notebook. Below are regions supported for Vertex AI. We recommend that you choose the region closest to you.\n", + "\n", + "- Americas: `us-central1`\n", + "- Europe: `europe-west4`\n", + "- Asia Pacific: `asia-east1`\n", + "\n", + "You may not use a multi-regional bucket for training with Vertex AI. Not all regions provide support for all Vertex AI services.\n", + "\n", + "Learn more about [Vertex AI regions](https://cloud.google.com/vertex-ai/docs/general/locations)" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "region", + "repo": "snippets_housekeeping.ipynb" + }, + "outputs": [], + "source": [ + "REGION = 'us-central1' #@param {type: \"string\"}" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "timestamp", + "repo": "snippets_housekeeping.ipynb" + }, + "source": [ + "#### Timestamp\n", + "\n", + "If you are in a live tutorial session, you might be using a shared test account or project. To avoid name collisions between users on resources created, you create a timestamp for each instance session, and append the timestamp onto the name of resources you create in this tutorial." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "timestamp", + "repo": "snippets_housekeeping.ipynb" + }, + "outputs": [], + "source": [ + "from datetime import datetime\n", + "\n", + "TIMESTAMP = datetime.now().strftime(\"%Y%m%d%H%M%S\")" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "gcp_authenticate", + "repo": "snippets_housekeeping.ipynb" + }, + "source": [ + "### Authenticate your Google Cloud account\n", + "\n", + "**If you are using Google Cloud Notebook**, your environment is already authenticated. Skip this step.\n", + "\n", + "**If you are using Colab**, run the cell below and follow the instructions when prompted to authenticate your account via oAuth.\n", + "\n", + "**Otherwise**, follow these steps:\n", + "\n", + "In the Cloud Console, go to the [Create service account key](https://console.cloud.google.com/apis/credentials/serviceaccountkey) page.\n", + "\n", + "**Click Create service account**.\n", + "\n", + "In the **Service account name** field, enter a name, and click **Create**.\n", + "\n", + "In the **Grant this service account access to project** section, click the Role drop-down list. Type \"Vertex\" into the filter box, and select **Vertex AI Administrator**. Type \"Storage Object Admin\" into the filter box, and select **Storage Object Admin**.\n", + "\n", + "Click Create. A JSON file that contains your key downloads to your local environment.\n", + "\n", + "Enter the path to your service account key as the `GOOGLE_APPLICATION_CREDENTIALS` variable in the cell below and run the cell." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "gcp_authenticate", + "repo": "snippets_housekeeping.ipynb" + }, + "outputs": [], + "source": [ + "# If you are running this notebook in Colab, run this cell and follow the\n", + "# instructions to authenticate your Google Cloud account. This provides access to your\n", + "# Cloud Storage bucket and lets you submit training jobs and prediction\n", + "# requests.\n", + "\n", + "# If on Google Cloud Notebook, then don't execute this code\n", + "if not os.path.exists(\"/opt/deeplearning/metadata/env_version\"):\n", + " if \"google.colab\" in sys.modules:\n", + " from google.colab import auth as google_auth\n", + "\n", + " google_auth.authenticate_user()\n", + "\n", + " # If you are running this notebook locally, replace the string below with the\n", + " # path to your service account key and run this cell to authenticate your Google Cloud\n", + " # account.\n", + " elif not os.getenv(\"IS_TESTING\"):\n", + " %env GOOGLE_APPLICATION_CREDENTIALS ''" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "bucket:custom", + "repo": "snippets_housekeeping.ipynb" + }, + "source": [ + "### Create a Cloud Storage bucket\n", + "\n", + "**The following steps are required, regardless of your notebook environment.**\n", + "\n", + "When you submit a custom training job using the Vertex client library for Python, you upload a Python package\n", + "containing your training code to a Cloud Storage bucket. Vertex runs\n", + "the code from this package. In this tutorial, Vertex also saves the\n", + "trained model that results from your job in the same bucket. You can then\n", + "create an `Endpoint` resource based on this output in order to serve\n", + "online predictions.\n", + "\n", + "Set the name of your Cloud Storage bucket below. Bucket names must be globally unique across all Google Cloud projects, including those outside of your organization." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "bucket", + "repo": "snippets_housekeeping.ipynb" + }, + "outputs": [], + "source": [ + "BUCKET_NAME = \"gs://[your-bucket-name]\" #@param {type:\"string\"}" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "autoset_bucket", + "repo": "snippets_housekeeping.ipynb" + }, + "outputs": [], + "source": [ + "if BUCKET_NAME == \"\" or BUCKET_NAME is None or BUCKET_NAME == \"gs://[your-bucket-name]\":\n", + " BUCKET_NAME = \"gs://\" + PROJECT_ID + \"aip-\" + TIMESTAMP" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "create_bucket", + "repo": "snippets_housekeeping.ipynb" + }, + "source": [ + "**Only if your bucket doesn't already exist**: Run the following cell to create your Cloud Storage bucket." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "create_bucket", + "repo": "snippets_housekeeping.ipynb" + }, + "outputs": [], + "source": [ + "! gsutil mb -l $REGION $BUCKET_NAME" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "validate_bucket", + "repo": "snippets_housekeeping.ipynb" + }, + "source": [ + "Finally, validate access to your Cloud Storage bucket by examining its contents:" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "validate_bucket", + "repo": "snippets_housekeeping.ipynb" + }, + "outputs": [], + "source": [ + "! gsutil ls -al $BUCKET_NAME" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "setup_vars", + "repo": "snippets_housekeeping.ipynb" + }, + "source": [ + "### Set up variables\n", + "\n", + "Next, set up some variables used throughout the tutorial.\n", + "### Import libraries and define constants" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "import_aip:v1beta1,protobuf", + "repo": "snippets_housekeeping.ipynb" + }, + "source": [ + "#### Import Vertex client library for Python\n", + "\n", + "Import the Vertex client library for Python into your Python environment." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "import_aip:v1beta1,protobuf", + "repo": "snippets_housekeeping.ipynb" + }, + "outputs": [], + "source": [ + "import time\n", + "\n", + "import google.cloud.aiplatform_v1beta1 as aip\n", + "\n", + "from google.protobuf import json_format\n", + "from google.protobuf.struct_pb2 import Value\n", + "from google.protobuf.struct_pb2 import Struct\n", + "from google.protobuf.json_format import MessageToJson\n", + "from google.protobuf.json_format import ParseDict" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "aip_constants", + "repo": "snippets_housekeeping.ipynb" + }, + "source": [ + "#### Vertex AI constants\n", + "\n", + "Setup up the following constants for Vertex AI:\n", + "\n", + "- `API_ENDPOINT`: The Vertex AI service endpoint for dataset, model, job, pipeline and endpoint services.\n", + "- `PARENT`: The Vertex AI location root path for `Dataset`, `Model`, `Job`, `Pipeline` and `Endpoint` resources." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "aip_constants", + "repo": "snippets_housekeeping.ipynb" + }, + "outputs": [], + "source": [ + "# API service endpoint\n", + "API_ENDPOINT = \"{0}-aiplatform.googleapis.com\".format(REGION)\n", + "\n", + "# Vertex AI location root path for your dataset, model and endpoint resources\n", + "PARENT = \"projects/\" + PROJECT_ID + \"/locations/\" + REGION" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "accelerators:training,prediction,cpu", + "repo": "snippets_housekeeping.ipynb" + }, + "source": [ + "#### Set hardware accelerators\n", + "\n", + "You can set hardware accelerators for training and prediction.\n", + "\n", + "Set the variables `TRAIN_GPU/TRAIN_NGPU` and `DEPLOY_GPU/DEPLOY_NGPU` to use a container image supporting a GPU and the number of GPUs allocated to the virtual machine (VM) instance. For example, to use a GPU container image with 4 Nvidia Telsa K80 GPUs allocated to each VM, you would specify:\n", + "\n", + " (aip.AcceleratorType.NVIDIA_TESLA_K80, 4)\n", + "\n", + "\n", + "Otherwise specify `(None, None)` to use a container image to run on a CPU.\n", + "\n", + "Learn [which accelerators are available in your region](https://cloud.google.com/vertex-ai/docs/general/locations#accelerators).\n", + "\n", + "*Note*: TF releases before 2.3 for GPU support will fail to load the custom trained model in this tutorial. It is a known issue and fixed in TF 2.3 -- which is caused by static graph ops that are generated in the serving function. If you encounter this issue on your own custom trained models, use a container image for TF 2.3 with GPU support." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "accelerators:training,prediction,cpu", + "repo": "snippets_housekeeping.ipynb" + }, + "outputs": [], + "source": [ + "if os.getenv(\"IS_TESTING_TRAIN_GPU\"):\n", + " TRAIN_GPU, TRAIN_NGPU = (aip.AcceleratorType.NVIDIA_TESLA_K80, int(os.getenv(\"IS_TESTING_TRAIN_GPU\")))\n", + "else:\n", + " TRAIN_GPU, TRAIN_NGPU = (aip.AcceleratorType.NVIDIA_TESLA_K80, 1)\n", + "\n", + "if os.getenv(\"IS_TESTING_DEPLOY_GPU\"):\n", + " DEPLOY_GPU, DEPLOY_NGPU = (aip.AcceleratorType.NVIDIA_TESLA_K80, int(os.getenv(\"IS_TESTING_DEPLOY_GPU\")))\n", + "else:\n", + " DEPLOY_GPU, DEPLOY_NGPU = (None, None)" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "container:training,prediction", + "repo": "snippets_housekeeping.ipynb" + }, + "source": [ + "#### Set pre-built containers\n", + "\n", + "Set the pre-built Docker container image for training and prediction.\n", + "\n", + "\n", + "For the latest list, see [Pre-built containers for training](https://cloud.google.com/ai-platform-unified/docs/training/pre-built-containers).\n", + "\n", + "\n", + "For the latest list, see [Pre-built containers for prediction](https://cloud.google.com/ai-platform-unified/docs/predictions/pre-built-containers)" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "container:training,prediction", + "repo": "snippets_housekeeping.ipynb" + }, + "outputs": [], + "source": [ + "if os.getenv(\"IS_TESTING_TF\"):\n", + " TF = os.getenv(\"IS_TESTING_TF\")\n", + "else:\n", + " TF = '2-1'\n", + "\n", + "if TF[0] == '2':\n", + " if TRAIN_GPU:\n", + " TRAIN_VERSION = 'tf-gpu.{}'.format(TF)\n", + " else:\n", + " TRAIN_VERSION = 'tf-cpu.{}'.format(TF)\n", + " if DEPLOY_GPU:\n", + " DEPLOY_VERSION = 'tf2-gpu.{}'.format(TF)\n", + " else:\n", + " DEPLOY_VERSION = 'tf2-cpu.{}'.format(TF)\n", + "else:\n", + " if TRAIN_GPU:\n", + " TRAIN_VERSION = 'tf-gpu.{}'.format(TF)\n", + " else:\n", + " TRAIN_VERSION = 'tf-cpu.{}'.format(TF)\n", + " if DEPLOY_GPU:\n", + " DEPLOY_VERSION = 'tf-gpu.{}'.format(TF)\n", + " else:\n", + " DEPLOY_VERSION = 'tf-cpu.{}'.format(TF)\n", + "\n", + "TRAIN_IMAGE = \"gcr.io/cloud-aiplatform/training/{}:latest\".format(TRAIN_VERSION)\n", + "DEPLOY_IMAGE = \"gcr.io/cloud-aiplatform/prediction/{}:latest\".format(DEPLOY_VERSION)\n", + "\n", + "print(\"Training:\", TRAIN_IMAGE, TRAIN_GPU, TRAIN_NGPU)\n", + "print(\"Deployment:\", DEPLOY_IMAGE, DEPLOY_GPU, DEPLOY_NGPU)" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "machine:training,prediction", + "repo": "snippets_housekeeping.ipynb" + }, + "source": [ + "#### Set machine type\n", + "\n", + "Next, set the machine type to use for training and prediction.\n", + "\n", + "- Set the variables `TRAIN_COMPUTE` and `DEPLOY_COMPUTE` to configure the compute resources for the VMs you will use for for training and prediction.\n", + " - `machine type`\n", + " - `n1-standard`: 3.75GB of memory per vCPU.\n", + " - `n1-highmem`: 6.5GB of memory per vCPU\n", + " - `n1-highcpu`: 0.9 GB of memory per vCPU\n", + " - `vCPUs`: number of \\[2, 4, 8, 16, 32, 64, 96 \\]\n", + "\n", + "*Note: The following is not supported for training:*\n", + "\n", + " - `standard`: 2 vCPUs\n", + " - `highcpu`: 2, 4 and 8 vCPUs\n", + "\n", + "*Note: You may also use n2 and e2 machine types for training and deployment, but they do not support GPUs*.\n", + "\n", + "Learn [which machine types are available for training](https://cloud.google.com/vertex-ai/docs/training/configure-compute) and [for prediction](https://cloud.google.com/vertex-ai/docs/predictions/configure-compute)" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "machine:training,prediction", + "repo": "snippets_housekeeping.ipynb" + }, + "outputs": [], + "source": [ + "if os.getenv(\"IS_TESTING_TRAIN_MACHINE\"):\n", + " MACHINE_TYPE = os.getenv(\"IS_TESTING_TRAIN_MACHINE\")\n", + "else:\n", + " MACHINE_TYPE = 'n1-standard'\n", + "\n", + "VCPU = '4'\n", + "TRAIN_COMPUTE = MACHINE_TYPE + '-' + VCPU\n", + "print('Train machine type', TRAIN_COMPUTE)\n", + "\n", + "if os.getenv(\"IS_TESTING_DEPLOY_MACHINE\"):\n", + " MACHINE_TYPE = os.getenv(\"IS_TESTING_DEPLOY_MACHINE\")\n", + "else:\n", + " MACHINE_TYPE = 'n1-standard'\n", + "\n", + "VCPU = '4'\n", + "DEPLOY_COMPUTE = MACHINE_TYPE + '-' + VCPU\n", + "print('Deploy machine type', DEPLOY_COMPUTE)" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "tutorial_start:custom", + "repo": "snippets_custom.ipynb" + }, + "source": [ + "# Tutorial\n", + "\n", + "Now you are ready to start creating your own custom model and training for Boston Housing." + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "clients:custom", + "repo": "snippets_housekeeping.ipynb" + }, + "source": [ + "## Set up clients\n", + "\n", + "The Vertex client library for Python works as a client/server model. On your side (the Python script) you will create a client that sends requests and receives responses from the Vertex AI server.\n", + "\n", + "You will use different clients in this tutorial for different steps in the workflow. So set them all up upfront.\n", + "\n", + "- Model Service for `Model` resources.\n", + "- Endpoint Service for deployment.\n", + "- Job Service for batch jobs and custom training.\n", + "- Prediction Service for serving." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "clients:custom", + "repo": "snippets_housekeeping.ipynb" + }, + "outputs": [], + "source": [ + "# client options same for all services\n", + "client_options = {\"api_endpoint\": API_ENDPOINT}\n", + "\n", + "\n", + "def create_job_client():\n", + " client = aip.JobServiceClient(\n", + " client_options=client_options\n", + " )\n", + " return client\n", + "\n", + "\n", + "def create_model_client():\n", + " client = aip.ModelServiceClient(\n", + " client_options=client_options\n", + " )\n", + " return client\n", + "\n", + "\n", + "def create_endpoint_client():\n", + " client = aip.EndpointServiceClient(\n", + " client_options=client_options\n", + " )\n", + " return client\n", + "\n", + "\n", + "def create_prediction_client():\n", + " client = aip.PredictionServiceClient(\n", + " client_options=client_options\n", + " )\n", + " return client\n", + "\n", + "\n", + "clients = {}\n", + "clients['job'] = create_job_client()\n", + "clients['model'] = create_model_client()\n", + "clients['endpoint'] = create_endpoint_client()\n", + "clients['prediction'] = create_prediction_client()\n", + "\n", + "for client in clients.items():\n", + " print(client)" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "train_custom_model", + "repo": "snippets_custom.ipynb" + }, + "source": [ + "## Train a model\n", + "\n", + "There are two ways you can train a custom model using a container image:\n", + "\n", + "- **Use a Google Cloud prebuilt container**. If you use a prebuilt container, you will additionally specify a Python package to install into the container image. This Python package contains your code for training a custom model.\n", + "\n", + "- **Use your own custom container image**. If you use your own container, the container needs to contain your code for training a custom model." + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "train_custom_job_specification:prebuilt_container", + "repo": "snippets_custom.ipynb" + }, + "source": [ + "## Prepare your custom job specification\n", + "\n", + "Now that your clients are ready, your first step is to create a Job Specification for your custom training job. The job specification will consist of the following:\n", + "\n", + "- `worker_pool_spec` : The specification of the type of machine(s) you will use for training and how many (single or distributed)\n", + "- `python_package_spec` : The specification of the Python package to be installed with the pre-built container." + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "train_custom_job_machine_specification", + "repo": "snippets_custom.ipynb" + }, + "source": [ + "### Prepare your machine specification\n", + "\n", + "Now define the machine specification for your custom training job. This tells Vertex AI what type of machine instance to provision for the training.\n", + " - `machine_type`: The type of Google Cloud instance to provision -- e.g., n1-standard-8.\n", + " - `accelerator_type`: The type, if any, of hardware accelerator. In this tutorial if you previously set the variable `TRAIN_GPU != None`, you are using a GPU; otherwise you will use a CPU.\n", + " - `accelerator_count`: The number of accelerators." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "train_custom_job_machine_specification", + "repo": "snippets_custom.ipynb" + }, + "outputs": [], + "source": [ + "if TRAIN_GPU:\n", + " machine_spec = {\n", + " \"machine_type\": TRAIN_COMPUTE,\n", + " \"accelerator_type\": TRAIN_GPU,\n", + " \"accelerator_count\": TRAIN_NGPU\n", + " }\n", + "else:\n", + " machine_spec = {\n", + " \"machine_type\": TRAIN_COMPUTE,\n", + " \"accelerator_count\": 0\n", + " }" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "train_custom_job_disk_specification", + "repo": "snippets_custom.ipynb" + }, + "source": [ + "### Prepare your disk specification\n", + "\n", + "(optional) Now define the disk specification for your custom training job. This tells Vertex AI what type and size of disk to provision in each machine instance for the training.\n", + "\n", + " - `boot_disk_type`: Either SSD or Standard. SSD is faster, and Standard is less expensive. Defaults to SSD.\n", + " - `boot_disk_size_gb`: Size of disk in GB." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "train_custom_job_disk_specification", + "repo": "snippets_custom.ipynb" + }, + "outputs": [], + "source": [ + "DISK_TYPE = \"pd-ssd\" # [ pd-ssd, pd-standard]\n", + "DISK_SIZE = 200 # GB\n", + "\n", + "disk_spec = {\n", + " \"boot_disk_type\": DISK_TYPE,\n", + " \"boot_disk_size_gb\": DISK_SIZE\n", + "}" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "train_custom_job_worker_pool_specification:prebuilt_container", + "repo": "snippets_custom.ipynb" + }, + "source": [ + "### Define the worker pool specification\n", + "\n", + "Next, you define the worker pool specification for your custom training job. The worker pool specification will consist of the following:\n", + "\n", + "- `replica_count`: The number of instances to provision of this machine type.\n", + "- `machine_spec`: The hardware specification.\n", + "- `disk_spec` : (optional) The disk storage specification.\n", + "\n", + "- `python_package`: The Python training package to install on the VM instance(s) and which Python module to invoke, along with command line arguments for the Python module.\n", + "\n", + "Let's dive deeper now into the python package specification:\n", + "\n", + "-`executor_image_spec`: This is the docker image which is configured for your custom training job.\n", + "\n", + "-`package_uris`: This is a list of the locations (URIs) of your python training packages to install on the provisioned instance. The locations need to be in a Cloud Storage bucket. These can be either individual python files or a zip (archive) of an entire package. In the later case, the job service will unzip (unarchive) the contents into the docker image.\n", + "\n", + "-`python_module`: The Python module (script) to invoke for running the custom training job. In this example, you will be invoking `trainer.task.py` -- note that it was not neccessary to append the `.py` suffix.\n", + "\n", + "-`args`: The command line arguments to pass to the corresponding Pythom module. In this example, you will be setting:\n", + " - `\"--model-dir=\" + MODEL_DIR` : The Cloud Storage location where to store the model artifacts. There are two ways to tell the training script where to save the model artifacts:\n", + " - direct: You pass the Cloud Storage location as a command line argument to your training script (set variable `DIRECT = True`), or\n", + " - indirect: The service passes the Cloud Storage location as the environment variable `AIP_MODEL_DIR` to your training script (set variable `DIRECT = False`). In this case, you tell the service the model artifact location in the job specification.\n", + " - `\"--epochs=\" + EPOCHS`: The number of epochs for training.\n", + " - `\"--steps=\" + STEPS`: The number of steps (batches) per epoch.\n", + " - `\"--distribute=\" + TRAIN_STRATEGY\"` : The training distribution strategy to use for single or distributed training.\n", + " - `\"single\"`: single device.\n", + " - `\"mirror\"`: all GPU devices on a single compute instance.\n", + " - `\"multi\"`: all GPU devices on all compute instances." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "train_custom_job_worker_pool_specification:prebuilt_container", + "repo": "snippets_custom.ipynb" + }, + "outputs": [], + "source": [ + "JOB_NAME = \"custom_job_\" + TIMESTAMP\n", + "MODEL_DIR = '{}/{}'.format(BUCKET_NAME, JOB_NAME)\n", + "\n", + "if not TRAIN_NGPU or TRAIN_NGPU < 2:\n", + " TRAIN_STRATEGY = \"single\"\n", + "else:\n", + " TRAIN_STRATEGY = \"mirror\"\n", + "\n", + "EPOCHS = 20\n", + "STEPS = 100\n", + "\n", + "DIRECT = True\n", + "if DIRECT:\n", + " CMDARGS = [\n", + " \"--model-dir=\" + MODEL_DIR,\n", + " \"--epochs=\" + str(EPOCHS),\n", + " \"--steps=\" + str(STEPS),\n", + " \"--distribute=\" + TRAIN_STRATEGY\n", + " ]\n", + "else:\n", + " CMDARGS = [\n", + " \"--epochs=\" + str(EPOCHS),\n", + " \"--steps=\" + str(STEPS),\n", + " \"--distribute=\" + TRAIN_STRATEGY\n", + " ]\n", + "\n", + "worker_pool_spec = [\n", + " {\n", + " \"replica_count\": 1,\n", + " \"machine_spec\": machine_spec,\n", + " \"disk_spec\": disk_spec,\n", + " \"python_package_spec\": {\n", + " \"executor_image_uri\": TRAIN_IMAGE,\n", + " \"package_uris\": [BUCKET_NAME + \"/trainer_boston.tar.gz\"],\n", + " \"python_module\": \"trainer.task\",\n", + " \"args\": CMDARGS,\n", + " }\n", + " }\n", + "]" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "assemble_custom_job_specification", + "repo": "snippets_custom.ipynb" + }, + "source": [ + "### Assemble a job specification\n", + "\n", + "Now assemble the complete description for the custom job specification:\n", + "\n", + "- `display_name`: The human readable name you assign to this custom job.\n", + "- `job_spec`: The specification for the custom job.\n", + " - `worker_pool_specs`: The specification for the machine VM instances.\n", + " - `base_output_directory`: This tells the service the Cloud Storage location where to save the model artifacts (when variable `DIRECT = False`). The service will then pass the location to the training script as the environment variable `AIP_MODEL_DIR`, and the path will be of the form:\n", + "\n", + " /model" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "assemble_custom_job_specification", + "repo": "snippets_custom.ipynb" + }, + "outputs": [], + "source": [ + "if DIRECT:\n", + " job_spec = {\n", + " \"worker_pool_specs\": worker_pool_spec\n", + " }\n", + "else:\n", + " job_spec = {\n", + " \"worker_pool_specs\": worker_pool_spec,\n", + " \"base_output_directory\": {\"output_uri_prefix\": MODEL_DIR}\n", + " }\n", + "\n", + "custom_job = {\n", + " \"display_name\": JOB_NAME,\n", + " \"job_spec\": job_spec\n", + "}" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "examine_training_package", + "repo": "snippets_custom.ipynb" + }, + "source": [ + "### Examine the training package\n", + "\n", + "#### Package layout\n", + "\n", + "Before you start the training, you will look at how a Python package is assembled for a custom training job. When unarchived, the package contains the following directory/file layout.\n", + "\n", + "- PKG-INFO\n", + "- README.md\n", + "- setup.cfg\n", + "- setup.py\n", + "- trainer\n", + " - \\_\\_init\\_\\_.py\n", + " - task.py\n", + "\n", + "The files `setup.cfg` and `setup.py` are the instructions for installing the package into the operating environment of the Docker image.\n", + "\n", + "The file `trainer/task.py` is the Python script for executing the custom training job. *Note*, when we referred to it in the worker pool specification, we replace the directory slash with a dot (`trainer.task`) and dropped the file suffix (`.py`).\n", + "\n", + "#### Package Assembly\n", + "\n", + "In the following cells, you will assemble the training package." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "examine_training_package", + "repo": "snippets_custom.ipynb" + }, + "outputs": [], + "source": [ + "# Make folder for Python training script\n", + "! rm -rf custom\n", + "! mkdir custom\n", + "\n", + "# Add package information\n", + "! touch custom/README.md\n", + "\n", + "setup_cfg = \"[egg_info]\\n\\ntag_build =\\n\\ntag_date = 0\"\n", + "! echo \"$setup_cfg\" > custom/setup.cfg\n", + "\n", + "setup_py = \"import setuptools\\n\\nsetuptools.setup(\\n\\n install_requires=[\\n\\n 'tensorflow_datasets==1.3.0',\\n\\n ],\\n\\n packages=setuptools.find_packages())\"\n", + "! echo \"$setup_py\" > custom/setup.py\n", + "\n", + "pkg_info = \"Metadata-Version: 1.0\\n\\nName: Boston Housing tabular regression\\n\\nVersion: 0.0.0\\n\\nSummary: Demostration training script\\n\\nHome-page: www.google.com\\n\\nAuthor: Google\\n\\nAuthor-email: aferlitsch@google.com\\n\\nLicense: Public\\n\\nDescription: Demo\\n\\nPlatform: Vertex\"\n", + "! echo \"$pkg_info\" > custom/PKG-INFO\n", + "\n", + "# Make the training subfolder\n", + "! mkdir custom/trainer\n", + "! touch custom/trainer/__init__.py" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "taskpy_contents:boston", + "repo": "snippets_custom.ipynb" + }, + "source": [ + "#### Task.py contents\n", + "\n", + "In the next cell, you write the contents of the training script task.py. I won't go into detail, it's just there for you to browse. In summary:\n", + "\n", + "- Get the directory where to save the model artifacts from the command line (`--model_dir`), and if not specified, then from the environment variable `AIP_MODEL_DIR`.\n", + "- Loads Boston Housing dataset from TF.Keras builtin datasets\n", + "- Builds a simple deep neural network model using TF.Keras model API.\n", + "- Compiles the model (`compile()`).\n", + "- Sets a training distribution strategy according to the argument `args.distribute`.\n", + "- Trains the model (`fit()`) with epochs specified by `args.epochs`.\n", + "- Saves the trained model (`save(args.model_dir)`) to the specified model directory.\n", + "- Saves the maximum value for each feature `f.write(str(params))` to the specified parameters file." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "taskpy_contents:boston", + "repo": "snippets_custom.ipynb" + }, + "outputs": [], + "source": [ + "%%writefile custom/trainer/task.py\n", + "# Single, Mirror and Multi-Machine Distributed Training for Boston Housing\n", + "\n", + "import tensorflow_datasets as tfds\n", + "import tensorflow as tf\n", + "from tensorflow.python.client import device_lib\n", + "import numpy as np\n", + "import argparse\n", + "import os\n", + "import sys\n", + "tfds.disable_progress_bar()\n", + "\n", + "parser = argparse.ArgumentParser()\n", + "parser.add_argument('--model-dir', dest='model_dir',\n", + " default=os.getenv('AIP_MODEL_DIR'), type=str, help='Model dir.')\n", + "parser.add_argument('--lr', dest='lr',\n", + " default=0.001, type=float,\n", + " help='Learning rate.')\n", + "parser.add_argument('--epochs', dest='epochs',\n", + " default=20, type=int,\n", + " help='Number of epochs.')\n", + "parser.add_argument('--steps', dest='steps',\n", + " default=100, type=int,\n", + " help='Number of steps per epoch.')\n", + "parser.add_argument('--distribute', dest='distribute', type=str, default='single',\n", + " help='distributed training strategy')\n", + "parser.add_argument('--param-file', dest='param_file',\n", + " default='/tmp/param.txt', type=str,\n", + " help='Output file for parameters')\n", + "args = parser.parse_args()\n", + "\n", + "print('Python Version = {}'.format(sys.version))\n", + "print('TensorFlow Version = {}'.format(tf.__version__))\n", + "print('TF_CONFIG = {}'.format(os.environ.get('TF_CONFIG', 'Not found')))\n", + "\n", + "# Single Machine, single compute device\n", + "if args.distribute == 'single':\n", + " if tf.test.is_gpu_available():\n", + " strategy = tf.distribute.OneDeviceStrategy(device=\"/gpu:0\")\n", + " else:\n", + " strategy = tf.distribute.OneDeviceStrategy(device=\"/cpu:0\")\n", + "# Single Machine, multiple compute device\n", + "elif args.distribute == 'mirror':\n", + " strategy = tf.distribute.MirroredStrategy()\n", + "# Multiple Machine, multiple compute device\n", + "elif args.distribute == 'multi':\n", + " strategy = tf.distribute.experimental.MultiWorkerMirroredStrategy()\n", + "\n", + "# Multi-worker configuration\n", + "print('num_replicas_in_sync = {}'.format(strategy.num_replicas_in_sync))\n", + "\n", + "\n", + "def make_dataset():\n", + "\n", + " # Scaling Boston Housing data features\n", + " def scale(feature):\n", + " max = np.max(feature)\n", + " feature = (feature / max).astype(np.float)\n", + " return feature, max\n", + "\n", + " (x_train, y_train), (x_test, y_test) = tf.keras.datasets.boston_housing.load_data(\n", + " path=\"boston_housing.npz\", test_split=0.2, seed=113\n", + " )\n", + " params = []\n", + " for _ in range(13):\n", + " x_train[_], max = scale(x_train[_])\n", + " x_test[_], _ = scale(x_test[_])\n", + " params.append(max)\n", + "\n", + " # store the normalization (max) value for each feature\n", + " with tf.io.gfile.GFile(args.param_file, 'w') as f:\n", + " f.write(str(params))\n", + " return (x_train, y_train), (x_test, y_test)\n", + "\n", + "\n", + "# Build the Keras model\n", + "def build_and_compile_dnn_model():\n", + " model = tf.keras.Sequential([\n", + " tf.keras.layers.Dense(128, activation='relu', input_shape=(13,)),\n", + " tf.keras.layers.Dense(128, activation='relu'),\n", + " tf.keras.layers.Dense(1, activation='linear')\n", + " ])\n", + " model.compile(\n", + " loss='mse',\n", + " optimizer=tf.keras.optimizers.RMSprop(learning_rate=args.lr))\n", + " return model\n", + "\n", + "NUM_WORKERS = strategy.num_replicas_in_sync\n", + "# Here the batch size scales up by number of workers since\n", + "# `tf.data.Dataset.batch` expects the global batch size.\n", + "BATCH_SIZE = 16\n", + "GLOBAL_BATCH_SIZE = BATCH_SIZE * NUM_WORKERS\n", + "\n", + "with strategy.scope():\n", + " # Creation of dataset, and model building/compiling need to be within\n", + " # `strategy.scope()`.\n", + " model = build_and_compile_dnn_model()\n", + "\n", + "# Train the model\n", + "(x_train, y_train), (x_test, y_test) = make_dataset()\n", + "model.fit(x_train, y_train, epochs=args.epochs, batch_size=GLOBAL_BATCH_SIZE)\n", + "model.save(args.model_dir)" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "tarball_training_script", + "repo": "snippets_custom.ipynb" + }, + "source": [ + "#### Store training script on your Cloud Storage bucket\n", + "\n", + "Next, you package the training folder into a compressed tar ball, and then store it in your Cloud Storage bucket." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "tarball_training_script", + "repo": "snippets_custom.ipynb" + }, + "outputs": [], + "source": [ + "! rm -f custom.tar custom.tar.gz\n", + "! tar cvf custom.tar custom\n", + "! gzip custom.tar\n", + "! gsutil cp custom.tar.gz $BUCKET_NAME/trainer_boston.tar.gz" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "train_custom_job", + "repo": "snippets_custom.ipynb" + }, + "source": [ + "### Train the model\n", + "\n", + "\n", + "Now start the training of your custom training job on Vertex AI. Use this helper function `create_custom_job`, which takes the following parameter:\n", + "\n", + "-`custom_job`: The specification for the custom job.\n", + "\n", + "The helper function calls job client service's `create_custom_job` method, with the following parameters:\n", + "\n", + "-`parent`: The Vertex AI location path to `Dataset`, `Model` and `Endpoint` resources.\n", + "-`custom_job`: The specification for the custom job.\n", + "\n", + "You will display a handful of the fields returned in `response` object, with the two that are of most interest are:\n", + "\n", + "-`response.name`: The Vertex AI fully qualified identifier assigned to this custom training job. You save this identifier for using in subsequent steps.\n", + "\n", + "-`response.state`: The current state of the custom training job." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "train_custom_job", + "repo": "snippets_custom.ipynb" + }, + "outputs": [], + "source": [ + "def create_custom_job(custom_job):\n", + " response = clients['job'].create_custom_job(parent=PARENT, custom_job=custom_job)\n", + " print(\"name:\", response.name)\n", + " print(\"display_name:\", response.display_name)\n", + " print(\"state:\", response.state)\n", + " print(\"create_time:\", response.create_time)\n", + " print(\"update_time:\", response.update_time)\n", + " return response\n", + "\n", + "\n", + "response = create_custom_job(custom_job)" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "job_id:response", + "repo": "snippets_custom.ipynb" + }, + "source": [ + "Now get the unique identifier for the custom job you created." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "job_id:response", + "repo": "snippets_custom.ipynb" + }, + "outputs": [], + "source": [ + "# The full unique ID for the custom job\n", + "job_id = response.name\n", + "# The short numeric ID for the custom job\n", + "job_short_id = job_id.split('/')[-1]\n", + "\n", + "print(job_id)" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "get_custom_job", + "repo": "snippets_custom.ipynb" + }, + "source": [ + "### Get information on a custom job\n", + "\n", + "Next, use this helper function `get_custom_job`, which takes the following parameter:\n", + "\n", + "- `name`: The Vertex AI fully qualified identifier for the custom job.\n", + "\n", + "The helper function calls the job client service's `get_custom_job` method, with the following parameter:\n", + "\n", + "- `name`: The Vertex AI fully qualified identifier for the custom job.\n", + "\n", + "If you recall, you got the Vertex AI fully qualified identifier for the custom job in the `response.name` field when you called the `create_custom_job` method, and saved the identifier in the variable `job_id`." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "get_custom_job", + "repo": "snippets_custom.ipynb" + }, + "outputs": [], + "source": [ + "def get_custom_job(name, silent=False):\n", + " response = clients['job'].get_custom_job(name=name)\n", + " if silent:\n", + " return response\n", + "\n", + " print(\"name:\", response.name)\n", + " print(\"display_name:\", response.display_name)\n", + " print(\"state:\", response.state)\n", + " print(\"create_time:\", response.create_time)\n", + " print(\"update_time:\", response.update_time)\n", + " return response\n", + "\n", + "\n", + "response = get_custom_job(job_id)" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "wait_training_complete:custom", + "repo": "snippets_custom.ipynb" + }, + "source": [ + "# Deploy the model\n", + "\n", + "Training the above model may take upwards of 20 minutes time.\n", + "\n", + "Once your model is done training, you can calculate the actual time it took to train the model by subtracting `end_time` from `start_time`. For your model, we will need to know the location of the saved model, which the Python script saved in your local Cloud Storage bucket at `MODEL_DIR + '/saved_model.pb'`." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "wait_training_complete:custom", + "repo": "snippets_custom.ipynb" + }, + "outputs": [], + "source": [ + "while True:\n", + " response = get_custom_job(job_id, True)\n", + " if response.state != aip.JobState.JOB_STATE_SUCCEEDED:\n", + " print(\"Training job has not completed:\", response.state)\n", + " model_path_to_deploy = None\n", + " if response.state == aip.JobState.JOB_STATE_FAILED:\n", + " break\n", + " else:\n", + " if not DIRECT:\n", + " MODEL_DIR = MODEL_DIR + \"/model\"\n", + " model_path_to_deploy = MODEL_DIR\n", + " print(\"Training Job Time:\", response.end_time - response.start_time)\n", + " print(\"Training Elapsed Time:\", response.update_time - response.create_time)\n", + " break\n", + " time.sleep(60)\n", + "\n", + "print(\"model_to_deploy:\", model_path_to_deploy)" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "load_saved_model", + "repo": "snippets_custom.ipynb" + }, + "source": [ + "## Load the saved model\n", + "\n", + "Your model is stored in a TensorFlow SavedModel format in a Cloud Storage bucket. Now load it from the Cloud Storage bucket, and then you can do some things, like evaluate the model, and do a prediction.\n", + "\n", + "To load, you use the TF.Keras `model.load_model()` method passing it the Cloud Storage path where the model is saved -- specified by `MODEL_DIR`." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "load_saved_model", + "repo": "snippets_custom.ipynb" + }, + "outputs": [], + "source": [ + "import tensorflow as tf\n", + "\n", + "model = tf.keras.models.load_model(MODEL_DIR)" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "evaluate_custom_model:tabular", + "repo": "snippets_custom.ipynb" + }, + "source": [ + "## Evaluate the model\n", + "\n", + "Now let's find out how good the model is.\n", + "\n", + "### Load evaluation data\n", + "\n", + "You will load the Boston Housing test (holdout) data from `tf.keras.datasets`, using the method `load_data()`. This returns the dataset as a tuple of two elements. The first element is the training data and the second is the test data. Each element is also a tuple of two elements: the feature data, and the corresponding labels (median value of owner-occupied home).\n", + "\n", + "You don't need the training data, and hence why we loaded it as `(_, _)`.\n", + "\n", + "Before you can run the data through evaluation, you need to preprocess it:\n", + "\n", + "`x_test`:\n", + "1. Normalize (rescale) the data in each column by dividing each value by the maximum value of that column. This replaces each single value with a 32-bit floating point number between 0 and 1." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "evaluate_custom_model:tabular,boston", + "repo": "snippets_custom.ipynb" + }, + "outputs": [], + "source": [ + "from tensorflow.keras.datasets import boston_housing\n", + "import numpy as np\n", + "\n", + "(_, _), (x_test, y_test) = boston_housing.load_data(\n", + " path=\"boston_housing.npz\", test_split=0.2, seed=113)\n", + "\n", + "def scale(feature):\n", + " max = np.max(feature)\n", + " feature = (feature / max).astype(np.float32)\n", + " return feature\n", + "\n", + "# Let's save one data item that has not been scaled\n", + "x_test_notscaled = x_test[0:1].copy()\n", + "\n", + "for _ in range(13):\n", + " x_test[_] = scale(x_test[_])\n", + "x_test = x_test.astype(np.float32)\n", + "\n", + "print(x_test.shape, x_test.dtype, y_test.shape)\n", + "print(\"scaled\", x_test[0])\n", + "print(\"unscaled\", x_test_notscaled)" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "perform_evaluation_custom", + "repo": "snippets_custom.ipynb" + }, + "source": [ + "### Perform the model evaluation\n", + "\n", + "Now evaluate how well the model in the custom job did." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "perform_evaluation_custom", + "repo": "snippets_custom.ipynb" + }, + "outputs": [], + "source": [ + "model.evaluate(x_test, y_test)" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "how_serving_function_works", + "repo": "snippets_custom.ipynb" + }, + "source": [ + "## Upload the model for serving\n", + "\n", + "Next, you will upload your TF.Keras model from the custom job to Vertex `Model` service, which will create a Vertex `Model` resource for your custom model. During upload, you need to define a serving function to convert data to the format your model expects. If you send encoded data to Vertex AI, your serving function ensures that the data is decoded on the model server before it is passed as input to your model.\n", + "\n", + "### How does the serving function work\n", + "\n", + "When you send a request to an online prediction server, the request is received by a HTTP server. The HTTP server extracts the prediction request from the HTTP request content body. The extracted prediction request is forwarded to the serving function. For Google pre-built prediction containers, the request content is passed to the serving function as a `tf.string`.\n", + "\n", + "The serving function consists of two parts:\n", + "\n", + "- `preprocessing function`:\n", + " - Converts the input (`tf.string`) to the input shape and data type of the underlying model (dynamic graph).\n", + " - Performs the same preprocessing of the data that was done during training the underlying model, including normalizing and scalingh.\n", + "- `post-processing function`:\n", + " - Converts the model output to format expected by the receiving application -- e.q., compresses the output.\n", + " - Packages the output for the the receiving application, including adding headings, and making JSON objects.\n", + "\n", + "Both the preprocessing and post-processing functions are converted to static graphs which are fused to the model. The output from the underlying model is passed to the post-processing function. The post-processing function passes the converted/packaged output back to the HTTP server. The HTTP server returns the output as the HTTP response content.\n", + "\n", + "One consideration you need to consider when building serving functions for TF.Keras models is that they run as static graphs. That means, you cannot use TF graph operations that require a dynamic graph. If you do, you will get an error during the compile of the serving function which will indicate that you are using an EagerTensor which is not supported." + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "serving_function_signature:xai", + "repo": "snippets_custom.ipynb" + }, + "source": [ + "## Get the serving function signature\n", + "\n", + "You can get the signatures of your model's input and output layers by reloading the model into memory, and querying it for the signatures corresponding to each layer.\n", + "\n", + "When making a prediction request, you need to route the request to the serving function instead of the model, so you need to know the input layer name of the serving function -- which you will use later when you make a prediction request.\n", + "\n", + "You also need to know the name of the serving function's input and output layer for constructing the explanation metadata -- which is discussed subsequently." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "serving_function_signature:xai", + "repo": "snippets_custom.ipynb" + }, + "outputs": [], + "source": [ + "loaded = tf.saved_model.load(model_path_to_deploy)\n", + "\n", + "serving_input = list(loaded.signatures['serving_default'].structured_input_signature[1].keys())[0]\n", + "print('Serving function input:', serving_input)\n", + "serving_output = list(loaded.signatures['serving_default'].structured_outputs.keys())[0]\n", + "print('Serving function output:', serving_output)\n", + "\n", + "input_name = model.input.name\n", + "print('Model input name:', input_name)\n", + "output_name = model.output.name\n", + "print('Model output name:', output_name)" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "explanation_spec", + "repo": "snippets_custom.ipynb" + }, + "source": [ + "### Explanation Specification\n", + "\n", + "To get explanations when doing a prediction, you must enable the explanation capability and set corresponding settings when you upload your custom model to a Vertex `Model` resource. These settings are referred to as the explanation metadata, which consists of:\n", + "\n", + "- `parameters`: This is the specification for the explainability algorithm to use for explanations on your model. You can choose between:\n", + " - Shapley - *Note*, not recommended for image data -- can be very long running\n", + " - XRAI\n", + " - Integrated Gradients\n", + "- `metadata`: This is the specification for how the algoithm is applied on your custom model.\n", + "\n", + "#### Explanation Parameters\n", + "\n", + "Let's first dive deeper into the settings for the explainability algorithm.\n", + "\n", + "#### Shapley\n", + "\n", + "Assigns credit for the outcome to each feature, and considers different permutations of the features. This method provides a sampling approximation of exact Shapley values.\n", + "\n", + "Use Cases:\n", + " - Classification and regression on tabular data.\n", + "\n", + "Parameters:\n", + "\n", + "- `path_count`: This is the number of paths over the features that will be processed by the algorithm. An exact approximation of the Shapley values requires M! paths, where M is the number of features. For the CIFAR10 dataset, this would be 784 (28*28).\n", + "\n", + "For any non-trival number of features, this is too compute expensive. You can reduce the number of paths over the features to M * `path_count`.\n", + "\n", + "#### Integrated Gradients\n", + "\n", + "A gradients-based method to efficiently compute feature attributions with the same axiomatic properties as the Shapley value.\n", + "\n", + "Use Cases:\n", + " - Classification and regression on tabular data.\n", + " - Classification on image data.\n", + "\n", + "Parameters:\n", + "\n", + "- `step_count`: This is the number of steps to approximate the remaining sum. The more steps, the more accurate the integral approximation. The general rule of thumb is 50 steps, but as you increase so does the compute time.\n", + "\n", + "#### XRAI\n", + "\n", + "Based on the integrated gradients method, XRAI assesses overlapping regions of the image to create a saliency map, which highlights relevant regions of the image rather than pixels.\n", + "\n", + "Use Cases:\n", + "\n", + " - Classification on image data.\n", + "\n", + "Parameters:\n", + "\n", + "- `step_count`: This is the number of steps to approximate the remaining sum. The more steps, the more accurate the integral approximation. The general rule of thumb is 50 steps, but as you increase so does the compute time.\n", + "\n", + "In the next code cell, set the variable `XAI` to which explainabilty algorithm you will use on your custom model." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "explanation_spec", + "repo": "snippets_custom.ipynb" + }, + "outputs": [], + "source": [ + "XAI = \"ig\" # [ shapley, ig, xrai ]\n", + "\n", + "if XAI == \"shapley\":\n", + " PARAMETERS = {\n", + " \"sampled_shapley_attribution\": {'path_count': 10}\n", + " }\n", + "elif XAI == \"ig\":\n", + " PARAMETERS = {\n", + " \"integrated_gradients_attribution\": {\"step_count\": 50}\n", + " }\n", + "elif XAI == \"xrai\":\n", + " PARAMETERS = {\n", + " \"xrai_attribution\": {\"step_count\": 50}\n", + " }\n", + "\n", + "parameters = aip.ExplanationParameters(PARAMETERS)" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "explanation_metadata:tabular", + "repo": "snippets_custom.ipynb" + }, + "source": [ + "#### Explanation Metadata\n", + "\n", + "Let's first dive deeper into the explanation metadata, which consists of:\n", + "\n", + "- `outputs`: A scalar value in the output to attribute -- what to explain. For example, in a probability output \\[0.1, 0.2, 0.7\\] for classification, one wants an explanation for 0.7. Consider the following formulae, where the output is `y` and that is what we want to explain.\n", + "\n", + " y = f(x)\n", + "\n", + "Consider the following formulae, where the outputs are `y` and `z`. Since we can only do attribution for one scalar value, we have to pick whether we want to explain the output `y` or `z`. Assume in this example the model is object detection and y and z are the bounding box and the object classification. You would want to pick which of the two outputs to explain.\n", + "\n", + " y, z = f(x)\n", + "\n", + "The dictionary format for `outputs` is:\n", + "\n", + " { \"outputs\": { \"[your_display_name]\":\n", + " \"output_tensor_name\": [layer]\n", + " }\n", + " }\n", + "\n", + "
\n", + "
    \n", + "
  • [your_display_name]: A human readable name you assign to the output to explain. A common example is \"probability\".
    • \n", + "
  • \"output_tensor_name\": The key/value field to identify the output layer to explain.
    • \n", + "
  • [layer]: The output layer to explain. In a single task model, like a tabular regressor, it is the last (topmost) layer in the model
    • .\n", + "
\n", + "
\n", + "\n", + "\n", + "- `inputs`: The features for attribution -- how they contributed to the output. Consider the following formulae, where `a` and `b` are the features. We have to pick which features to explain how the contributed. Assume that this model is deployed for A/B testing, where `a` are the data_items for the prediction and `b` identifies whether the model instance is A or B. You would want to pick `a` (or some subset of) for the features, and not `b` since it does not contribute to the prediction.\n", + "\n", + " y = f(a,b)\n", + "\n", + "The minimum dictionary format for `inputs` is:\n", + "\n", + " { \"inputs\": { \"[your_display_name]\":\n", + " \"input_tensor_name\": [layer]\n", + " }\n", + " }\n", + "\n", + "
\n", + "
    \n", + "
  • [your_display_name]: A human readable name you assign to the input to explain. A common example is \"features\".
    • \n", + "
  • \"input_tensor_name\": The key/value field to identify the input layer for the feature attribution.
    • \n", + "
  • [layer]: The input layer for feature attribution. In a single input tensor model, it is the first (bottom-most) layer in the model.
    • \n", + "
\n", + "
\n", + "\n", + "Since the inputs to the model are tabular, you can specify the following two additional fields as reporting/visualization aids:\n", + "\n", + "
\n", + "
    \n", + "
  • \"encoding\": \"BAG_OF_FEATURES\" : Indicates that the inputs are set of tabular features.
    • \n", + "
  • \"index_feature_mapping\": [ feature-names ] : A list of human readable names for each feature. For this example, we use the feature names specified in the dataset.
    • \n", + "
  • \"modality\": \"numeric\": Indicates the field values are numeric.
    • \n", + "
\n", + "
" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "explanation_metadata:tabular", + "repo": "snippets_custom.ipynb" + }, + "outputs": [], + "source": [ + "INPUT_METADATA = {'input_tensor_name': serving_input,\n", + " \"encoding\": \"BAG_OF_FEATURES\",\n", + " \"modality\": \"numeric\",\n", + " \"index_feature_mapping\": [\"crim\", \"zn\", \"indus\", \"chas\", \"nox\", \"rm\", \"age\",\n", + " \"dis\", \"rad\", \"tax\", \"ptratio\", \"b\", \"lstat\"]\n", + "}\n", + "\n", + "OUTPUT_METADATA = {'output_tensor_name': serving_output}\n", + "\n", + "input_metadata = aip.ExplanationMetadata.InputMetadata(INPUT_METADATA)\n", + "output_metadata = aip.ExplanationMetadata.OutputMetadata(OUTPUT_METADATA)\n", + "\n", + "metadata = aip.ExplanationMetadata(\n", + " inputs = {'features': input_metadata},\n", + " outputs = {'medv' : output_metadata}\n", + ")\n", + "\n", + "explanation_spec = aip.ExplanationSpec(\n", + " metadata=metadata,\n", + " parameters=parameters\n", + ")" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "upload_the_model:explanation", + "repo": "snippets_custom.ipynb" + }, + "source": [ + "### Upload the model\n", + "\n", + "Use this helper function `upload_model` to upload your model, stored in SavedModel format, up to the `Model` service, which will instantiate a Vertex `Model` resource instance for your model. Once you've done that, you can use the `Model` resource instance in the same way as any other Vertex `Model` resource instance, such as deploying to an `Endpoint` resource for serving predictions.\n", + "\n", + "The helper function takes the following parameters:\n", + "\n", + "- `display_name`: A human readable name for the `Endpoint` service.\n", + "- `image_uri`: The container image for the model deployment.\n", + "- `model_uri`: The Cloud Storage path to our SavedModel artificat. For this tutorial, this is the Cloud Storage location where the `trainer/task.py` saved the model artifacts, which we specified in the variable `MODEL_DIR`.\n", + "\n", + "The helper function calls the `Model` client service's method `upload_model`, which takes the following parameters:\n", + "\n", + "- `parent`: The Vertex AI location root path for `Dataset`, `Model` and `Endpoint` resources.\n", + "- `model`: The specification for the Vertex `Model` resource instance.\n", + "\n", + "Let's now dive deeper into the Vertex model specification `model`. This is a dictionary object that consists of the following fields:\n", + "\n", + "- `display_name`: A human readable name for the `Model` resource.\n", + "- `metadata_schema_uri`: Since your model was built without an Vertex `Dataset` resource, you will leave this blank (`''`).\n", + "- `artifact_uri`: The Cloud Storage path where the model is stored in SavedModel format.\n", + "- `container_spec`: This is the specification for the Docker container that will be installed on the `Endpoint` resource, from which the `Model` resource will serve predictions. Use the variable you set earlier `DEPLOY_GPU != None` to use a GPU; otherwise only a CPU is allocated.\n", + "- `explanation_spec`: This is the specification for enabling explainability for your model.\n", + "\n", + "Uploading a model into a Vertex `Model` resource returns a long running operation, since it may take a few moments. You call response.result(), which is a synchronous call and will return when the Vertex Model resource is ready.\n", + "\n", + "The helper function returns the Vertex AI fully qualified identifier for the corresponding Vertex `Model` instance upload_model_response.model. You will save the identifier for subsequent steps in the variable model_to_deploy_id." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "upload_the_model:explanation", + "repo": "snippets_custom.ipynb" + }, + "outputs": [], + "source": [ + "IMAGE_URI = DEPLOY_IMAGE\n", + "\n", + "\n", + "def upload_model(display_name, image_uri, model_uri):\n", + "\n", + " model = aip.Model(display_name=display_name,\n", + " artifact_uri=model_uri,\n", + " metadata_schema_uri=\"\",\n", + " explanation_spec=explanation_spec,\n", + " container_spec={\n", + " \"image_uri\": image_uri\n", + " })\n", + "\n", + " response = clients['model'].upload_model(parent=PARENT, model=model)\n", + " print(\"Long running operation:\", response.operation.name)\n", + " upload_model_response = response.result(timeout=180)\n", + " print(\"upload_model_response\")\n", + " print(\" model:\", upload_model_response.model)\n", + " return upload_model_response.model\n", + "\n", + "\n", + "model_to_deploy_id = upload_model(\"boston-\" + TIMESTAMP, IMAGE_URI, model_path_to_deploy)" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "get_model", + "repo": "snippets_common.ipynb" + }, + "source": [ + "### Get `Model` resource information\n", + "\n", + "Now let's get the model information for just your model. Use this helper function `get_model`, with the following parameter:\n", + "\n", + "- `name`: The Vertex AI unique identifier for the `Model` resource.\n", + "\n", + "This helper function calls the Vertex AI `Model` client service's method `get_model`, with the following parameter:\n", + "\n", + "- `name`: The Vertex AI unique identifier for the `Model` resource." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "get_model", + "repo": "snippets_common.ipynb" + }, + "outputs": [], + "source": [ + "def get_model(name):\n", + " response = clients['model'].get_model(name=name)\n", + " print(response)\n", + "\n", + "\n", + "get_model(model_to_deploy_id)" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "deploy:batch_prediction", + "repo": "snippets_common.ipynb" + }, + "source": [ + "## Model deployment for batch prediction\n", + "\n", + "Now deploy the trained Vertex `Model` resource you created for batch prediction. This differs from deploying a `Model` resource for online prediction.\n", + "\n", + "For online prediction, you:\n", + "\n", + "1. Create an `Endpoint` resource for deploying the `Model` resource to.\n", + "\n", + "2. Deploy the `Model` resource to the `Endpoint` resource.\n", + "\n", + "3. Make online prediction requests to the `Endpoint` resource.\n", + "\n", + "For batch-prediction, you:\n", + "\n", + "1. Create a batch prediction job.\n", + "\n", + "2. The job service will provision resources for the batch prediction request.\n", + "\n", + "3. The results of the batch prediction request are returned to the caller.\n", + "\n", + "4. The job service will unprovision the resoures for the batch prediction request." + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "make_prediction", + "repo": "snippets_common.ipynb" + }, + "source": [ + "## Send a batch prediction request\n", + "\n", + "Send a batch prediction to your deployed model." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "get_test_items:test,tabular", + "repo": "snippets_common.ipynb" + }, + "outputs": [], + "source": [ + "test_item_1 = x_test[0]\n", + "test_label_1 = y_test[0]\n", + "test_item_2 = x_test[1]\n", + "test_label_2 = y_test[1]\n", + "print(test_item_1.shape)" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "make_batch_file:custom,tabular", + "repo": "snippets_custom.ipynb" + }, + "source": [ + "### Make the batch input file\n", + "\n", + "Now make a batch input file, which you will store in your local Cloud Storage bucket. Each instance in the prediction request is a dictionary entry of the form:\n", + "\n", + " {serving_input: content}\n", + "\n", + "- `serving_input`: the name of the input layer of the underlying model.\n", + "- `content`: The feature values of the test item as a list." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "make_batch_file:custom,tabular", + "repo": "snippets_custom.ipynb" + }, + "outputs": [], + "source": [ + "import json\n", + "\n", + "\n", + "gcs_input_uri = BUCKET_NAME + \"/\" + \"test.jsonl\"\n", + "with tf.io.gfile.GFile(gcs_input_uri, 'w') as f:\n", + " data = {serving_input: test_item_1.tolist()}\n", + " f.write(json.dumps(data) + '\\n')\n", + " data = {serving_input: test_item_2.tolist()}\n", + " f.write(json.dumps(data) + '\\n')" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "instance_scaling", + "repo": "snippets_common.ipynb" + }, + "source": [ + "### Compute instance scaling\n", + "\n", + "You have several choices on scaling the compute instances for handling your batch prediction requests:\n", + "\n", + "- Single Instance: The batch prediction requests are processed on a single compute instance.\n", + " - Set the minimum (`MIN_NODES`) and maximum (`MAX_NODES`) number of compute instances to one.\n", + "\n", + "- Manual Scaling: The batch prediction requests are split across a fixed number of compute instances that you manually specified.\n", + " - Set the minimum (`MIN_NODES`) and maximum (`MAX_NODES`) number of compute instances to the same number of nodes. When a model is first deployed to the instance, the fixed number of compute instances are provisioned and batch prediction requests are evenly distributed across them.\n", + "\n", + "- Auto Scaling: The batch prediction requests are split across a scaleable number of compute instances.\n", + " - Set the minimum (`MIN_NODES`) number of compute instances to provision when a model is first deployed and to de-provision, and set the maximum (`MAX_NODES) number of compute instances to provision, depending on load conditions.\n", + "\n", + "The minimum number of compute instances corresponds to the field `min_replica_count` and the maximum number of compute instances corresponds to the field `max_replica_count`, in your subsequent deployment request." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "instance_scaling", + "repo": "snippets_common.ipynb" + }, + "outputs": [], + "source": [ + "MIN_NODES = 1\n", + "MAX_NODES = 1" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "make_batch_request:custom", + "repo": "snippets_custom.ipynb" + }, + "source": [ + "### Make batch prediction request\n", + "\n", + "Now that your batch of two test items is ready, let's do the batch request. Use this helper function `create_batch_prediction_job`, with the following parameters:\n", + "\n", + "- `display_name`: The human readable name for the prediction job.\n", + "- `model_name`: The Vertex AI fully qualified identifier for the `Model` resource.\n", + "- `gcs_source_uri`: The Cloud Storage path to the input file -- which you created above.\n", + "- `gcs_destination_output_uri_prefix`: The Cloud Storage path that the service will write the predictions to.\n", + "- `parameters`: Additional filtering parameters for serving prediction results.\n", + "\n", + "The helper function calls the job client service's `create_batch_prediction_job` metho, with the following parameters:\n", + "\n", + "- `parent`: The Vertex AI location root path for Dataset, Model and Pipeline resources.\n", + "- `batch_prediction_job`: The specification for the batch prediction job.\n", + "\n", + "Let's now dive into the specification for the `batch_prediction_job`:\n", + "\n", + "- `display_name`: The human readable name for the prediction batch job.\n", + "- `model`: The Vertex AI fully qualified identifier for the `Model` resource.\n", + "- `dedicated_resources`: The compute resources to provision for the batch prediction job.\n", + " - `machine_spec`: The compute instance to provision. Use the variable you set earlier `DEPLOY_GPU != None` to use a GPU; otherwise only a CPU is allocated.\n", + " - `starting_replica_count`: The number of compute instances to initially provision, which you set earlier as the variable `MIN_NODES`.\n", + " - `max_replica_count`: The maximum number of compute instances to scale to, which you set earlier as the variable `MAX_NODES`.\n", + "- `model_parameters`: Additional filtering parameters for serving prediction results. No Additional parameters are supported for custom models.\n", + "- `input_config`: The input source and format type for the instances to predict.\n", + " - `instances_format`: The format of the batch prediction request file: `csv` or `jsonl`.\n", + " - `gcs_source`: A list of one or more Cloud Storage paths to your batch prediction requests.\n", + "- `output_config`: The output destination and format for the predictions.\n", + " - `prediction_format`: The format of the batch prediction response file: `csv` or `jsonl`.\n", + " - `gcs_destination`: The output destination for the predictions.\n", + "\n", + "This call is an asychronous operation. You will print from the response object a few select fields, including:\n", + "\n", + "- `name`: The Vertex AI fully qualified identifier assigned to the batch prediction job.\n", + "- `display_name`: The human readable name for the prediction batch job.\n", + "- `model`: The Vertex AI fully qualified identifier for the Model resource.\n", + "- `generate_explanations`: Whether True/False explanations were provided with the predictions (explainability).\n", + "- `state`: The state of the prediction job (pending, running, etc).\n", + "\n", + "Since this call will take a few moments to execute, you will likely get `JobState.JOB_STATE_PENDING` for `state`." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "make_batch_request:custom", + "repo": "snippets_custom.ipynb" + }, + "outputs": [], + "source": [ + "BATCH_MODEL = \"boston_batch-\" + TIMESTAMP\n", + "\n", + "\n", + "def create_batch_prediction_job(display_name, model_name, gcs_source_uri, gcs_destination_output_uri_prefix, parameters=None):\n", + "\n", + " if DEPLOY_GPU:\n", + " machine_spec = {\n", + " \"machine_type\": DEPLOY_COMPUTE,\n", + " \"accelerator_type\": DEPLOY_GPU,\n", + " \"accelerator_count\": DEPLOY_NGPU,\n", + " }\n", + " else:\n", + " machine_spec = {\n", + " \"machine_type\": DEPLOY_COMPUTE,\n", + " \"accelerator_count\": 0,\n", + " }\n", + "\n", + " batch_prediction_job = {\n", + " \"display_name\": display_name,\n", + " # Format: 'projects/{project}/locations/{location}/models/{model_id}'\n", + " \"model\": model_name,\n", + " \"model_parameters\": json_format.ParseDict(parameters, Value()),\n", + " \"input_config\": {\n", + " \"instances_format\": IN_FORMAT,\n", + " \"gcs_source\": {\"uris\": [gcs_source_uri]},\n", + " },\n", + " \"output_config\": {\n", + " \"predictions_format\": OUT_FORMAT,\n", + " \"gcs_destination\": {\"output_uri_prefix\": gcs_destination_output_uri_prefix},\n", + " },\n", + " \"dedicated_resources\": {\n", + " \"machine_spec\": machine_spec,\n", + " \"starting_replica_count\": MIN_NODES,\n", + " \"max_replica_count\": MAX_NODES\n", + " }\n", + " , 'generate_explanation': True\n", + " }\n", + " response = clients['job'].create_batch_prediction_job(\n", + " parent=PARENT, batch_prediction_job=batch_prediction_job\n", + " )\n", + " print(\"response\")\n", + " print(\" name:\", response.name)\n", + " print(\" display_name:\", response.display_name)\n", + " print(\" model:\", response.model)\n", + " try:\n", + " print(\" generate_explanation:\", response.generate_explanation)\n", + " except:\n", + " pass\n", + " print(\" state:\", response.state)\n", + " print(\" create_time:\", response.create_time)\n", + " print(\" start_time:\", response.start_time)\n", + " print(\" end_time:\", response.end_time)\n", + " print(\" update_time:\", response.update_time)\n", + " print(\" labels:\", response.labels)\n", + " return response\n", + "\n", + "\n", + "IN_FORMAT = 'jsonl'\n", + "OUT_FORMAT = 'jsonl'\n", + "\n", + "response = create_batch_prediction_job(BATCH_MODEL, model_to_deploy_id, gcs_input_uri, BUCKET_NAME)" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "batch_job_id:response", + "repo": "snippets_common.ipynb" + }, + "source": [ + "Now get the unique identifier for the batch prediction job you created." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "batch_job_id:response", + "repo": "snippets_common.ipynb" + }, + "outputs": [], + "source": [ + "# The full unique ID for the batch job\n", + "batch_job_id = response.name\n", + "# The short numeric ID for the batch job\n", + "batch_job_short_id = batch_job_id.split('/')[-1]\n", + "\n", + "print(batch_job_id)" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "get_batch_prediction_job", + "repo": "snippets_common.ipynb" + }, + "source": [ + "### Get information on a batch prediction job\n", + "\n", + "Use this helper function `get_batch_prediction_job`, with the following paramter:\n", + "\n", + "- `job_name`: The Vertex AI fully qualified identifier for the batch prediction job.\n", + "\n", + "The helper function calls the job client service's `get_batch_prediction_job` method, with the following paramter:\n", + "\n", + "- `name`: The Vertex AI fully qualified identifier for the batch prediction job. In this tutorial, you will pass it the Vertex fully qualified identifier for your batch prediction job -- `batch_job_id`\n", + "\n", + "The helper function will return the Cloud Storage path to where the predictions are stored -- `gcs_destination`." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "get_batch_prediction_job", + "repo": "snippets_common.ipynb" + }, + "outputs": [], + "source": [ + "def get_batch_prediction_job(job_name, silent=False):\n", + " response = clients['job'].get_batch_prediction_job(name=job_name)\n", + " if silent:\n", + " return response.output_config.gcs_destination.output_uri_prefix, response.state\n", + "\n", + " print(\"response\")\n", + " print(\" name:\", response.name)\n", + " print(\" display_name:\", response.display_name)\n", + " print(\" model:\", response.model)\n", + " try: # not all data types support explanations\n", + " print(\" generate_explanation:\", response.generate_explanation)\n", + " except:\n", + " pass\n", + " print(\" state:\", response.state)\n", + " print(\" error:\", response.error)\n", + " gcs_destination = response.output_config.gcs_destination\n", + " print(\" gcs_destination\")\n", + " print(\" output_uri_prefix:\", gcs_destination.output_uri_prefix)\n", + " return gcs_destination.output_uri_prefix, response.state\n", + "\n", + "\n", + "predictions, state = get_batch_prediction_job(batch_job_id)" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "get_the_explanations:custom,tabular", + "repo": "snippets_custom.ipynb" + }, + "source": [ + "### Get the predictions\n", + "\n", + "When the batch prediction is done processing, the job state will be `JOB_STATE_SUCCEEDED`.\n", + "\n", + "Finally you view the predictions stored at the Cloud Storage path you set as output. The predictions will be in a JSONL format, which you indicated at the time you made the batch prediction job. The predictions are in a subdirectory starting with the name `prediction`. Within that subdirectory there is a file named `prediction.results-xxxxx-of-xxxxx`.\n", + "\n", + "Now display (cat) the contents. You will see multiple JSON objects, one for each prediction.\n", + "\n", + "Finally you view the explanations stored at the Cloud Storage path you set as output. The explanations will be in a JSONL format, which you indicated at the time you made the batch explanation job. The explanations are in a subdirectory starting with the name `prediction`. Within that subdirectory there is a file named `explanations.results-xxxxx-of-xxxxx`.\n", + "\n", + "Let's display (cat) the contents. You will a row for each prediction -- in this case, there is just one row. The row contains:\n", + "\n", + "- `dense_input`: The input for the prediction.\n", + "- `prediction`: The predicted value." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "get_the_explanations:custom,tabular", + "repo": "snippets_custom.ipynb" + }, + "outputs": [], + "source": [ + "def get_latest_predictions(gcs_out_dir):\n", + " ''' Get the latest prediction subfolder using the timestamp in the subfolder name'''\n", + " folders = !gsutil ls $gcs_out_dir\n", + " latest = \"\"\n", + " for folder in folders:\n", + " subfolder = folder.split('/')[-2]\n", + " if subfolder.startswith('prediction-'):\n", + " if subfolder > latest:\n", + " latest = folder[:-1]\n", + " return latest\n", + "\n", + "while True:\n", + " predictions, state = get_batch_prediction_job(batch_job_id, True)\n", + " if state != aip.JobState.JOB_STATE_SUCCEEDED:\n", + " print(\"The job has not completed:\", state)\n", + " if state == aip.JobState.JOB_STATE_FAILED:\n", + " break\n", + " else:\n", + " folder = get_latest_predictions(predictions)\n", + " ! gsutil ls $folder/explanation.results*\n", + "\n", + " print(\"Results:\")\n", + " ! gsutil cat $folder/explanation.results*\n", + "\n", + " print(\"Errors:\")\n", + " ! gsutil cat $folder/prediction.errors*\n", + " break\n", + " time.sleep(60)" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "cleanup", + "repo": "snippets_common.ipynb" + }, + "source": [ + "# Cleaning up\n", + "\n", + "To clean up all Google Cloud resources used in this project, you can [delete the GCP\n", + "project](https://cloud.google.com/resource-manager/docs/creating-managing-projects#shutting_down_projects) you used for the tutorial.\n", + "\n", + "Otherwise, you can delete the individual resources you created in this tutorial:\n", + "\n", + "- Dataset\n", + "- Pipeline\n", + "- Model\n", + "- Endpoint\n", + "- Batch Job\n", + "- Custom Job\n", + "- Hyperparameter Tuning Job\n", + "- Cloud Storage Bucket" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "cleanup", + "repo": "snippets_common.ipynb" + }, + "outputs": [], + "source": [ + "delete_dataset = True\n", + "delete_pipeline = True\n", + "delete_model = True\n", + "delete_endpoint = True\n", + "delete_batchjob = True\n", + "delete_customjob = True\n", + "delete_hptjob = True\n", + "delete_bucket = True\n", + "\n", + "# Delete the dataset using the Vertex fully qualified identifier for the dataset\n", + "try:\n", + " if delete_dataset and 'dataset_id' in globals():\n", + " clients['dataset'].delete_dataset(name=dataset_id)\n", + "except Exception as e:\n", + " print(e)\n", + "\n", + "# Delete the training pipeline using the Vertex fully qualified identifier for the pipeline\n", + "try:\n", + " if delete_pipeline and 'pipeline_id' in globals():\n", + " clients['pipeline'].delete_training_pipeline(name=pipeline_id)\n", + "except Exception as e:\n", + " print(e)\n", + "\n", + "# Delete the model using the Vertex fully qualified identifier for the model\n", + "try:\n", + " if delete_model and 'model_to_deploy_id' in globals():\n", + " clients['model'].delete_model(name=model_to_deploy_id)\n", + "except Exception as e:\n", + " print(e)\n", + "\n", + "# Delete the endpoint using the Vertex fully qualified identifier for the endpoint\n", + "try:\n", + " if delete_endpoint and 'endpoint_id' in globals():\n", + " clients['endpoint'].delete_endpoint(name=endpoint_id)\n", + "except Exception as e:\n", + " print(e)\n", + "\n", + "# Delete the batch job using the Vertex fully qualified identifier for the batch job\n", + "try:\n", + " if delete_batchjob and 'batch_job_id' in globals():\n", + " clients['job'].delete_batch_prediction_job(name=batch_job_id)\n", + "except Exception as e:\n", + " print(e)\n", + "\n", + "# Delete the custom job using the Vertex fully qualified identifier for the custom job\n", + "try:\n", + " if delete_customjob and 'job_id' in globals():\n", + " clients['job'].delete_custom_job(name=job_id)\n", + "except Exception as e:\n", + " print(e)\n", + "\n", + "# Delete the hyperparameter tuning job using the Vertex fully qualified identifier for the hyperparameter tuning job\n", + "try:\n", + " if delete_hptjob and 'hpt_job_id' in globals():\n", + " clients['job'].delete_hyperparameter_tuning_job(name=hpt_job_id)\n", + "except Exception as e:\n", + " print(e)\n", + "\n", + "if delete_bucket and 'BUCKET_NAME' in globals():\n", + " ! gsutil rm -r $BUCKET_NAME" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [] + } + ], + "metadata": { + "environment": { + "name": "common-cu110.m71", + "type": "gcloud", + "uri": "gcr.io/deeplearning-platform-release/base-cu110:m71" + }, + "kernelspec": { + "display_name": "Python 3", + "language": "python", + "name": "python3" + }, + "language_info": { + "codemirror_mode": { + "name": "ipython", + "version": 3 + }, + "file_extension": ".py", + "mimetype": "text/x-python", + "name": "python", + "nbconvert_exporter": "python", + "pygments_lexer": "ipython3", + "version": "3.7.10" + }, + "vars": { + "AIP": "AI Platform", + "AUTOML": "AutoML", + "AUTOML_URL": "https://cloud.google.com/vertex-ai/docs/start/automl-users", + "BATCHJOB_WAIT": "60", + "BATCH_OR_ONLINE": "batch", + "BQ": "BigQuery", + "CLOUD_SDK": "Cloud SDK", + "COLAB": "Colab", + "DATASET_ALIAS": "boston", + "DATASET_NAME": "Boston Housing", + "DOCKER": "Docker", + "EDGE": "Edge", + "EMAIL": "cdpe@gmail.com", + "EXPLAIN_BATCH": ", 'generate_explanation': True", + "GAPIC": "client library", + "GAPICP": "client library for Python", + "GCONSOLE": "Cloud Console", + "GCP": "Google Cloud", + "GCS": "Cloud Storage", + "GNOTEBOOK": "Google Cloud Notebook", + "MODEL_TYPE": "tabular regression", + "NOTEBOOK": "showcase_custom_tabular_regression_batch_explain.ipynb", + "REGION": "us-central1", + "REPO": "ai-platform-unified/notebooks/community/gapic/custom", + "TENSORFLOW": "TensorFlow", + "TFLite": "TFLite", + "TFServing": "TF Serving", + "TFV": "2-1", + "TITLE": "Custom training tabular regression model for batch prediction with explanation", + "TRAINING": "training", + "TRAINING_TIME": "20", + "TRAINING_WAIT": "60", + "VERTEX_AI": "Vertex AI", + "YEAR": "2020", + "null": "null", + "uCAIP": "Vertex" + } + }, + "nbformat": 4, + "nbformat_minor": 4 +} diff --git a/notebooks/official/explainable_ai/gapic-custom_tabular_regression_online_explain.ipynb b/notebooks/official/explainable_ai/gapic-custom_tabular_regression_online_explain.ipynb new file mode 100644 index 000000000..87a520dc3 --- /dev/null +++ b/notebooks/official/explainable_ai/gapic-custom_tabular_regression_online_explain.ipynb @@ -0,0 +1,2392 @@ +{ + "cells": [ + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "copyright" + }, + "outputs": [], + "source": [ + "# Copyright 2020 Google LLC\n", + "#\n", + "# Licensed under the Apache License, Version 2.0 (the \"License\");\n", + "# you may not use this file except in compliance with the License.\n", + "# You may obtain a copy of the License at\n", + "#\n", + "# https://www.apache.org/licenses/LICENSE-2.0\n", + "#\n", + "# Unless required by applicable law or agreed to in writing, software\n", + "# distributed under the License is distributed on an \"AS IS\" BASIS,\n", + "# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n", + "# See the License for the specific language governing permissions and\n", + "# limitations under the License." + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "title" + }, + "source": [ + "# Vertex client library for Python: Custom training tabular regression model for online prediction with explanation\n", + "\n", + "\n", + " \n", + " \n", + "
\n", + " \n", + " \"Colab Run in Colab\n", + " \n", + " \n", + " \n", + " \"GitHub\n", + " View on GitHub\n", + " \n", + "
\n", + "


" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "overview:custom,xai" + }, + "source": [ + "## Overview\n", + "\n", + "\n", + "This tutorial demonstrates how to use the Vertex client library for Python to train and deploy a custom tabular regression model for online prediction with explanation." + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "dataset:custom,boston,lrg" + }, + "source": [ + "### Dataset\n", + "\n", + "The dataset used for this tutorial is the [Boston Housing Prices dataset](https://www.cs.toronto.edu/~delve/data/boston/bostonDetail.html). The version of the dataset you will use in this tutorial is built into TensorFlow. The trained model predicts the median price of a house in units of 1K USD." + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "objective:custom,training,online_prediction,xai" + }, + "source": [ + "### Objective\n", + "\n", + "In this tutorial, you create a custom trained model from a Python script in a Google prebuilt Docker container using the Vertex client library for Python, and then do a prediction with explanations on the deployed model by sending data. Alternatively, you can create custom trained models using `gcloud` command-line tool or online using Cloud Console.\n", + "\n", + "The steps performed include:\n", + "\n", + "- Create a Vertex custom job for training a model.\n", + "- Train a TensorFlow model.\n", + "- Retrieve and load the model artifacts.\n", + "- View the model evaluation.\n", + "- Set explanation parameters.\n", + "- Upload the model as a Vertex `Model` resource.\n", + "- Deploy the `Model` resource to a serving `Endpoint` resource.\n", + "- Make a prediction with explanation.\n", + "- Undeploy the `Model` resource." + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "costs" + }, + "source": [ + "### Costs\n", + "\n", + "This tutorial uses billable components of Google Cloud (GCP):\n", + "\n", + "* Vertex AI\n", + "* Cloud Storage\n", + "\n", + "Learn about [Vertex AI\n", + "pricing](https://cloud.google.com/vertex-ai/pricing) and [Cloud Storage\n", + "pricing](https://cloud.google.com/storage/pricing), and use the [Pricing\n", + "Calculator](https://cloud.google.com/products/calculator/)\n", + "to generate a cost estimate based on your projected usage." + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "install_aip" + }, + "source": [ + "## Installation\n", + "\n", + "Install the latest version of Vertex client library for Python." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "install_aip" + }, + "outputs": [], + "source": [ + "import os\n", + "import sys\n", + "\n", + "# Google Cloud Notebook\n", + "if os.path.exists(\"/opt/deeplearning/metadata/env_version\"):\n", + " USER_FLAG = \"--user\"\n", + "else:\n", + " USER_FLAG = \"\"\n", + "\n", + "! pip3 install --upgrade google-cloud-aiplatform $USER_FLAG" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "install_storage" + }, + "source": [ + "Install the latest GA version of *google-cloud-storage* library as well." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "install_storage" + }, + "outputs": [], + "source": [ + "! pip3 install -U google-cloud-storage $USER_FLAG" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "install_other_packages" + }, + "source": [ + "### Install other packages\n", + "\n", + "Install other packages required for this tutorial." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "install_tabulate" + }, + "outputs": [], + "source": [ + "! pip3 install -U tabulate $USER_FLAG" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "restart" + }, + "source": [ + "### Restart the kernel\n", + "\n", + "Once you've installed the Vertex client library for Python and *google-cloud-storage* library, you need to restart the notebook kernel so it can find the packages." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "restart" + }, + "outputs": [], + "source": [ + "if not os.getenv(\"IS_TESTING\"):\n", + " # Automatically restart kernel after installs\n", + " import IPython\n", + "\n", + " app = IPython.Application.instance()\n", + " app.kernel.do_shutdown(True)" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "before_you_begin" + }, + "source": [ + "## Before you begin\n", + "\n", + "### GPU runtime\n", + "\n", + "*Make sure you're running this notebook in a GPU runtime if you have that option. In Colab, select* **Runtime > Change Runtime Type > GPU**\n", + "\n", + "### Set up your Google Cloud project\n", + "\n", + "**The following steps are required, regardless of your notebook environment.**\n", + "\n", + "1. [Select or create a Google Cloud project](https://console.cloud.google.com/cloud-resource-manager). When you first create an account, you get a $300 free credit towards your compute/storage costs.\n", + "\n", + "2. [Make sure that billing is enabled for your project.](https://cloud.google.com/billing/docs/how-to/modify-project)\n", + "\n", + "3. [Enable the Vertex AI APIs and Compute Engine APIs.](https://console.cloud.google.com/flows/enableapi?apiid=ml.googleapis.com,compute_component)\n", + "\n", + "4. [The Google Cloud SDK](https://cloud.google.com/sdk) is already installed in Google Cloud Notebook.\n", + "\n", + "5. Enter your project ID in the cell below. Then run the cell to make sure the\n", + "Cloud SDK uses the right project for all the commands in this notebook.\n", + "\n", + "**Note**: Jupyter runs lines prefixed with `!` as shell commands, and it interpolates Python variables prefixed with `$`." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "set_project_id" + }, + "outputs": [], + "source": [ + "PROJECT_ID = \"[your-project-id]\" # @param {type:\"string\"}" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "autoset_project_id" + }, + "outputs": [], + "source": [ + "if PROJECT_ID == \"\" or PROJECT_ID is None or PROJECT_ID == \"[your-project-id]\":\n", + " # Get your GCP project id from gcloud\n", + " shell_output = !gcloud config list --format 'value(core.project)' 2>/dev/null\n", + " PROJECT_ID = shell_output[0]\n", + " print(\"Project ID:\", PROJECT_ID)" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "set_gcloud_project_id" + }, + "outputs": [], + "source": [ + "! gcloud config set project $PROJECT_ID" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "region" + }, + "source": [ + "#### Region\n", + "\n", + "You can also change the `REGION` variable, which is used for operations\n", + "throughout the rest of this notebook. Below are regions supported for Vertex. We recommend that you choose the region closest to you.\n", + "\n", + "- Americas: `us-central1`\n", + "- Europe: `europe-west4`\n", + "- Asia Pacific: `asia-east1`\n", + "\n", + "You may not use a multi-regional bucket for training with Vertex. Not all regions provide support for all Vertex services.\n", + "\n", + "Learn more about [Vertex regions](https://cloud.google.com/vertex-ai/docs/general/locations)" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "region" + }, + "outputs": [], + "source": [ + "REGION = \"us-central1\" # @param {type: \"string\"}" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "timestamp" + }, + "source": [ + "#### Timestamp\n", + "\n", + "If you are in a live tutorial session, you might be using a shared test account or project. To avoid name collisions between users on resources created, you create a timestamp for each instance session, and append the timestamp onto the name of resources you create in this tutorial." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "timestamp" + }, + "outputs": [], + "source": [ + "from datetime import datetime\n", + "\n", + "TIMESTAMP = datetime.now().strftime(\"%Y%m%d%H%M%S\")" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "gcp_authenticate" + }, + "source": [ + "### Authenticate your Google Cloud account\n", + "\n", + "**If you are using Google Cloud Notebook**, your environment is already authenticated. Skip this step.\n", + "\n", + "**If you are using Colab**, run the cell below and follow the instructions when prompted to authenticate your account via oAuth.\n", + "\n", + "**Otherwise**, follow these steps:\n", + "\n", + "In the Cloud Console, go to the [Create service account key](https://console.cloud.google.com/apis/credentials/serviceaccountkey) page.\n", + "\n", + "**Click Create service account**.\n", + "\n", + "In the **Service account name** field, enter a name, and click **Create**.\n", + "\n", + "In the **Grant this service account access to project** section, click the Role drop-down list. Type \"Vertex\" into the filter box, and select **Vertex AI Administrator**. Type \"Storage Object Admin\" into the filter box, and select **Storage Object Admin**.\n", + "\n", + "Click Create. A JSON file that contains your key downloads to your local environment.\n", + "\n", + "Enter the path to your service account key as the `GOOGLE_APPLICATION_CREDENTIALS` variable in the cell below and run the cell." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "gcp_authenticate" + }, + "outputs": [], + "source": [ + "# If you are running this notebook in Colab, run this cell and follow the\n", + "# instructions to authenticate your GCP account. This provides access to your\n", + "# Cloud Storage bucket and lets you submit training jobs and prediction\n", + "# requests.\n", + "\n", + "# If on Google Cloud Notebook, then don't execute this code\n", + "if not os.path.exists(\"/opt/deeplearning/metadata/env_version\"):\n", + " if \"google.colab\" in sys.modules:\n", + " from google.colab import auth as google_auth\n", + "\n", + " google_auth.authenticate_user()\n", + "\n", + " # If you are running this notebook locally, replace the string below with the\n", + " # path to your service account key and run this cell to authenticate your GCP\n", + " # account.\n", + " elif not os.getenv(\"IS_TESTING\"):\n", + " %env GOOGLE_APPLICATION_CREDENTIALS ''" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "bucket:custom" + }, + "source": [ + "### Create a Cloud Storage bucket\n", + "\n", + "**The following steps are required, regardless of your notebook environment.**\n", + "\n", + "When you submit a custom training job using the Vertex client library for Python, you upload a Python package\n", + "containing your training code to a Cloud Storage bucket. Vertex runs\n", + "the code from this package. In this tutorial, Vertex also saves the\n", + "trained model that results from your job in the same bucket. You can then\n", + "create an `Endpoint` resource based on this output in order to serve\n", + "online predictions.\n", + "\n", + "Set the name of your Cloud Storage bucket below. Bucket names must be globally unique across all Google Cloud projects, including those outside of your organization." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "bucket" + }, + "outputs": [], + "source": [ + "BUCKET_NAME = \"gs://[your-bucket-name]\" # @param {type:\"string\"}" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "autoset_bucket" + }, + "outputs": [], + "source": [ + "if BUCKET_NAME == \"\" or BUCKET_NAME is None or BUCKET_NAME == \"gs://[your-bucket-name]\":\n", + " BUCKET_NAME = \"gs://\" + PROJECT_ID + \"aip-\" + TIMESTAMP" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "create_bucket" + }, + "source": [ + "**Only if your bucket doesn't already exist**: Run the following cell to create your Cloud Storage bucket." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "create_bucket" + }, + "outputs": [], + "source": [ + "! gsutil mb -l $REGION $BUCKET_NAME" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "validate_bucket" + }, + "source": [ + "Finally, validate access to your Cloud Storage bucket by examining its contents:" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "validate_bucket" + }, + "outputs": [], + "source": [ + "! gsutil ls -al $BUCKET_NAME" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "setup_vars" + }, + "source": [ + "### Set up variables\n", + "\n", + "Next, set up some variables used throughout the tutorial.\n", + "### Import libraries and define constants" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "import_aip:v1beta1,protobuf" + }, + "source": [ + "#### Import Vertex client library for Python\n", + "\n", + "Import the Vertex client library for Python into your Python environment." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "import_aip:v1beta1,protobuf" + }, + "outputs": [], + "source": [ + "import time\n", + "\n", + "import google.cloud.aiplatform_v1beta1 as aip\n", + "from google.protobuf import json_format\n", + "from google.protobuf.json_format import MessageToJson, ParseDict\n", + "from google.protobuf.struct_pb2 import Struct, Value" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "aip_constants" + }, + "source": [ + "#### Vertex AI constants\n", + "\n", + "Setup up the following constants for Vertex AI:\n", + "\n", + "- `API_ENDPOINT`: The Vertex AI API service endpoint for dataset, model, job, pipeline and endpoint services.\n", + "- `PARENT`: The Vertex AI location root path for dataset, model, job, pipeline and endpoint resources." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "aip_constants" + }, + "outputs": [], + "source": [ + "# API service endpoint\n", + "API_ENDPOINT = \"{}-aiplatform.googleapis.com\".format(REGION)\n", + "\n", + "# Vertex location root path for your dataset, model and endpoint resources\n", + "PARENT = \"projects/\" + PROJECT_ID + \"/locations/\" + REGION" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "accelerators:training,prediction,cpu" + }, + "source": [ + "#### Set hardware accelerators\n", + "\n", + "You can set hardware accelerators for training and prediction.\n", + "\n", + "Set the variables `TRAIN_GPU/TRAIN_NGPU` and `DEPLOY_GPU/DEPLOY_NGPU` to use a container image supporting a GPU and the number of GPUs allocated to the virtual machine (VM) instance. For example, to use a GPU container image with 4 Nvidia Telsa K80 GPUs allocated to each VM, you would specify:\n", + "\n", + " (aip.AcceleratorType.NVIDIA_TESLA_K80, 4)\n", + "\n", + "\n", + "Otherwise specify `(None, None)` to use a container image to run on a CPU.\n", + "\n", + "Learn [which accelerators are available in your region](https://cloud.google.com/vertex-ai/docs/general/locations#accelerators).\n", + "\n", + "*Note*: TF releases before 2.3 for GPU support will fail to load the custom trained model in this tutorial. It is a known issue and fixed in TF 2.3 -- which is caused by static graph ops that are generated in the serving function. If you encounter this issue on your own custom trained models, use a container image for TF 2.3 with GPU support." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "accelerators:training,prediction,cpu" + }, + "outputs": [], + "source": [ + "if os.getenv(\"IS_TESTING_TRAIN_GPU\"):\n", + " TRAIN_GPU, TRAIN_NGPU = (\n", + " aip.AcceleratorType.NVIDIA_TESLA_K80,\n", + " int(os.getenv(\"IS_TESTING_TRAIN_GPU\")),\n", + " )\n", + "else:\n", + " TRAIN_GPU, TRAIN_NGPU = (aip.AcceleratorType.NVIDIA_TESLA_K80, 1)\n", + "\n", + "if os.getenv(\"IS_TESTING_DEPLOY_GPU\"):\n", + " DEPLOY_GPU, DEPLOY_NGPU = (\n", + " aip.AcceleratorType.NVIDIA_TESLA_K80,\n", + " int(os.getenv(\"IS_TESTING_DEPLOY_GPU\")),\n", + " )\n", + "else:\n", + " DEPLOY_GPU, DEPLOY_NGPU = (None, None)" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "container:training,prediction" + }, + "source": [ + "#### Set pre-built containers\n", + "\n", + "Set the pre-built Docker container image for training and prediction.\n", + "\n", + "\n", + "For the latest list, see [Pre-built containers for training](https://cloud.google.com/ai-platform-unified/docs/training/pre-built-containers).\n", + "\n", + "\n", + "For the latest list, see [Pre-built containers for prediction](https://cloud.google.com/ai-platform-unified/docs/predictions/pre-built-containers)" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "container:training,prediction" + }, + "outputs": [], + "source": [ + "if os.getenv(\"IS_TESTING_TF\"):\n", + " TF = os.getenv(\"IS_TESTING_TF\")\n", + "else:\n", + " TF = \"2-1\"\n", + "\n", + "if TF[0] == \"2\":\n", + " if TRAIN_GPU:\n", + " TRAIN_VERSION = \"tf-gpu.{}\".format(TF)\n", + " else:\n", + " TRAIN_VERSION = \"tf-cpu.{}\".format(TF)\n", + " if DEPLOY_GPU:\n", + " DEPLOY_VERSION = \"tf2-gpu.{}\".format(TF)\n", + " else:\n", + " DEPLOY_VERSION = \"tf2-cpu.{}\".format(TF)\n", + "else:\n", + " if TRAIN_GPU:\n", + " TRAIN_VERSION = \"tf-gpu.{}\".format(TF)\n", + " else:\n", + " TRAIN_VERSION = \"tf-cpu.{}\".format(TF)\n", + " if DEPLOY_GPU:\n", + " DEPLOY_VERSION = \"tf-gpu.{}\".format(TF)\n", + " else:\n", + " DEPLOY_VERSION = \"tf-cpu.{}\".format(TF)\n", + "\n", + "TRAIN_IMAGE = \"gcr.io/cloud-aiplatform/training/{}:latest\".format(TRAIN_VERSION)\n", + "DEPLOY_IMAGE = \"gcr.io/cloud-aiplatform/prediction/{}:latest\".format(DEPLOY_VERSION)\n", + "\n", + "print(\"Training:\", TRAIN_IMAGE, TRAIN_GPU, TRAIN_NGPU)\n", + "print(\"Deployment:\", DEPLOY_IMAGE, DEPLOY_GPU, DEPLOY_NGPU)" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "machine:training,prediction" + }, + "source": [ + "#### Set machine type\n", + "\n", + "Next, set the machine type to use for training and prediction.\n", + "\n", + "- Set the variables `TRAIN_COMPUTE` and `DEPLOY_COMPUTE` to configure the compute resources for the VMs you will use for for training and prediction.\n", + " - `machine type`\n", + " - `n1-standard`: 3.75GB of memory per vCPU.\n", + " - `n1-highmem`: 6.5GB of memory per vCPU\n", + " - `n1-highcpu`: 0.9 GB of memory per vCPU\n", + " - `vCPUs`: number of \\[2, 4, 8, 16, 32, 64, 96 \\]\n", + "\n", + "*Note: The following is not supported for training:*\n", + "\n", + " - `standard`: 2 vCPUs\n", + " - `highcpu`: 2, 4 and 8 vCPUs\n", + "\n", + "*Note: You may also use n2 and e2 machine types for training and deployment, but they do not support GPUs*.\n", + "\n", + "Learn [which machine types are available for training](https://cloud.google.com/vertex-ai/docs/training/configure-compute) and [for prediction](https://cloud.google.com/vertex-ai/docs/predictions/configure-compute)" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "machine:training,prediction" + }, + "outputs": [], + "source": [ + "if os.getenv(\"IS_TESTING_TRAIN_MACHINE\"):\n", + " MACHINE_TYPE = os.getenv(\"IS_TESTING_TRAIN_MACHINE\")\n", + "else:\n", + " MACHINE_TYPE = \"n1-standard\"\n", + "\n", + "VCPU = \"4\"\n", + "TRAIN_COMPUTE = MACHINE_TYPE + \"-\" + VCPU\n", + "print(\"Train machine type\", TRAIN_COMPUTE)\n", + "\n", + "if os.getenv(\"IS_TESTING_DEPLOY_MACHINE\"):\n", + " MACHINE_TYPE = os.getenv(\"IS_TESTING_DEPLOY_MACHINE\")\n", + "else:\n", + " MACHINE_TYPE = \"n1-standard\"\n", + "\n", + "VCPU = \"4\"\n", + "DEPLOY_COMPUTE = MACHINE_TYPE + \"-\" + VCPU\n", + "print(\"Deploy machine type\", DEPLOY_COMPUTE)" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "tutorial_start:custom" + }, + "source": [ + "# Tutorial\n", + "\n", + "Now you are ready to start creating your own custom model and training for Boston Housing." + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "clients:custom" + }, + "source": [ + "## Set up clients\n", + "\n", + "The Vertex client library for Python works as a client/server model. On your side (the Python script) you will create a client that sends requests and receives responses from the Vertex server.\n", + "\n", + "You will use different clients in this tutorial for different steps in the workflow. So set them all up upfront.\n", + "\n", + "- Model Service for `Model` resources.\n", + "- Endpoint Service for deployment.\n", + "- Job Service for batch jobs and custom training.\n", + "- Prediction Service for serving." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "clients:custom" + }, + "outputs": [], + "source": [ + "# client options same for all services\n", + "client_options = {\"api_endpoint\": API_ENDPOINT}\n", + "\n", + "\n", + "def create_job_client():\n", + " client = aip.JobServiceClient(client_options=client_options)\n", + " return client\n", + "\n", + "\n", + "def create_model_client():\n", + " client = aip.ModelServiceClient(client_options=client_options)\n", + " return client\n", + "\n", + "\n", + "def create_endpoint_client():\n", + " client = aip.EndpointServiceClient(client_options=client_options)\n", + " return client\n", + "\n", + "\n", + "def create_prediction_client():\n", + " client = aip.PredictionServiceClient(client_options=client_options)\n", + " return client\n", + "\n", + "\n", + "clients = {}\n", + "clients[\"job\"] = create_job_client()\n", + "clients[\"model\"] = create_model_client()\n", + "clients[\"endpoint\"] = create_endpoint_client()\n", + "clients[\"prediction\"] = create_prediction_client()\n", + "\n", + "for client in clients.items():\n", + " print(client)" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "train_custom_model" + }, + "source": [ + "## Train a model\n", + "\n", + "There are two ways you can train a custom model using a container image:\n", + "\n", + "- **Use a Google Cloud prebuilt container**. If you use a prebuilt container, you will additionally specify a Python package to install into the container image. This Python package contains your code for training a custom model.\n", + "\n", + "- **Use your own custom container image**. If you use your own container, the container needs to contain your code for training a custom model." + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "train_custom_job_specification:prebuilt_container" + }, + "source": [ + "## Prepare your custom job specification\n", + "\n", + "Now that your clients are ready, your first step is to create a Job Specification for your custom training job. The job specification will consist of the following:\n", + "\n", + "- `worker_pool_spec` : The specification of the type of machine(s) you will use for training and how many (single or distributed)\n", + "- `python_package_spec` : The specification of the Python package to be installed with the pre-built container." + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "train_custom_job_machine_specification" + }, + "source": [ + "### Prepare your machine specification\n", + "\n", + "Now define the machine specification for your custom training job. This tells Vertex what type of machine instance to provision for the training.\n", + " - `machine_type`: The type of GCP instance to provision -- e.g., n1-standard-8.\n", + " - `accelerator_type`: The type, if any, of hardware accelerator. In this tutorial if you previously set the variable `TRAIN_GPU != None`, you are using a GPU; otherwise you will use a CPU.\n", + " - `accelerator_count`: The number of accelerators." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "train_custom_job_machine_specification" + }, + "outputs": [], + "source": [ + "if TRAIN_GPU:\n", + " machine_spec = {\n", + " \"machine_type\": TRAIN_COMPUTE,\n", + " \"accelerator_type\": TRAIN_GPU,\n", + " \"accelerator_count\": TRAIN_NGPU,\n", + " }\n", + "else:\n", + " machine_spec = {\"machine_type\": TRAIN_COMPUTE, \"accelerator_count\": 0}" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "train_custom_job_disk_specification" + }, + "outputs": [], + "source": [ + "DISK_TYPE = \"pd-ssd\" # [ pd-ssd, pd-standard]\n", + "DISK_SIZE = 200 # GB\n", + "\n", + "disk_spec = {\"boot_disk_type\": DISK_TYPE, \"boot_disk_size_gb\": DISK_SIZE}" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "train_custom_job_worker_pool_specification:prebuilt_container" + }, + "source": [ + "### Define the worker pool specification\n", + "\n", + "Next, you define the worker pool specification for your custom training job. The worker pool specification will consist of the following:\n", + "\n", + "- `replica_count`: The number of instances to provision of this machine type.\n", + "- `machine_spec`: The hardware specification.\n", + "- `disk_spec` : (optional) The disk storage specification.\n", + "\n", + "- `python_package`: The Python training package to install on the VM instance(s) and which Python module to invoke, along with command line arguments for the Python module.\n", + "\n", + "Let's dive deeper now into the python package specification:\n", + "\n", + "-`executor_image_spec`: This is the docker image which is configured for your custom training job.\n", + "\n", + "-`package_uris`: This is a list of the locations (URIs) of your python training packages to install on the provisioned instance. The locations need to be in a Cloud Storage bucket. These can be either individual python files or a zip (archive) of an entire package. In the later case, the job service will unzip (unarchive) the contents into the docker image.\n", + "\n", + "-`python_module`: The Python module (script) to invoke for running the custom training job. In this example, you will be invoking `trainer.task.py` -- note that it was not neccessary to append the `.py` suffix.\n", + "\n", + "-`args`: The command line arguments to pass to the corresponding Pythom module. In this example, you will be setting:\n", + " - `\"--model-dir=\" + MODEL_DIR` : The Cloud Storage location where to store the model artifacts. There are two ways to tell the training script where to save the model artifacts:\n", + " - direct: You pass the Cloud Storage location as a command line argument to your training script (set variable `DIRECT = True`), or\n", + " - indirect: The service passes the Cloud Storage location as the environment variable `AIP_MODEL_DIR` to your training script (set variable `DIRECT = False`). In this case, you tell the service the model artifact location in the job specification.\n", + " - `\"--epochs=\" + EPOCHS`: The number of epochs for training.\n", + " - `\"--steps=\" + STEPS`: The number of steps (batches) per epoch.\n", + " - `\"--distribute=\" + TRAIN_STRATEGY\"` : The training distribution strategy to use for single or distributed training.\n", + " - `\"single\"`: single device.\n", + " - `\"mirror\"`: all GPU devices on a single compute instance.\n", + " - `\"multi\"`: all GPU devices on all compute instances." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "train_custom_job_worker_pool_specification:prebuilt_container" + }, + "outputs": [], + "source": [ + "JOB_NAME = \"custom_job_\" + TIMESTAMP\n", + "MODEL_DIR = \"{}/{}\".format(BUCKET_NAME, JOB_NAME)\n", + "\n", + "if not TRAIN_NGPU or TRAIN_NGPU < 2:\n", + " TRAIN_STRATEGY = \"single\"\n", + "else:\n", + " TRAIN_STRATEGY = \"mirror\"\n", + "\n", + "EPOCHS = 20\n", + "STEPS = 100\n", + "\n", + "DIRECT = True\n", + "if DIRECT:\n", + " CMDARGS = [\n", + " \"--model-dir=\" + MODEL_DIR,\n", + " \"--epochs=\" + str(EPOCHS),\n", + " \"--steps=\" + str(STEPS),\n", + " \"--distribute=\" + TRAIN_STRATEGY,\n", + " ]\n", + "else:\n", + " CMDARGS = [\n", + " \"--epochs=\" + str(EPOCHS),\n", + " \"--steps=\" + str(STEPS),\n", + " \"--distribute=\" + TRAIN_STRATEGY,\n", + " ]\n", + "\n", + "worker_pool_spec = [\n", + " {\n", + " \"replica_count\": 1,\n", + " \"machine_spec\": machine_spec,\n", + " \"disk_spec\": disk_spec,\n", + " \"python_package_spec\": {\n", + " \"executor_image_uri\": TRAIN_IMAGE,\n", + " \"package_uris\": [BUCKET_NAME + \"/trainer_boston.tar.gz\"],\n", + " \"python_module\": \"trainer.task\",\n", + " \"args\": CMDARGS,\n", + " },\n", + " }\n", + "]" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "assemble_custom_job_specification" + }, + "source": [ + "### Assemble a job specification\n", + "\n", + "Now assemble the complete description for the custom job specification:\n", + "\n", + "- `display_name`: The human readable name you assign to this custom job.\n", + "- `job_spec`: The specification for the custom job.\n", + " - `worker_pool_specs`: The specification for the machine VM instances.\n", + " - `base_output_directory`: This tells the service the Cloud Storage location where to save the model artifacts (when variable `DIRECT = False`). The service will then pass the location to the training script as the environment variable `AIP_MODEL_DIR`, and the path will be of the form:\n", + "\n", + " /model" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "assemble_custom_job_specification" + }, + "outputs": [], + "source": [ + "if DIRECT:\n", + " job_spec = {\"worker_pool_specs\": worker_pool_spec}\n", + "else:\n", + " job_spec = {\n", + " \"worker_pool_specs\": worker_pool_spec,\n", + " \"base_output_directory\": {\"output_uri_prefix\": MODEL_DIR},\n", + " }\n", + "\n", + "custom_job = {\"display_name\": JOB_NAME, \"job_spec\": job_spec}" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "examine_training_package" + }, + "source": [ + "### Examine the training package\n", + "\n", + "#### Package layout\n", + "\n", + "Before you start the training, you will look at how a Python package is assembled for a custom training job. When unarchived, the package contains the following directory/file layout.\n", + "\n", + "- PKG-INFO\n", + "- README.md\n", + "- setup.cfg\n", + "- setup.py\n", + "- trainer\n", + " - \\_\\_init\\_\\_.py\n", + " - task.py\n", + "\n", + "The files `setup.cfg` and `setup.py` are the instructions for installing the package into the operating environment of the Docker image.\n", + "\n", + "The file `trainer/task.py` is the Python script for executing the custom training job. *Note*, when we referred to it in the worker pool specification, we replace the directory slash with a dot (`trainer.task`) and dropped the file suffix (`.py`).\n", + "\n", + "#### Package Assembly\n", + "\n", + "In the following cells, you will assemble the training package." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "examine_training_package" + }, + "outputs": [], + "source": [ + "# Make folder for Python training script\n", + "! rm -rf custom\n", + "! mkdir custom\n", + "\n", + "# Add package information\n", + "! touch custom/README.md\n", + "\n", + "setup_cfg = \"[egg_info]\\n\\ntag_build =\\n\\ntag_date = 0\"\n", + "! echo \"$setup_cfg\" > custom/setup.cfg\n", + "\n", + "setup_py = \"import setuptools\\n\\nsetuptools.setup(\\n\\n install_requires=[\\n\\n 'tensorflow_datasets==1.3.0',\\n\\n ],\\n\\n packages=setuptools.find_packages())\"\n", + "! echo \"$setup_py\" > custom/setup.py\n", + "\n", + "pkg_info = \"Metadata-Version: 1.0\\n\\nName: Boston Housing tabular regression\\n\\nVersion: 0.0.0\\n\\nSummary: Demostration training script\\n\\nHome-page: www.google.com\\n\\nAuthor: Google\\n\\nAuthor-email: aferlitsch@google.com\\n\\nLicense: Public\\n\\nDescription: Demo\\n\\nPlatform: Vertex\"\n", + "! echo \"$pkg_info\" > custom/PKG-INFO\n", + "\n", + "# Make the training subfolder\n", + "! mkdir custom/trainer\n", + "! touch custom/trainer/__init__.py" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "taskpy_contents:boston" + }, + "source": [ + "#### Task.py contents\n", + "\n", + "In the next cell, you write the contents of the training script task.py. I won't go into detail, it's just there for you to browse. In summary:\n", + "\n", + "- Get the directory where to save the model artifacts from the command line (`--model_dir`), and if not specified, then from the environment variable `AIP_MODEL_DIR`.\n", + "- Loads Boston Housing dataset from TF.Keras builtin datasets\n", + "- Builds a simple deep neural network model using TF.Keras model API.\n", + "- Compiles the model (`compile()`).\n", + "- Sets a training distribution strategy according to the argument `args.distribute`.\n", + "- Trains the model (`fit()`) with epochs specified by `args.epochs`.\n", + "- Saves the trained model (`save(args.model_dir)`) to the specified model directory.\n", + "- Saves the maximum value for each feature `f.write(str(params))` to the specified parameters file." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "taskpy_contents:boston" + }, + "outputs": [], + "source": [ + "%%writefile custom/trainer/task.py\n", + "# Single, Mirror and Multi-Machine Distributed Training for Boston Housing\n", + "\n", + "import tensorflow_datasets as tfds\n", + "import tensorflow as tf\n", + "from tensorflow.python.client import device_lib\n", + "import numpy as np\n", + "import argparse\n", + "import os\n", + "import sys\n", + "tfds.disable_progress_bar()\n", + "\n", + "parser = argparse.ArgumentParser()\n", + "parser.add_argument('--model-dir', dest='model_dir',\n", + " default=os.getenv('AIP_MODEL_DIR'), type=str, help='Model dir.')\n", + "parser.add_argument('--lr', dest='lr',\n", + " default=0.001, type=float,\n", + " help='Learning rate.')\n", + "parser.add_argument('--epochs', dest='epochs',\n", + " default=20, type=int,\n", + " help='Number of epochs.')\n", + "parser.add_argument('--steps', dest='steps',\n", + " default=100, type=int,\n", + " help='Number of steps per epoch.')\n", + "parser.add_argument('--distribute', dest='distribute', type=str, default='single',\n", + " help='distributed training strategy')\n", + "parser.add_argument('--param-file', dest='param_file',\n", + " default='/tmp/param.txt', type=str,\n", + " help='Output file for parameters')\n", + "args = parser.parse_args()\n", + "\n", + "print('Python Version = {}'.format(sys.version))\n", + "print('TensorFlow Version = {}'.format(tf.__version__))\n", + "print('TF_CONFIG = {}'.format(os.environ.get('TF_CONFIG', 'Not found')))\n", + "\n", + "# Single Machine, single compute device\n", + "if args.distribute == 'single':\n", + " if tf.test.is_gpu_available():\n", + " strategy = tf.distribute.OneDeviceStrategy(device=\"/gpu:0\")\n", + " else:\n", + " strategy = tf.distribute.OneDeviceStrategy(device=\"/cpu:0\")\n", + "# Single Machine, multiple compute device\n", + "elif args.distribute == 'mirror':\n", + " strategy = tf.distribute.MirroredStrategy()\n", + "# Multiple Machine, multiple compute device\n", + "elif args.distribute == 'multi':\n", + " strategy = tf.distribute.experimental.MultiWorkerMirroredStrategy()\n", + "\n", + "# Multi-worker configuration\n", + "print('num_replicas_in_sync = {}'.format(strategy.num_replicas_in_sync))\n", + "\n", + "\n", + "def make_dataset():\n", + "\n", + " # Scaling Boston Housing data features\n", + " def scale(feature):\n", + " max = np.max(feature)\n", + " feature = (feature / max).astype(np.float)\n", + " return feature, max\n", + "\n", + " (x_train, y_train), (x_test, y_test) = tf.keras.datasets.boston_housing.load_data(\n", + " path=\"boston_housing.npz\", test_split=0.2, seed=113\n", + " )\n", + " params = []\n", + " for _ in range(13):\n", + " x_train[_], max = scale(x_train[_])\n", + " x_test[_], _ = scale(x_test[_])\n", + " params.append(max)\n", + "\n", + " # store the normalization (max) value for each feature\n", + " with tf.io.gfile.GFile(args.param_file, 'w') as f:\n", + " f.write(str(params))\n", + " return (x_train, y_train), (x_test, y_test)\n", + "\n", + "\n", + "# Build the Keras model\n", + "def build_and_compile_dnn_model():\n", + " model = tf.keras.Sequential([\n", + " tf.keras.layers.Dense(128, activation='relu', input_shape=(13,)),\n", + " tf.keras.layers.Dense(128, activation='relu'),\n", + " tf.keras.layers.Dense(1, activation='linear')\n", + " ])\n", + " model.compile(\n", + " loss='mse',\n", + " optimizer=tf.keras.optimizers.RMSprop(learning_rate=args.lr))\n", + " return model\n", + "\n", + "NUM_WORKERS = strategy.num_replicas_in_sync\n", + "# Here the batch size scales up by number of workers since\n", + "# `tf.data.Dataset.batch` expects the global batch size.\n", + "BATCH_SIZE = 16\n", + "GLOBAL_BATCH_SIZE = BATCH_SIZE * NUM_WORKERS\n", + "\n", + "with strategy.scope():\n", + " # Creation of dataset, and model building/compiling need to be within\n", + " # `strategy.scope()`.\n", + " model = build_and_compile_dnn_model()\n", + "\n", + "# Train the model\n", + "(x_train, y_train), (x_test, y_test) = make_dataset()\n", + "model.fit(x_train, y_train, epochs=args.epochs, batch_size=GLOBAL_BATCH_SIZE)\n", + "model.save(args.model_dir)" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "tarball_training_script" + }, + "source": [ + "#### Store training script on your Cloud Storage bucket\n", + "\n", + "Next, you package the training folder into a compressed tar ball, and then store it in your Cloud Storage bucket." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "tarball_training_script" + }, + "outputs": [], + "source": [ + "! rm -f custom.tar custom.tar.gz\n", + "! tar cvf custom.tar custom\n", + "! gzip custom.tar\n", + "! gsutil cp custom.tar.gz $BUCKET_NAME/trainer_boston.tar.gz" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "train_custom_job" + }, + "source": [ + "### Train the model\n", + "\n", + "\n", + "Now start the training of your custom training job on Vertex. Use this helper function `create_custom_job`, which takes the following parameter:\n", + "\n", + "-`custom_job`: The specification for the custom job.\n", + "\n", + "The helper function calls job client service's `create_custom_job` method, with the following parameters:\n", + "\n", + "-`parent`: The Vertex location path to `Dataset`, `Model` and `Endpoint` resources.\n", + "-`custom_job`: The specification for the custom job.\n", + "\n", + "You will display a handful of the fields returned in `response` object, with the two that are of most interest are:\n", + "\n", + "-`response.name`: The Vertex fully qualified identifier assigned to this custom training job. You save this identifier for using in subsequent steps.\n", + "\n", + "-`response.state`: The current state of the custom training job." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "train_custom_job" + }, + "outputs": [], + "source": [ + "def create_custom_job(custom_job):\n", + " response = clients[\"job\"].create_custom_job(parent=PARENT, custom_job=custom_job)\n", + " print(\"name:\", response.name)\n", + " print(\"display_name:\", response.display_name)\n", + " print(\"state:\", response.state)\n", + " print(\"create_time:\", response.create_time)\n", + " print(\"update_time:\", response.update_time)\n", + " return response\n", + "\n", + "\n", + "response = create_custom_job(custom_job)" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "job_id:response" + }, + "source": [ + "Now get the unique identifier for the custom job you created." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "job_id:response" + }, + "outputs": [], + "source": [ + "# The full unique ID for the custom job\n", + "job_id = response.name\n", + "# The short numeric ID for the custom job\n", + "job_short_id = job_id.split(\"/\")[-1]\n", + "\n", + "print(job_id)" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "get_custom_job" + }, + "source": [ + "### Get information on a custom job\n", + "\n", + "Next, use this helper function `get_custom_job`, which takes the following parameter:\n", + "\n", + "- `name`: The Vertex fully qualified identifier for the custom job.\n", + "\n", + "The helper function calls the job client service's `get_custom_job` method, with the following parameter:\n", + "\n", + "- `name`: The Vertex fully qualified identifier for the custom job.\n", + "\n", + "If you recall, you got the Vertex fully qualified identifier for the custom job in the `response.name` field when you called the `create_custom_job` method, and saved the identifier in the variable `job_id`." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "get_custom_job" + }, + "outputs": [], + "source": [ + "def get_custom_job(name, silent=False):\n", + " response = clients[\"job\"].get_custom_job(name=name)\n", + " if silent:\n", + " return response\n", + "\n", + " print(\"name:\", response.name)\n", + " print(\"display_name:\", response.display_name)\n", + " print(\"state:\", response.state)\n", + " print(\"create_time:\", response.create_time)\n", + " print(\"update_time:\", response.update_time)\n", + " return response\n", + "\n", + "\n", + "response = get_custom_job(job_id)" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "wait_training_complete:custom" + }, + "source": [ + "# Deploy the model\n", + "\n", + "Training the above model may take upwards of 20 minutes time.\n", + "\n", + "Once your model is done training, you can calculate the actual time it took to train the model by subtracting `end_time` from `start_time`. For your model, we will need to know the location of the saved model, which the Python script saved in your local Cloud Storage bucket at `MODEL_DIR + '/saved_model.pb'`." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "wait_training_complete:custom" + }, + "outputs": [], + "source": [ + "while True:\n", + " response = get_custom_job(job_id, True)\n", + " if response.state != aip.JobState.JOB_STATE_SUCCEEDED:\n", + " print(\"Training job has not completed:\", response.state)\n", + " model_path_to_deploy = None\n", + " if response.state == aip.JobState.JOB_STATE_FAILED:\n", + " break\n", + " else:\n", + " if not DIRECT:\n", + " MODEL_DIR = MODEL_DIR + \"/model\"\n", + " model_path_to_deploy = MODEL_DIR\n", + " print(\"Training Job Time:\", response.end_time - response.start_time)\n", + " print(\"Training Elapsed Time:\", response.update_time - response.create_time)\n", + " break\n", + " time.sleep(60)\n", + "\n", + "print(\"model_to_deploy:\", model_path_to_deploy)" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "load_saved_model" + }, + "source": [ + "## Load the saved model\n", + "\n", + "Your model is stored in a TensorFlow SavedModel format in a Cloud Storage bucket. Now load it from the Cloud Storage bucket, and then you can do some things, like evaluate the model, and do a prediction.\n", + "\n", + "To load, you use the TF.Keras `model.load_model()` method passing it the Cloud Storage path where the model is saved -- specified by `MODEL_DIR`." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "load_saved_model" + }, + "outputs": [], + "source": [ + "import tensorflow as tf\n", + "\n", + "model = tf.keras.models.load_model(MODEL_DIR)" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "evaluate_custom_model:tabular" + }, + "source": [ + "## Evaluate the model\n", + "\n", + "Now let's find out how good the model is.\n", + "\n", + "### Load evaluation data\n", + "\n", + "You will load the Boston Housing test (holdout) data from `tf.keras.datasets`, using the method `load_data()`. This returns the dataset as a tuple of two elements. The first element is the training data and the second is the test data. Each element is also a tuple of two elements: the feature data, and the corresponding labels (median value of owner-occupied home).\n", + "\n", + "You don't need the training data, and hence why we loaded it as `(_, _)`.\n", + "\n", + "Before you can run the data through evaluation, you need to preprocess it:\n", + "\n", + "`x_test`:\n", + "1. Normalize (rescale) the data in each column by dividing each value by the maximum value of that column. This replaces each single value with a 32-bit floating point number between 0 and 1." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "evaluate_custom_model:tabular,boston" + }, + "outputs": [], + "source": [ + "import numpy as np\n", + "from tensorflow.keras.datasets import boston_housing\n", + "\n", + "(_, _), (x_test, y_test) = boston_housing.load_data(\n", + " path=\"boston_housing.npz\", test_split=0.2, seed=113\n", + ")\n", + "\n", + "\n", + "def scale(feature):\n", + " max = np.max(feature)\n", + " feature = (feature / max).astype(np.float32)\n", + " return feature\n", + "\n", + "\n", + "# Let's save one data item that has not been scaled\n", + "x_test_notscaled = x_test[0:1].copy()\n", + "\n", + "for _ in range(13):\n", + " x_test[_] = scale(x_test[_])\n", + "x_test = x_test.astype(np.float32)\n", + "\n", + "print(x_test.shape, x_test.dtype, y_test.shape)\n", + "print(\"scaled\", x_test[0])\n", + "print(\"unscaled\", x_test_notscaled)" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "perform_evaluation_custom" + }, + "source": [ + "### Perform the model evaluation\n", + "\n", + "Now evaluate how well the model in the custom job did." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "perform_evaluation_custom" + }, + "outputs": [], + "source": [ + "model.evaluate(x_test, y_test)" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "how_serving_function_works" + }, + "source": [ + "## Upload the model for serving\n", + "\n", + "Next, you will upload your TF.Keras model from the custom job to Vertex `Model` service, which will create a Vertex `Model` resource for your custom model. During upload, you need to define a serving function to convert data to the format your model expects. If you send encoded data to Vertex, your serving function ensures that the data is decoded on the model server before it is passed as input to your model.\n", + "\n", + "### How does the serving function work\n", + "\n", + "When you send a request to an online prediction server, the request is received by a HTTP server. The HTTP server extracts the prediction request from the HTTP request content body. The extracted prediction request is forwarded to the serving function. For Google pre-built prediction containers, the request content is passed to the serving function as a `tf.string`.\n", + "\n", + "The serving function consists of two parts:\n", + "\n", + "- `preprocessing function`:\n", + " - Converts the input (`tf.string`) to the input shape and data type of the underlying model (dynamic graph).\n", + " - Performs the same preprocessing of the data that was done during training the underlying model, including normalizing and scalingh.\n", + "- `post-processing function`:\n", + " - Converts the model output to format expected by the receiving application -- e.q., compresses the output.\n", + " - Packages the output for the the receiving application, including adding headings, and making JSON objects.\n", + "\n", + "Both the preprocessing and post-processing functions are converted to static graphs which are fused to the model. The output from the underlying model is passed to the post-processing function. The post-processing function passes the converted/packaged output back to the HTTP server. The HTTP server returns the output as the HTTP response content.\n", + "\n", + "One consideration you need to consider when building serving functions for TF.Keras models is that they run as static graphs. That means, you cannot use TF graph operations that require a dynamic graph. If you do, you will get an error during the compile of the serving function which will indicate that you are using an EagerTensor which is not supported." + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "serving_function_signature:xai" + }, + "source": [ + "## Get the serving function signature\n", + "\n", + "You can get the signatures of your model's input and output layers by reloading the model into memory, and querying it for the signatures corresponding to each layer.\n", + "\n", + "When making a prediction request, you need to route the request to the serving function instead of the model, so you need to know the input layer name of the serving function -- which you will use later when you make a prediction request.\n", + "\n", + "You also need to know the name of the serving function's input and output layer for constructing the explanation metadata -- which is discussed subsequently." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "serving_function_signature:xai" + }, + "outputs": [], + "source": [ + "loaded = tf.saved_model.load(model_path_to_deploy)\n", + "\n", + "serving_input = list(\n", + " loaded.signatures[\"serving_default\"].structured_input_signature[1].keys()\n", + ")[0]\n", + "print(\"Serving function input:\", serving_input)\n", + "serving_output = list(loaded.signatures[\"serving_default\"].structured_outputs.keys())[0]\n", + "print(\"Serving function output:\", serving_output)\n", + "\n", + "input_name = model.input.name\n", + "print(\"Model input name:\", input_name)\n", + "output_name = model.output.name\n", + "print(\"Model output name:\", output_name)" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "explanation_spec:ig" + }, + "source": [ + "### Explanation Specification\n", + "\n", + "To get explanations when doing a prediction, you must enable the explanation capability and set corresponding settings when you upload your custom model to an Vertex `Model` resource. These settings are referred to as the explanation metadata, which consists of:\n", + "\n", + "- `parameters`: This is the specification for the explainability algorithm to use for explanations on your model.\n", + "- `metadata`: This is the specification for how the algoithm is applied on your custom model.\n", + "\n", + "See [Compare Methods](https://cloud.google.com/vertex-ai/docs/explainable-ai/overview#compare-method) for a comparision of supported algoritmes.\n", + "\n", + "In this tutorial, you use the `integrated gradients` algorithm, which has the following parameter:\n", + "\n", + "`step_count`: This is the number of steps to approximate the remaining sum. The more steps, the more accurate the integral approximation. The general rule of thumb is 50 steps, but as you increase so does the compute time." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "explanation_spec:ig" + }, + "outputs": [], + "source": [ + "XAI = \"ig\" # [ shapley, ig, xrai ]\n", + "\n", + "PARAMETERS = {\"integrated_gradients_attribution\": {\"step_count\": 50}}\n", + "\n", + "\n", + "parameters = aip.ExplanationParameters(PARAMETERS)" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "explanation_metadata:tabular" + }, + "source": [ + "#### Explanation Metadata\n", + "\n", + "Let's first dive deeper into the explanation metadata, which consists of:\n", + "\n", + "- `outputs`: A scalar value in the output to attribute -- what to explain. For example, in a probability output \\[0.1, 0.2, 0.7\\] for classification, one wants an explanation for 0.7. Consider the following formulae, where the output is `y` and that is what we want to explain.\n", + "\n", + " y = f(x)\n", + "\n", + "Consider the following formulae, where the outputs are `y` and `z`. Since we can only do attribution for one scalar value, we have to pick whether we want to explain the output `y` or `z`. Assume in this example the model is object detection and y and z are the bounding box and the object classification. You would want to pick which of the two outputs to explain.\n", + "\n", + " y, z = f(x)\n", + "\n", + "The dictionary format for `outputs` is:\n", + "\n", + " { \"outputs\": { \"[your_display_name]\":\n", + " \"output_tensor_name\": [layer]\n", + " }\n", + " }\n", + "\n", + "
\n", + "
    \n", + "
  • [your_display_name]: A human readable name you assign to the output to explain. A common example is \"probability\".
    • \n", + "
  • \"output_tensor_name\": The key/value field to identify the output layer to explain.
    • \n", + "
  • [layer]: The output layer to explain. In a single task model, like a tabular regressor, it is the last (topmost) layer in the model
    • .\n", + "
\n", + "
\n", + "\n", + "\n", + "- `inputs`: The features for attribution -- how they contributed to the output. Consider the following formulae, where `a` and `b` are the features. We have to pick which features to explain how the contributed. Assume that this model is deployed for A/B testing, where `a` are the data_items for the prediction and `b` identifies whether the model instance is A or B. You would want to pick `a` (or some subset of) for the features, and not `b` since it does not contribute to the prediction.\n", + "\n", + " y = f(a,b)\n", + "\n", + "The minimum dictionary format for `inputs` is:\n", + "\n", + " { \"inputs\": { \"[your_display_name]\":\n", + " \"input_tensor_name\": [layer]\n", + " }\n", + " }\n", + "\n", + "
\n", + "
    \n", + "
  • [your_display_name]: A human readable name you assign to the input to explain. A common example is \"features\".
    • \n", + "
  • \"input_tensor_name\": The key/value field to identify the input layer for the feature attribution.
    • \n", + "
  • [layer]: The input layer for feature attribution. In a single input tensor model, it is the first (bottom-most) layer in the model.
    • \n", + "
\n", + "
\n", + "\n", + "Since the inputs to the model are tabular, you can specify the following two additional fields as reporting/visualization aids:\n", + "\n", + "
\n", + "
    \n", + "
  • \"encoding\": \"BAG_OF_FEATURES\" : Indicates that the inputs are set of tabular features.
    • \n", + "
  • \"index_feature_mapping\": [ feature-names ] : A list of human readable names for each feature. For this example, we use the feature names specified in the dataset.
    • \n", + "
  • \"modality\": \"numeric\": Indicates the field values are numeric.
    • \n", + "
\n", + "
" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "explanation_metadata:tabular" + }, + "outputs": [], + "source": [ + "INPUT_METADATA = {\n", + " \"input_tensor_name\": serving_input,\n", + " \"encoding\": \"BAG_OF_FEATURES\",\n", + " \"modality\": \"numeric\",\n", + " \"index_feature_mapping\": [\n", + " \"crim\",\n", + " \"zn\",\n", + " \"indus\",\n", + " \"chas\",\n", + " \"nox\",\n", + " \"rm\",\n", + " \"age\",\n", + " \"dis\",\n", + " \"rad\",\n", + " \"tax\",\n", + " \"ptratio\",\n", + " \"b\",\n", + " \"lstat\",\n", + " ],\n", + "}\n", + "\n", + "OUTPUT_METADATA = {\"output_tensor_name\": serving_output}\n", + "\n", + "input_metadata = aip.ExplanationMetadata.InputMetadata(INPUT_METADATA)\n", + "output_metadata = aip.ExplanationMetadata.OutputMetadata(OUTPUT_METADATA)\n", + "\n", + "metadata = aip.ExplanationMetadata(\n", + " inputs={\"features\": input_metadata}, outputs={\"medv\": output_metadata}\n", + ")\n", + "\n", + "explanation_spec = aip.ExplanationSpec(metadata=metadata, parameters=parameters)" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "upload_the_model:explanation" + }, + "source": [ + "### Upload the model\n", + "\n", + "Use this helper function `upload_model` to upload your model, stored in SavedModel format, up to the `Model` service, which will instantiate a Vertex `Model` resource instance for your model. Once you've done that, you can use the `Model` resource instance in the same way as any other Vertex `Model` resource instance, such as deploying to an `Endpoint` resource for serving predictions.\n", + "\n", + "The helper function takes the following parameters:\n", + "\n", + "- `display_name`: A human readable name for the `Endpoint` service.\n", + "- `image_uri`: The container image for the model deployment.\n", + "- `model_uri`: The Cloud Storage path to our SavedModel artificat. For this tutorial, this is the Cloud Storage location where the `trainer/task.py` saved the model artifacts, which we specified in the variable `MODEL_DIR`.\n", + "\n", + "The helper function calls the `Model` client service's method `upload_model`, which takes the following parameters:\n", + "\n", + "- `parent`: The Vertex location root path for `Dataset`, `Model` and `Endpoint` resources.\n", + "- `model`: The specification for the Vertex `Model` resource instance.\n", + "\n", + "Let's now dive deeper into the Vertex model specification `model`. This is a dictionary object that consists of the following fields:\n", + "\n", + "- `display_name`: A human readable name for the `Model` resource.\n", + "- `metadata_schema_uri`: Since your model was built without an Vertex `Dataset` resource, you will leave this blank (`''`).\n", + "- `artifact_uri`: The Cloud Storage path where the model is stored in SavedModel format.\n", + "- `container_spec`: This is the specification for the Docker container that will be installed on the `Endpoint` resource, from which the `Model` resource will serve predictions. Use the variable you set earlier `DEPLOY_GPU != None` to use a GPU; otherwise only a CPU is allocated.\n", + "- `explanation_spec`: This is the specification for enabling explainability for your model.\n", + "\n", + "Uploading a model into a Vertex Model resource returns a long running operation, since it may take a few moments. You call response.result(), which is a synchronous call and will return when the Vertex Model resource is ready.\n", + "\n", + "The helper function returns the Vertex fully qualified identifier for the corresponding Vertex Model instance upload_model_response.model. You will save the identifier for subsequent steps in the variable model_to_deploy_id." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "upload_the_model:explanation" + }, + "outputs": [], + "source": [ + "IMAGE_URI = DEPLOY_IMAGE\n", + "\n", + "\n", + "def upload_model(display_name, image_uri, model_uri):\n", + "\n", + " model = aip.Model(\n", + " display_name=display_name,\n", + " artifact_uri=model_uri,\n", + " metadata_schema_uri=\"\",\n", + " explanation_spec=explanation_spec,\n", + " container_spec={\"image_uri\": image_uri},\n", + " )\n", + "\n", + " response = clients[\"model\"].upload_model(parent=PARENT, model=model)\n", + " print(\"Long running operation:\", response.operation.name)\n", + " upload_model_response = response.result(timeout=180)\n", + " print(\"upload_model_response\")\n", + " print(\" model:\", upload_model_response.model)\n", + " return upload_model_response.model\n", + "\n", + "\n", + "model_to_deploy_id = upload_model(\n", + " \"boston-\" + TIMESTAMP, IMAGE_URI, model_path_to_deploy\n", + ")" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "get_model" + }, + "source": [ + "### Get `Model` resource information\n", + "\n", + "Now let's get the model information for just your model. Use this helper function `get_model`, with the following parameter:\n", + "\n", + "- `name`: The Vertex unique identifier for the `Model` resource.\n", + "\n", + "This helper function calls the Vertex `Model` client service's method `get_model`, with the following parameter:\n", + "\n", + "- `name`: The Vertex unique identifier for the `Model` resource." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "get_model" + }, + "outputs": [], + "source": [ + "def get_model(name):\n", + " response = clients[\"model\"].get_model(name=name)\n", + " print(response)\n", + "\n", + "\n", + "get_model(model_to_deploy_id)" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "create_endpoint:custom" + }, + "source": [ + "## Deploy the `Model` resource\n", + "\n", + "Now deploy the trained Vertex custom `Model` resource. This requires two steps:\n", + "\n", + "1. Create an `Endpoint` resource for deploying the `Model` resource to.\n", + "\n", + "2. Deploy the `Model` resource to the `Endpoint` resource." + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "create_endpoint" + }, + "source": [ + "### Create an `Endpoint` resource\n", + "\n", + "Use this helper function `create_endpoint` to create an endpoint to deploy the model to for serving predictions, with the following parameter:\n", + "\n", + "- `display_name`: A human readable name for the `Endpoint` resource.\n", + "\n", + "The helper function uses the endpoint client service's `create_endpoint` method, which takes the following parameter:\n", + "\n", + "- `display_name`: A human readable name for the `Endpoint` resource.\n", + "\n", + "Creating an `Endpoint` resource returns a long running operation, since it may take a few moments to provision the `Endpoint` resource for serving. You call `response.result()`, which is a synchronous call and will return when the Endpoint resource is ready. The helper function returns the Vertex fully qualified identifier for the `Endpoint` resource: `response.name`." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "create_endpoint" + }, + "outputs": [], + "source": [ + "ENDPOINT_NAME = \"boston_endpoint-\" + TIMESTAMP\n", + "\n", + "\n", + "def create_endpoint(display_name):\n", + " endpoint = {\"display_name\": display_name}\n", + " response = clients[\"endpoint\"].create_endpoint(parent=PARENT, endpoint=endpoint)\n", + " print(\"Long running operation:\", response.operation.name)\n", + "\n", + " result = response.result(timeout=300)\n", + " print(\"result\")\n", + " print(\" name:\", result.name)\n", + " print(\" display_name:\", result.display_name)\n", + " print(\" description:\", result.description)\n", + " print(\" labels:\", result.labels)\n", + " print(\" create_time:\", result.create_time)\n", + " print(\" update_time:\", result.update_time)\n", + " return result\n", + "\n", + "\n", + "result = create_endpoint(ENDPOINT_NAME)" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "endpoint_id:result" + }, + "source": [ + "Now get the unique identifier for the `Endpoint` resource you created." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "endpoint_id:result" + }, + "outputs": [], + "source": [ + "# The full unique ID for the endpoint\n", + "endpoint_id = result.name\n", + "# The short numeric ID for the endpoint\n", + "endpoint_short_id = endpoint_id.split(\"/\")[-1]\n", + "\n", + "print(endpoint_id)" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "instance_scaling" + }, + "outputs": [], + "source": [ + "MIN_NODES = 1\n", + "MAX_NODES = 1" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "deploy_model:dedicated,v1beta1" + }, + "source": [ + "### Deploy `Model` resource to the `Endpoint` resource\n", + "\n", + "Use this helper function `deploy_model` to deploy the model to the endpoint you created for serving predictions, with the following parameters:\n", + "\n", + "- `model`: The Vertex fully qualified identifier of the `Model` resource to upload (deploy) from the training pipeline.\n", + "- `deploy_model_display_name`: A human readable name for the deployed model.\n", + "- `endpoint`: The Vertex fully qualified `Endpoint` resource identifier to deploy the `Model` resource to.\n", + "\n", + "The helper function calls the `Endpoint` client service's method `deploy_model`, which takes the following parameters:\n", + "\n", + "- `endpoint`: The Vertex fully qualified `Endpoint` resource identifier to deploy the `Model` resource to to.\n", + "- `deployed_model`: The requirements for deploying the model.\n", + "- `traffic_split`: Percent of traffic at endpoint that goes to this model, which is specified as a dictioney of one or more key/value pairs.\n", + " - If only one model, then specify as **{ \"0\": 100 }**, where \"0\" refers to this model being uploaded and 100 means 100% of the traffic.\n", + " - If there are existing models on the endpoint, for which the traffic will be split, then specify as, where `model_id` is the model id of an existing model to the deployed endpoint. The percents must add up to 100.\n", + "\n", + " { \"0\": percent, model_id: percent, ... }\n", + "\n", + "Let's now dive deeper into the `deployed_model` parameter. This parameter is specified as a Python dictionary with the minimum required fields:\n", + "\n", + "- `model`: The Vertex fully qualified identifier of the (upload) `Model` resource to deploy.\n", + "- `display_name`: A human readable name for the deployed model.\n", + "- `dedicated_resources`: This refers to how many compute instances (replicas) that are scaled for serving prediction requests.\n", + " - `machine_spec`: The compute instance to provision. Use the variable you set earlier `DEPLOY_GPU != None` to use a GPU; otherwise only a CPU is allocated.\n", + " - `min_replica_count`: The number of compute instances to initially provision, which you set earlier as the variable `MIN_NODES`.\n", + " - `max_replica_count`: The maximum number of compute instances to scale to, which you set earlier as the variable `MAX_NODES`.\n", + "- `enable_container_logging`: This enables logging of container events, such as execution failures (default is container logging is disabled). Container logging is typically enabled when debugging the deployment and then disabled when deployed for production.\n", + "\n", + "#### Traffic Split\n", + "\n", + "Let's now dive deeper into the `traffic_split` parameter. This parameter is specified as a Python dictionary. This might at first be a tad bit confusing. Let me explain, you can deploy more than one instance of your model to an endpoint, and then set how much (percent) goes to each instance.\n", + "\n", + "Why would you do that? Perhaps you already have a previous version deployed in production -- let's call that v1. You got better model evaluation on v2, but you don't know for certain that it is really better until you deploy to production. So in the case of traffic split, you might want to deploy v2 to the same endpoint as v1, but it only get's say 10% of the traffic. That way, you can monitor how well it does without disrupting the majority of users -- until you make a final decision.\n", + "\n", + "#### Response\n", + "\n", + "The method returns a long running operation `response`. We will wait sychronously for the operation to complete by calling the `response.result()`, which will block until the model is deployed. If this is the first time a model is deployed to the endpoint, it may take a few additional minutes to complete provisioning of resources." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "deploy_model:dedicated,v1beta1" + }, + "outputs": [], + "source": [ + "DEPLOYED_NAME = \"boston_deployed-\" + TIMESTAMP\n", + "\n", + "\n", + "def deploy_model(\n", + " model, deployed_model_display_name, endpoint, traffic_split={\"0\": 100}\n", + "):\n", + "\n", + " if DEPLOY_GPU:\n", + " machine_spec = {\n", + " \"machine_type\": DEPLOY_COMPUTE,\n", + " \"accelerator_type\": DEPLOY_GPU,\n", + " \"accelerator_count\": DEPLOY_NGPU,\n", + " }\n", + " else:\n", + " machine_spec = {\n", + " \"machine_type\": DEPLOY_COMPUTE,\n", + " \"accelerator_count\": 0,\n", + " }\n", + "\n", + " deployed_model = {\n", + " \"model\": model,\n", + " \"display_name\": deployed_model_display_name,\n", + " \"dedicated_resources\": {\n", + " \"min_replica_count\": MIN_NODES,\n", + " \"max_replica_count\": MAX_NODES,\n", + " \"machine_spec\": machine_spec,\n", + " },\n", + " \"enable_container_logging\": False,\n", + " }\n", + "\n", + " response = clients[\"endpoint\"].deploy_model(\n", + " endpoint=endpoint, deployed_model=deployed_model, traffic_split=traffic_split\n", + " )\n", + "\n", + " print(\"Long running operation:\", response.operation.name)\n", + " result = response.result()\n", + " print(\"result\")\n", + " deployed_model = result.deployed_model\n", + " print(\" deployed_model\")\n", + " print(\" id:\", deployed_model.id)\n", + " print(\" model:\", deployed_model.model)\n", + " print(\" display_name:\", deployed_model.display_name)\n", + " print(\" create_time:\", deployed_model.create_time)\n", + "\n", + " return deployed_model.id\n", + "\n", + "\n", + "deployed_model_id = deploy_model(model_to_deploy_id, DEPLOYED_NAME, endpoint_id)" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "make_prediction:xai" + }, + "source": [ + "## Send a online prediction request with explainability\n", + "\n", + "Send a online prediction with explainability to your deployed model. In this method, the predicted response will include an explanation on how the features contributed to the explanation." + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "get_test_item:test" + }, + "source": [ + "### Get test item\n", + "\n", + "You will use an example out of the test (holdout) portion of the dataset as a test item." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "get_test_item:test,tabular" + }, + "outputs": [], + "source": [ + "test_item = x_test[0]\n", + "test_label = y_test[0]\n", + "print(test_item.shape)" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "send_explain_request:tabular" + }, + "source": [ + "### Send the prediction with explanation request\n", + "\n", + "Ok, now you have a test data item. Use this helper function `explain_data`, which takes the parameters:\n", + "\n", + "- `data_item`: The test data items as a numpy 1D array of floating point values.\n", + "- `endpoint`: The AI Platform (Unified) fully qualified identifier for the endpoint where the model was deployed.\n", + "- `parameters_dict`: Additional parameters for serving.\n", + "\n", + "This function uses the prediction client service and calls the `explain` method with the parameters:\n", + "\n", + "- `endpoint`: The AI Platform (Unified) fully qualified identifier for the endpoint where the model was deployed.\n", + "- `instances`: A list of instances (data items) to predict.\n", + "- `parameters`: Additional parameters for serving. *Note*, custom models do not support additional parameters.\n", + "- `deployed_model_id`: The AI Platform (Unified) fully qualified identifier for the deployed model, when more than one model is deployed at the endpoint. Otherwise, if only one model deployed, can be set to `None`.\n", + "\n", + "To pass the test data to the prediction service, you must package it for transmission to the serving binary as follows:\n", + "\n", + " 1. Convert the data item from a 1D numpy array to a 1D Python list.\n", + " 2. Convert the prediction request to a serialized Google protobuf (`json_format.ParseDict()`)\n", + "\n", + "\n", + "Each instance in the prediction request is a dictionary entry of the form:\n", + "\n", + " {input_name: content}\n", + "\n", + "- `input_name`: the name of the input layer of the underlying model.\n", + "- `content`: The data item as a 1D Python list.\n", + "\n", + "Since the `explain()` method can take multiple data items (instances), you will send your single data item as a list of one data item. As a final step, you package the instances list into Google's protobuf format -- which is what we pass to the `explain()` method.\n", + "\n", + "The `response` object returns a list, where each element in the list corresponds to the corresponding image in the request. You will see in the output for each prediction:\n", + "\n", + "- `deployed_model_id` -- The AI Platform (Unified) fully qualified identifer for the model that did the prediction/explanation.\n", + "- `predictions` -- The predicated median value of a house in units of 1K USD.\n", + "- `explanations` -- How each feature contributed to the prediction." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "send_explain_request:tabular" + }, + "outputs": [], + "source": [ + "def explain_data(\n", + " data_items, endpoint, parameters_dict, deployed_model_id, silent=False\n", + "):\n", + " parameters = json_format.ParseDict(parameters_dict, Value())\n", + "\n", + " # The format of each instance should conform to the deployed model's prediction input schema.\n", + " instances = [\n", + " json_format.ParseDict({serving_input: s.tolist()}, Value()) for s in data_items\n", + " ]\n", + "\n", + " response = clients[\"prediction\"].explain(\n", + " endpoint=endpoint,\n", + " instances=instances,\n", + " parameters=parameters,\n", + " deployed_model_id=deployed_model_id,\n", + " )\n", + " if silent:\n", + " return response\n", + "\n", + " print(\"response\")\n", + " print(\" deployed_model_id:\", response.deployed_model_id)\n", + "\n", + " predictions = response.predictions\n", + " print(\"predictions\")\n", + " for prediction in predictions:\n", + " print(\" prediction:\", prediction)\n", + "\n", + " explanations = response.explanations\n", + " print(\"explanations\")\n", + " for explanation in explanations:\n", + " print(explanation)\n", + " return response\n", + "\n", + "\n", + "response = explain_data([test_item], endpoint_id, None, None)" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "understanding_explanations:boston" + }, + "source": [ + "### Understanding the explanations response\n", + "\n", + "First, let's look at the median house price your model predicted and compare it to the actual value." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "understanding_explanations:boston" + }, + "outputs": [], + "source": [ + "predictions = response.predictions\n", + "print(\"Predicted Median House Value:\", predictions[0][0], \"1K USD\")\n", + "print(\"Actual Mean House Value:\", test_label, \"1K USD\")" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "examine_feature_attributions" + }, + "source": [ + "### Examine feature attributions\n", + "\n", + "Next you will look at the feature attributions for this particular example. Positive attribution values mean a particular feature pushed your model prediction up by that amount, and vice versa for negative attribution values." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "examine_feature_attributions:boston" + }, + "outputs": [], + "source": [ + "from tabulate import tabulate\n", + "\n", + "feature_names = [\n", + " \"crim\",\n", + " \"zn\",\n", + " \"indus\",\n", + " \"chas\",\n", + " \"nox\",\n", + " \"rm\",\n", + " \"age\",\n", + " \"dis\",\n", + " \"rad\",\n", + " \"tax\",\n", + " \"ptratio\",\n", + " \"b\",\n", + " \"lstat\",\n", + "]\n", + "attributions = response.explanations[0].attributions[0].feature_attributions\n", + "\n", + "rows = []\n", + "for i, val in enumerate(feature_names):\n", + " rows.append([val, test_item[i], attributions[val]])\n", + "print(tabulate(rows, headers=[\"Feature name\", \"Feature value\", \"Attribution value\"]))" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "check_explanations_baselines" + }, + "source": [ + "### Check your explanations and baselines\n", + "\n", + "To better make sense of the feature attributions you're getting, you should compare them with your model's baseline. In most cases, the sum of your attribution values + the baseline should be very close to your model's predicted value for each input. Also note that for regression models, the `baseline_score` returned from AI Explanations will be the same for each example sent to your model. For classification models, each class will have its own baseline.\n", + "\n", + "In this section you'll send 10 test examples to your model for prediction in order to compare the feature attributions with the baseline. Then you'll run each test example's attributions through a sanity check in the `sanity_check_explanations` method.\n", + "\n", + "#### Get explanations" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "check_explanations_baselines:boston" + }, + "outputs": [], + "source": [ + "# Prepare 10 test examples to your model for prediction\n", + "pred_batch = []\n", + "for i in range(10):\n", + " pred_batch.append(x_test[i])\n", + "\n", + "response = explain_data(pred_batch, endpoint_id, None, None, silent=True)" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "sanity_check_explanations" + }, + "source": [ + "#### Sanity check\n", + "\n", + "In the function below you perform a sanity check on the explanations." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "sanity_check_explanations" + }, + "outputs": [], + "source": [ + "def sanity_check_explanations(\n", + " explanation, prediction, mean_tgt_value=None, variance_tgt_value=None\n", + "):\n", + " passed_test = 0\n", + " total_test = 1\n", + " # `attributions` is a dict where keys are the feature names\n", + " # and values are the feature attributions for each feature\n", + " baseline_score = explanation.attributions[0].baseline_output_value\n", + " print(\"baseline:\", baseline_score)\n", + "\n", + " # Sanity check 1\n", + " # The prediction at the input is equal to that at the baseline.\n", + " # Please use a different baseline. Some suggestions are: random input, training\n", + " # set mean.\n", + " if abs(prediction - baseline_score) <= 0.05:\n", + " print(\"Warning: example score and baseline score are too close.\")\n", + " print(\"You might not get attributions.\")\n", + " else:\n", + " passed_test += 1\n", + " print(\"Sanity Check 1: Passed\")\n", + "\n", + " print(passed_test, \" out of \", total_test, \" sanity checks passed.\")\n", + "\n", + "\n", + "i = 0\n", + "for explanation in response.explanations:\n", + " try:\n", + " prediction = np.max(response.predictions[i][\"scores\"])\n", + " except TypeError:\n", + " prediction = np.max(response.predictions[i])\n", + " sanity_check_explanations(explanation, prediction)\n", + " i += 1" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "undeploy_model" + }, + "source": [ + "## Undeploy the `Model` resource\n", + "\n", + "To undeploy your `Model` resource from the serving `Endpoint` resoure, use this helper function `undeploy_model`, which takes the following parameters:\n", + "\n", + "- `deployed_model_id`: The model deployment identifier returned by the endpoint service when the `Model` resource was deployed to.\n", + "- `endpoint`: The Vertex fully qualified identifier for the `Endpoint` resource where the `Model` is deployed to.\n", + "\n", + "This function calls the endpoint client service's method `undeploy_model`, with the following parameters:\n", + "\n", + "- `deployed_model_id`: The model deployment identifier returned by the endpoint service when the `Model` resource was deployed.\n", + "- `endpoint`: The Vertex fully qualified identifier for the `Endpoint` resource where the `Model` resource is deployed.\n", + "- `traffic_split`: How to split traffic among the remaining deployed models on the `Endpoint` resource.\n", + "\n", + "Since this is the only deployed model on the `Endpoint` resource, you can leave `traffic_split` empty by setting it to {}." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "undeploy_model" + }, + "outputs": [], + "source": [ + "def undeploy_model(deployed_model_id, endpoint):\n", + " response = clients[\"endpoint\"].undeploy_model(\n", + " endpoint=endpoint, deployed_model_id=deployed_model_id, traffic_split={}\n", + " )\n", + " print(response)\n", + "\n", + "\n", + "undeploy_model(deployed_model_id, endpoint_id)" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "cleanup" + }, + "source": [ + "# Cleaning up\n", + "\n", + "To clean up all Google Cloud resources used in this project, you can [delete the GCP\n", + "project](https://cloud.google.com/resource-manager/docs/creating-managing-projects#shutting_down_projects) you used for the tutorial.\n", + "\n", + "Otherwise, you can delete the individual resources you created in this tutorial:\n", + "\n", + "- Dataset\n", + "- Pipeline\n", + "- Model\n", + "- Endpoint\n", + "- Batch Job\n", + "- Custom Job\n", + "- Hyperparameter Tuning Job\n", + "- Cloud Storage Bucket" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "cleanup" + }, + "outputs": [], + "source": [ + "delete_dataset = True\n", + "delete_pipeline = True\n", + "delete_model = True\n", + "delete_endpoint = True\n", + "delete_batchjob = True\n", + "delete_customjob = True\n", + "delete_hptjob = True\n", + "delete_bucket = True\n", + "\n", + "# Delete the dataset using the Vertex fully qualified identifier for the dataset\n", + "try:\n", + " if delete_dataset and \"dataset_id\" in globals():\n", + " clients[\"dataset\"].delete_dataset(name=dataset_id)\n", + "except Exception as e:\n", + " print(e)\n", + "\n", + "# Delete the training pipeline using the Vertex fully qualified identifier for the pipeline\n", + "try:\n", + " if delete_pipeline and \"pipeline_id\" in globals():\n", + " clients[\"pipeline\"].delete_training_pipeline(name=pipeline_id)\n", + "except Exception as e:\n", + " print(e)\n", + "\n", + "# Delete the model using the Vertex fully qualified identifier for the model\n", + "try:\n", + " if delete_model and \"model_to_deploy_id\" in globals():\n", + " clients[\"model\"].delete_model(name=model_to_deploy_id)\n", + "except Exception as e:\n", + " print(e)\n", + "\n", + "# Delete the endpoint using the Vertex fully qualified identifier for the endpoint\n", + "try:\n", + " if delete_endpoint and \"endpoint_id\" in globals():\n", + " clients[\"endpoint\"].delete_endpoint(name=endpoint_id)\n", + "except Exception as e:\n", + " print(e)\n", + "\n", + "# Delete the batch job using the Vertex fully qualified identifier for the batch job\n", + "try:\n", + " if delete_batchjob and \"batch_job_id\" in globals():\n", + " clients[\"job\"].delete_batch_prediction_job(name=batch_job_id)\n", + "except Exception as e:\n", + " print(e)\n", + "\n", + "# Delete the custom job using the Vertex fully qualified identifier for the custom job\n", + "try:\n", + " if delete_customjob and \"job_id\" in globals():\n", + " clients[\"job\"].delete_custom_job(name=job_id)\n", + "except Exception as e:\n", + " print(e)\n", + "\n", + "# Delete the hyperparameter tuning job using the Vertex fully qualified identifier for the hyperparameter tuning job\n", + "try:\n", + " if delete_hptjob and \"hpt_job_id\" in globals():\n", + " clients[\"job\"].delete_hyperparameter_tuning_job(name=hpt_job_id)\n", + "except Exception as e:\n", + " print(e)\n", + "\n", + "if delete_bucket and \"BUCKET_NAME\" in globals():\n", + " ! gsutil rm -r $BUCKET_NAME" + ] + } + ], + "metadata": { + "colab": { + "name": "gapic-custom_tabular_regression_online_explain.ipynb", + "toc_visible": true + }, + "environment": { + "name": "common-cu110.m71", + "type": "gcloud", + "uri": "gcr.io/deeplearning-platform-release/base-cu110:m71" + }, + "kernelspec": { + "display_name": "Python 3", + "language": "python", + "name": "python3" + }, + "language_info": { + "codemirror_mode": { + "name": "ipython", + "version": 3 + }, + "file_extension": ".py", + "mimetype": "text/x-python", + "name": "python", + "nbconvert_exporter": "python", + "pygments_lexer": "ipython3", + "version": "3.7.10" + } + }, + "nbformat": 4, + "nbformat_minor": 4 +}