xiaohei/model_garden
1
0
Fork 0
mirror of https://github.com/GoogleCloudPlatform/vertex-ai-samples.git synced 2026-09-26 14:42:04 +00:00
Code Issues Packages Projects Releases Wiki Activity

Fixes the exception in batch-explain notebook in the official folder (#810)

Browse Source 
* fixes exception(reg-test), replaces timestamp with uuid, minor changes

* ran linter test

* resolved review comments: license year, Vertex AI SDK, dataset after objective and future tense

* ran linter test

Co-authored-by: Andrew Ferlitsch <aferlitsch@google.com>
This commit is contained in:
Krishna Chaitanya Movva
2022-08-11 10:25:51 -07:00
committed by GitHub
co-authored by Andrew Ferlitsch
parent b9d4457474
commit 161965cfb0
1 changed files with 86 additions and 63 deletions
Download Patch File Download Diff File Expand all files Collapse all files
notebooks/official/explainable_ai/sdk_custom_image_classification_batch_explain.ipynb
+86 -63
View File
@@ -29,22 +29,22 @@
"id": "title"
},
"source": [
"# Vertex SDK: Custom training image classification model for batch prediction with explainabilty\n",
"# Vertex AI SDK: Custom training image classification model for batch prediction with explainabilty\n",
"\n",
"<table align=\"left\">\n",
" <td>\n",
" <a href=\"https://colab.research.google.com/github/GoogleCloudPlatform/vertex-ai-samples/blob/main/notebooks/official/automl/sdk_custom_image_classification_batch_explain.ipynb\">\n",
" <a href=\"https://colab.research.google.com/github/GoogleCloudPlatform/vertex-ai-samples/blob/main/notebooks/official/explainable_ai/sdk_custom_image_classification_batch_explain.ipynb\">\n",
" <img src=\"https://cloud.google.com/ml-engine/images/colab-logo-32px.png\" alt=\"Colab logo\"> Run in Colab\n",
" </a>\n",
" </td>\n",
" <td>\n",
" <a href=\"https://github.com/GoogleCloudPlatform/vertex-ai-samples/blob/main/notebooks/official/automl/sdk_custom_image_classification_batch_explain.ipynb\">\n",
" <a href=\"https://github.com/GoogleCloudPlatform/vertex-ai-samples/blob/main/notebooks/official/explainable_ai/sdk_custom_image_classification_batch_explain.ipynb\">\n",
" <img src=\"https://cloud.google.com/ml-engine/images/github-logo-32px.png\" alt=\"GitHub logo\">\n",
" View on GitHub\n",
" </a>\n",
" </td>\n",
" <td>\n",
" <a href=\"https://console.cloud.google.com/vertex-ai/workbench/deploy-notebook?download_url=https://raw.githubusercontent.com/GoogleCloudPlatform/vertex-ai-samples/tree/main/notebooks/official/explainable_ai/sdk_custom_image_classification_batch_explain.ipynb\">\n",
" <a href=\"https://console.cloud.google.com/vertex-ai/workbench/deploy-notebook?download_url=https://raw.githubusercontent.com/GoogleCloudPlatform/vertex-ai-samples/blob/main/notebooks/official/explainable_ai/sdk_custom_image_classification_batch_explain.ipynb\">\n",
" <img src=\"https://lh3.googleusercontent.com/UiNooY4LUgW_oTvpsNhPpQzsstV5W8F7rYgxgGBD85cWJoLmrOzhVs_ksK_vgx40SHs7jCqkTkCk=e14-rj-sc0xffffff-h130-w32\" alt=\"Vertex AI logo\">\n",
" Open in Vertex AI Workbench\n",
" </a>\n",
@@ -65,17 +65,6 @@
"This tutorial demonstrates how to use the Vertex AI SDK to train and deploy a custom image classification model for batch prediction with explanation."
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "dataset:custom,cifar10,icn"
},
"source": [
"### Dataset\n",
"\n",
"The dataset used for this tutorial is the [CIFAR10 dataset](https://www.tensorflow.org/datasets/catalog/cifar10) from [TensorFlow Datasets](https://www.tensorflow.org/datasets/catalog/overview). The version of the dataset you will use is built into TensorFlow. The trained model predicts which type of class an image is from ten classes: airplane, automobile, bird, cat, deer, dog, frog, horse, ship, truck."
]
},
{
"cell_type": "markdown",
"metadata": {
@@ -91,7 +80,7 @@
"- `Vertex AI Training`\n",
"- `Vertex AI Batch Prediction`\n",
"- `Vertex Explainable AI`\n",
"- `Vertex AI Model` resource\n",
"- `Vertex AI Models`\n",
"\n",
"The steps performed include:\n",
"\n",
@@ -102,6 +91,17 @@
"- Make a batch prediction with explanations."
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "dataset:custom,cifar10,icn"
},
"source": [
"### Dataset\n",
"\n",
"The dataset used for this tutorial is the [CIFAR10 dataset](https://www.tensorflow.org/datasets/catalog/cifar10) from [TensorFlow Datasets](https://www.tensorflow.org/datasets/catalog/overview). The version of the dataset you use in this notebook is built into TensorFlow. The trained model predicts the class of the provided input image from 10 classes namely airplane, automobile, bird, cat, deer, dog, frog, horse, ship and truck."
]
},
{
"cell_type": "markdown",
"metadata": {
@@ -190,11 +190,11 @@
"! pip3 install --upgrade google-cloud-aiplatform $USER_FLAG -q\n",
"! pip3 install {USER_FLAG} --upgrade google-cloud-storage -q\n",
"! pip3 install --upgrade tensorflow $USER_FLAG -q\n",
"! pip3 install --upgrade opencv-python-headless $USER_FLAG -q\n",
"\n",
"if os.getenv(\"IS_TESTING\"):\n",
" ! apt-get update && apt-get install -y python3-opencv-headless\n",
" ! apt-get install -y libgl1-mesa-dev\n",
" ! pip3 install --upgrade opencv-python-headless $USER_FLAG"
" ! apt-get install -y libgl1-mesa-dev"
]
},
{
@@ -256,6 +256,17 @@
"**Note**: Jupyter runs lines prefixed with `!` as shell commands, and it interpolates Python variables prefixed with `$`."
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "5aee4379e8e5"
},
"source": [
"#### Set your project ID\n",
"\n",
"**If you don't know your project ID**, you may be able to get your project ID using `gcloud`."
]
},
{
"cell_type": "code",
"execution_count": null,
@@ -321,7 +332,10 @@
},
"outputs": [],
"source": [
"REGION = \"us-central1\" # @param {type: \"string\"}"
"REGION = \"[your-region]\" # @param {type: \"string\"}\n",
"\n",
"if REGION == \"[your-region]\":\n",
" REGION = \"us-central1\""
]
},
{
@@ -330,9 +344,9 @@
"id": "timestamp"
},
"source": [
"#### Timestamp\n",
"#### UUID\n",
"\n",
"If you are in a live tutorial session, you might be using a shared test account or project. To avoid name collisions between users on resources created, you create a timestamp for each instance session, and append the timestamp onto the name of resources you create in this tutorial."
"If you are in a live tutorial session, you might be using a shared test account or project. To avoid name collisions between users on resources created, you create a uuid for each instance session, and append it onto the name of resources you create in this tutorial."
]
},
{
@@ -343,9 +357,16 @@
},
"outputs": [],
"source": [
"from datetime import datetime\n",
"import random\n",
"import string\n",
"\n",
"TIMESTAMP = datetime.now().strftime(\"%Y%m%d%H%M%S\")"
"\n",
"# Generate a uuid of length 8\n",
"def generate_uuid():\n",
" return \"\".join(random.choices(string.ascii_lowercase + string.digits, k=8))\n",
"\n",
"\n",
"UUID = generate_uuid()"
]
},
{
@@ -426,7 +447,7 @@
"\n",
"**The following steps are required, regardless of your notebook environment.**\n",
"\n",
"When you initialize the Vertex SDK for Python, you specify a Cloud Storage staging bucket. The staging bucket is where all the data associated with your dataset and model resources are retained across sessions.\n",
"When you initialize the Vertex AI SDK for Python, you specify a Cloud Storage staging bucket. The staging bucket is where all the data associated with your dataset and model resources are retained across sessions.\n",
"\n",
"Set the name of your Cloud Storage bucket below. Bucket names must be globally unique across all Google Cloud projects, including those outside of your organization."
]
@@ -439,7 +460,7 @@
},
"outputs": [],
"source": [
"BUCKET_NAME = \"[your-bucket-name]\" # @param {type:\"string\"}BUCKET_NAME = \"[your-bucket-name]\" # @param {type:\"string\"}\n",
"BUCKET_NAME = \"[your-bucket-name]\" # @param {type:\"string\"}\n",
"BUCKET_URI = f\"gs://{BUCKET_NAME}\""
]
},
@@ -452,7 +473,7 @@
"outputs": [],
"source": [
"if BUCKET_NAME == \"\" or BUCKET_NAME is None or BUCKET_NAME == \"[your-bucket-name]\":\n",
" BUCKET_NAME = PROJECT_ID + \"aip-\" + TIMESTAMP\n",
" BUCKET_NAME = PROJECT_ID + \"aip-\" + UUID\n",
" BUCKET_URI = \"gs://\" + BUCKET_NAME"
]
},
@@ -516,7 +537,7 @@
},
"outputs": [],
"source": [
"import google.cloud.aiplatform as aip"
"from google.cloud import aiplatform"
]
},
{
@@ -538,7 +559,7 @@
},
"outputs": [],
"source": [
"aip.init(project=PROJECT_ID, staging_bucket=BUCKET_URI)"
"aiplatform.init(project=PROJECT_ID, staging_bucket=BUCKET_URI)"
]
},
{
@@ -560,7 +581,7 @@
"\n",
"Learn more [here](https://cloud.google.com/vertex-ai/docs/general/locations#accelerators) hardware accelerator support for your region\n",
"\n",
"*Note*: TF releases before 2.3 for GPU support will fail to load the custom model in this tutorial. It is a known issue and fixed in TF 2.3 -- which is caused by static graph ops that are generated in the serving function. If you encounter this issue on your own custom models, use a container image for TF 2.3 with GPU support."
"*Note*: TF releases before 2.3 for GPU support fail to load the custom model in this tutorial. It is a known issue and is fixed in TF 2.3 -- which is caused by static graph ops that are generated in the serving function. If you encounter this issue on your own custom models, use a container image for TF 2.3 with GPU support."
]
},
{
@@ -573,7 +594,7 @@
"source": [
"if os.getenv(\"IS_TESTING_TRAIN_GPU\"):\n",
" TRAIN_GPU, TRAIN_NGPU = (\n",
" aip.gapic.AcceleratorType.NVIDIA_TESLA_K80,\n",
" aiplatform.gapic.AcceleratorType.NVIDIA_TESLA_K80,\n",
" int(os.getenv(\"IS_TESTING_TRAIN_GPU\")),\n",
" )\n",
"else:\n",
@@ -581,7 +602,7 @@
"\n",
"if os.getenv(\"IS_TESTING_DEPLOY_GPU\"):\n",
" DEPLOY_GPU, DEPLOY_NGPU = (\n",
" aip.gapic.AcceleratorType.NVIDIA_TESLA_K80,\n",
" aiplatform.gapic.AcceleratorType.NVIDIA_TESLA_K80,\n",
" int(os.getenv(\"IS_TESTING_DEPLOY_GPU\")),\n",
" )\n",
"else:\n",
@@ -659,7 +680,7 @@
"\n",
"Next, set the machine type to use for training and prediction.\n",
"\n",
"- Set the variables `TRAIN_COMPUTE` and `DEPLOY_COMPUTE` to configure the compute resources for the VMs you will use for for training and prediction.\n",
"- Set the variables `TRAIN_COMPUTE` and `DEPLOY_COMPUTE` to configure the compute resources for the VMs you're going to use for for training and prediction.\n",
" - `machine type`\n",
" - `n1-standard`: 3.75GB of memory per vCPU.\n",
" - `n1-highmem`: 6.5GB of memory per vCPU\n",
@@ -722,7 +743,7 @@
"\n",
"#### Package layout\n",
"\n",
"Before you start the training, you will look at how a Python package is assembled for a custom training job. When unarchived, the package contains the following directory/file layout.\n",
"Before you start the training, you look at how a Python package is assembled for a custom training job. When unarchived, the package contains the following directory/file layout.\n",
"\n",
"- PKG-INFO\n",
"- README.md\n",
@@ -738,7 +759,7 @@
"\n",
"#### Package Assembly\n",
"\n",
"In the following cells, you will assemble the training package."
"In the following cells, you create the training package."
]
},
{
@@ -952,8 +973,8 @@
},
"outputs": [],
"source": [
"job = aip.CustomTrainingJob(\n",
" display_name=\"cifar10_\" + TIMESTAMP,\n",
"job = aiplatform.CustomTrainingJob(\n",
" display_name=\"cifar10_\" + UUID,\n",
" script_path=\"custom/trainer/task.py\",\n",
" container_uri=TRAIN_IMAGE,\n",
" requirements=[\"gcsfs==0.7.1\", \"tensorflow-datasets==4.4\"],\n",
@@ -988,7 +1009,7 @@
},
"outputs": [],
"source": [
"MODEL_DIR = \"{}/{}\".format(BUCKET_NAME, TIMESTAMP)\n",
"MODEL_DIR = \"{}/{}\".format(BUCKET_URI, UUID)\n",
"\n",
"EPOCHS = 20\n",
"STEPS = 100\n",
@@ -1094,9 +1115,9 @@
"\n",
"### Load evaluation data\n",
"\n",
"You will load the CIFAR10 test (holdout) data from `tf.keras.datasets`, using the method `load_data()`. This returns the dataset as a tuple of two elements. The first element is the training data and the second is the test data. Each element is also a tuple of two elements: the image data, and the corresponding labels.\n",
"Load the CIFAR10 test (holdout) data from `tf.keras.datasets`, using the method `load_data()`. This returns the dataset as a tuple of two elements. The first element is the training data and the second is the test data. Each element is also a tuple of two elements: the image data, and the corresponding labels.\n",
"\n",
"You don't need the training data, and hence why we loaded it as `(_, _)`.\n",
"You don't need the training data, and hence why it was loaded into `(_, _)`.\n",
"\n",
"Before you can run the data through evaluation, you need to preprocess it:\n",
"\n",
@@ -1104,7 +1125,7 @@
"1. Normalize (rescale) the pixel data by dividing each pixel by 255. This replaces each single byte integer pixel with a 32-bit floating point number between 0 and 1.\n",
"\n",
"`y_test`:<br/>\n",
"2. The labels are currently scalar (sparse). If you look back at the `compile()` step in the `trainer/task.py` script, you will find that it was compiled for sparse labels. So we don't need to do anything more."
"2. The labels are currently scalar (sparse). At the `compile()` step in the `trainer/task.py` script, it can be noticed that it was compiled for sparse labels already."
]
},
{
@@ -1238,7 +1259,7 @@
"\n",
"You can get the signatures of your model's input and output layers by reloading the model into memory, and querying it for the signatures corresponding to each layer.\n",
"\n",
"When making a prediction request, you need to route the request to the serving function instead of the model, so you need to know the input layer name of the serving function -- which you will use later when you make a prediction request.\n",
"When making a prediction request, you need to route the request to the serving function instead of the model, so you need to know the input layer name of the serving function -- which you use later when you make a prediction request.\n",
"\n",
"You also need to know the name of the serving function's input and output layer for constructing the explanation metadata -- which is discussed subsequently."
]
@@ -1295,7 +1316,7 @@
"\n",
"Parameters:\n",
"\n",
"- `path_count`: This is the number of paths over the features that will be processed by the algorithm. An exact approximation of the Shapley values requires M! paths, where M is the number of features. For the CIFAR10 dataset, this would be 784 (28*28).\n",
"- `path_count`: The number of paths over the features that are processed by the algorithm. An exact approximation of the Shapley values requires M! paths, where M is the number of features. For the CIFAR10 dataset, this would be 784 (28*28).\n",
"\n",
"For any non-trival number of features, this is too compute expensive. You can reduce the number of paths over the features to M * `path_count`.\n",
"\n",
@@ -1323,7 +1344,7 @@
"\n",
"- `step_count`: This is the number of steps to approximate the remaining sum. The more steps, the more accurate the integral approximation. The general rule of thumb is 50 steps, but as you increase so does the compute time.\n",
"\n",
"In the next code cell, set the variable `XAI` to which explainabilty algorithm you will use on your custom model."
"In the next code cell, set the variable `XAI` to which explainabilty algorithm you want to use on your custom model."
]
},
{
@@ -1343,7 +1364,7 @@
"elif XAI == \"xrai\":\n",
" PARAMETERS = {\"xrai_attribution\": {\"step_count\": 50}}\n",
"\n",
"parameters = aip.explain.ExplanationParameters(PARAMETERS)"
"parameters = aiplatform.explain.ExplanationParameters(PARAMETERS)"
]
},
{
@@ -1422,10 +1443,10 @@
"\n",
"OUTPUT_METADATA = {\"output_tensor_name\": serving_output}\n",
"\n",
"input_metadata = aip.explain.ExplanationMetadata.InputMetadata(INPUT_METADATA)\n",
"output_metadata = aip.explain.ExplanationMetadata.OutputMetadata(OUTPUT_METADATA)\n",
"input_metadata = aiplatform.explain.ExplanationMetadata.InputMetadata(INPUT_METADATA)\n",
"output_metadata = aiplatform.explain.ExplanationMetadata.OutputMetadata(OUTPUT_METADATA)\n",
"\n",
"metadata = aip.explain.ExplanationMetadata(\n",
"metadata = aiplatform.explain.ExplanationMetadata(\n",
" inputs={\"image\": input_metadata}, outputs={\"class\": output_metadata}\n",
")"
]
@@ -1458,8 +1479,8 @@
},
"outputs": [],
"source": [
"model = aip.Model.upload(\n",
" display_name=\"cifar10_\" + TIMESTAMP,\n",
"model = aiplatform.Model.upload(\n",
" display_name=\"cifar10_\" + UUID,\n",
" artifact_uri=MODEL_DIR,\n",
" serving_container_image_uri=DEPLOY_IMAGE,\n",
" explanation_parameters=parameters,\n",
@@ -1478,7 +1499,7 @@
"source": [
"### Get test items\n",
"\n",
"You will use examples out of the test (holdout) portion of the dataset as a test items."
"Use examples from the test (holdout) portion of the dataset as a test items."
]
},
{
@@ -1532,7 +1553,7 @@
"source": [
"### Copy test item(s)\n",
"\n",
"For the batch prediction, you will copy the test items over to your Cloud Storage bucket."
"For the batch prediction, copy the test items over to your Cloud Storage bucket."
]
},
{
@@ -1543,11 +1564,11 @@
},
"outputs": [],
"source": [
"! gsutil cp tmp1.jpg $BUCKET_NAME/tmp1.jpg\n",
"! gsutil cp tmp2.jpg $BUCKET_NAME/tmp2.jpg\n",
"! gsutil cp tmp1.jpg $BUCKET_URI/tmp1.jpg\n",
"! gsutil cp tmp2.jpg $BUCKET_URI/tmp2.jpg\n",
"\n",
"test_item_1 = BUCKET_NAME + \"/tmp1.jpg\"\n",
"test_item_2 = BUCKET_NAME + \"/tmp2.jpg\""
"test_item_1 = BUCKET_URI + \"/tmp1.jpg\"\n",
"test_item_2 = BUCKET_URI + \"/tmp2.jpg\""
]
},
{
@@ -1558,7 +1579,7 @@
"source": [
"### Make the batch input file\n",
"\n",
"Now make a batch input file, which you will store in your local Cloud Storage bucket. The batch input file can only be in JSONL format. For JSONL file, you make one dictionary entry per line for each data item (instance). The dictionary contains the key/value pairs:\n",
"Now make a batch input file, which is then stored to your Cloud Storage bucket. The batch input file can only be in JSONL format. For JSONL file, you make one dictionary entry per line for each data item (instance). The dictionary contains the key/value pairs:\n",
"\n",
"- `input_name`: the name of the input layer of the underlying model.\n",
"- `'b64'`: A key that indicates the content is base64 encoded.\n",
@@ -1568,7 +1589,7 @@
"\n",
" {serving_input: {'b64': content}}\n",
"\n",
"To pass the image data to the prediction service you encode the bytes into base64 -- which makes the content safe from modification when transmitting binary data over the network.\n",
"To pass the image data to the prediction service you encode the bytes into base64. It makes the content safe from modification when transmitting binary data over the network.\n",
"\n",
"- `tf.io.read_file`: Read the compressed JPG images into memory as raw bytes.\n",
"- `base64.b64encode`: Encode the raw bytes into a base64 encoded string."
@@ -1585,7 +1606,7 @@
"import base64\n",
"import json\n",
"\n",
"gcs_input_uri = BUCKET_NAME + \"/\" + \"test.jsonl\"\n",
"gcs_input_uri = BUCKET_URI + \"/\" + \"test.jsonl\"\n",
"with tf.io.gfile.GFile(gcs_input_uri, \"w\") as f:\n",
" bytes = tf.io.read_file(test_item_1)\n",
" b64str = base64.b64encode(bytes.numpy()).decode(\"utf-8\")\n",
@@ -1613,7 +1634,7 @@
"- `instances_format`: The format for the input instances, either 'csv' or 'jsonl'. Defaults to 'jsonl'.\n",
"- `predictions_format`: The format for the output predictions, either 'csv' or 'jsonl'. Defaults to 'jsonl'.\n",
"- `machine_type`: The type of machine to use for training.\n",
"- `sync`: If set to True, the call will block while waiting for the asynchronous batch job to complete."
"- `sync`: Whether to execute the job synchronously. If False, the job executes in concurrent Future and any downstream object gets immediately returned and synced when the Future has completed."
]
},
{
@@ -1628,9 +1649,9 @@
"MAX_NODES = 1\n",
"\n",
"batch_predict_job = model.batch_predict(\n",
" job_display_name=\"cifar10_\" + TIMESTAMP,\n",
" job_display_name=\"cifar10_\" + UUID,\n",
" gcs_source=gcs_input_uri,\n",
" gcs_destination_prefix=BUCKET_NAME,\n",
" gcs_destination_prefix=BUCKET_URI,\n",
" instances_format=\"jsonl\",\n",
" model_parameters=None,\n",
" machine_type=DEPLOY_COMPUTE,\n",
@@ -1717,7 +1738,9 @@
"To clean up all Google Cloud resources used in this project, you can [delete the Google Cloud\n",
"project](https://cloud.google.com/resource-manager/docs/creating-managing-projects#shutting_down_projects) you used for the tutorial.\n",
"\n",
"Otherwise, you can delete the individual resources you created in this tutorial."
"Otherwise, you can delete the individual resources you created in this tutorial. \n",
"\n",
"Set `delete_bucket` to **True** to delete the Cloud Storage bucket."
]
},
{
@@ -1735,7 +1758,7 @@
"\n",
"\n",
"if delete_bucket or os.getenv(\"IS_TESTING\"):\n",
" ! gsutil rm -r $BUCKET_NAME"
" ! gsutil rm -r $BUCKET_URI"
]
}
],