xiaohei/model_garden
1
0
Fork 0
mirror of https://github.com/GoogleCloudPlatform/vertex-ai-samples.git synced 2026-09-26 14:42:04 +00:00
Code Issues Packages Projects Releases Wiki Activity

upgrade: current notebook standards (#655)

Browse Source 
* update: current standards

* update: current standards

* Update sdk_custom_image_classification_batch_explain.ipynb

* fix: bucket nit
This commit is contained in:
Andrew Ferlitsch
2022-06-22 16:59:39 -07:00
committed by GitHub
parent 9928276dc3
commit 41fc83ad03
1 changed files with 78 additions and 144 deletions
Download Patch File Download Diff File Expand all files Collapse all files
notebooks/official/explainable_ai/sdk_custom_image_classification_batch_explain.ipynb
+78 -144
View File
@@ -33,18 +33,19 @@
"\n",
"<table align=\"left\">\n",
" <td>\n",
" <a href=\"https://colab.research.google.com/github/GoogleCloudPlatform/vertex-ai-samples/tree/master/notebooks/official/automl/sdk_custom_image_classification_batch_explain.ipynb\">\n",
" <a href=\"https://colab.research.google.com/github/GoogleCloudPlatform/vertex-ai-samples/blob/main/notebooks/official/automl/sdk_custom_image_classification_batch_explain.ipynb\">\n",
" <img src=\"https://cloud.google.com/ml-engine/images/colab-logo-32px.png\" alt=\"Colab logo\"> Run in Colab\n",
" </a>\n",
" </td>\n",
" <td>\n",
" <a href=\"https://github.com/GoogleCloudPlatform/vertex-ai-samples/tree/master/notebooks/official/automl/sdk_custom_image_classification_batch_explain.ipynb\">\n",
" <a href=\"https://github.com/GoogleCloudPlatform/vertex-ai-samples/blob/main/notebooks/official/automl/sdk_custom_image_classification_batch_explain.ipynb\">\n",
" <img src=\"https://cloud.google.com/ml-engine/images/github-logo-32px.png\" alt=\"GitHub logo\">\n",
" View on GitHub\n",
" </a>\n",
" </td>\n",
" <td>\n",
" <a href=\"https://console.cloud.google.com/vertex-ai/notebooks/deploy-notebook?download_url=https://github.com/GoogleCloudPlatform/vertex-ai-samples/tree/master/notebooks/official/automl/sdk_custom_image_classification_batch_explain.ipynb\">\n",
" <a href=\"https://console.cloud.google.com/vertex-ai/workbench/deploy-notebook?download_url=https://raw.githubusercontent.com/GoogleCloudPlatform/vertex-ai-samples/tree/main/notebooks/official/explainable_ai/sdk_custom_image_classification_batch_explain.ipynb\">\n",
" <img src=\"https://lh3.googleusercontent.com/UiNooY4LUgW_oTvpsNhPpQzsstV5W8F7rYgxgGBD85cWJoLmrOzhVs_ksK_vgx40SHs7jCqkTkCk=e14-rj-sc0xffffff-h130-w32\" alt=\"Vertex AI logo\">\n",
" Open in Vertex AI Workbench\n",
" </a>\n",
" </td>\n",
@@ -61,7 +62,7 @@
"## Overview\n",
"\n",
"\n",
"This tutorial demonstrates how to use the Vertex SDK to train and deploy a custom image classification model for batch prediction with explanation."
"This tutorial demonstrates how to use the Vertex AI SDK to train and deploy a custom image classification model for batch prediction with explanation."
]
},
{
@@ -83,16 +84,21 @@
"source": [
"### Objective\n",
"\n",
"In this tutorial, you create a custom model, with a training pipeline, from a Python script in a Google prebuilt Docker container using the Vertex SDK, and then do a batch prediction with explanations on the uploaded model. You can alternatively create custom models using `gcloud` command-line tool or online using Cloud Console.\n",
"In this tutorial, you learn to use `Vertex AI Training and Explainable AI` to create a custom image classification model with explanations, and then you learn to use `Vertex AI Batch Prediction` to make a batch prediction request with explanations. You can alternatively create custom models using `gcloud` command-line tool or online using Cloud Console.\n",
"\n",
"This tutorial uses the following Google Cloud ML services:\n",
"\n",
"- `Vertex AI Training`\n",
"- `Vertex AI Batch Prediction`\n",
"- `Vertex Explainable AI`\n",
"- `Vertex AI Model` resource\n",
"\n",
"The steps performed include:\n",
"\n",
"- Create a Vertex custom job for training a model.\n",
"- Train the TensorFlow model.\n",
"- Retrieve and load the model artifacts.\n",
"- View the model evaluation.\n",
"- Set explanation parameters.\n",
"- Upload the model as a Vertex `Model` resource.\n",
"- Create a `Vertex AI` custom job for training a TensorFlow model.\n",
"- View the model evaluation for the trained model.\n",
"- Set explanation parameters for when the model is deployed.\n",
"- Upload the trained model artifacts and explanation parameters as a `Model` resource.\n",
"- Make a batch prediction with explanations."
]
},
@@ -124,7 +130,7 @@
"source": [
"### Set up your local development environment\n",
"\n",
"If you are using Colab or Google Cloud Notebook, your environment already meets all the requirements to run this notebook. You can skip this step.\n",
"If you are using Colab or Vertex AI Workbench Notebook, your environment already meets all the requirements to run this notebook. You can skip this step.\n",
"\n",
"Otherwise, make sure your environment meets this notebook's requirements. You need the following:\n",
"\n",
@@ -157,7 +163,7 @@
"source": [
"## Installation\n",
"\n",
"Install the latest version of Vertex SDK for Python."
"Install the packages required for executing this notebook."
]
},
{
@@ -170,55 +176,21 @@
"source": [
"import os\n",
"\n",
"# Google Cloud Notebook\n",
"if os.path.exists(\"/opt/deeplearning/metadata/env_version\"):\n",
"# The Vertex AI Workbench Notebook product has specific requirements\n",
"IS_WORKBENCH_NOTEBOOK = os.getenv(\"DL_ANACONDA_HOME\") and not os.getenv(\"VIRTUAL_ENV\")\n",
"IS_USER_MANAGED_WORKBENCH_NOTEBOOK = os.path.exists(\n",
" \"/opt/deeplearning/metadata/env_version\"\n",
")\n",
"\n",
"# Vertex AI Notebook requires dependencies to be installed with '--user'\n",
"USER_FLAG = \"\"\n",
"if IS_WORKBENCH_NOTEBOOK:\n",
" USER_FLAG = \"--user\"\n",
"else:\n",
" USER_FLAG = \"\"\n",
"\n",
"! pip3 install --upgrade google-cloud-aiplatform $USER_FLAG -q\n",
"! pip3 install {USER_FLAG} --upgrade google-cloud-storage -q\n",
"! pip3 install --upgrade tensorflow $USER_FLAG -q\n",
"\n",
"! pip3 install --upgrade google-cloud-aiplatform $USER_FLAG"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "install_storage"
},
"source": [
"Install the latest GA version of *google-cloud-storage* library as well."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "install_storage"
},
"outputs": [],
"source": [
"! pip3 install -U google-cloud-storage $USER_FLAG"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "install_tensorflow"
},
"outputs": [],
"source": [
"if os.getenv(\"IS_TESTING\"):\n",
" ! pip3 install --upgrade tensorflow $USER_FLAG"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "install_cv2"
},
"outputs": [],
"source": [
"if os.getenv(\"IS_TESTING\"):\n",
" ! apt-get update && apt-get install -y python3-opencv-headless\n",
" ! apt-get install -y libgl1-mesa-dev\n",
@@ -384,23 +356,31 @@
"source": [
"### Authenticate your Google Cloud account\n",
"\n",
"**If you are using Google Cloud Notebook**, your environment is already authenticated. Skip this step.\n",
"**If you are using Workbench AI Notebooks**, your environment is already\n",
"authenticated. Skip this step.\n",
"\n",
"**If you are using Colab**, run the cell below and follow the instructions when prompted to authenticate your account via oAuth.\n",
"**If you are using Colab**, run the cell below and follow the instructions\n",
"when prompted to authenticate your account via oAuth.\n",
"\n",
"**Otherwise**, follow these steps:\n",
"\n",
"In the Cloud Console, go to the [Create service account key](https://console.cloud.google.com/apis/credentials/serviceaccountkey) page.\n",
"1. In the Cloud Console, go to the [**Create service account key**\n",
" page](https://console.cloud.google.com/apis/credentials/serviceaccountkey).\n",
"\n",
"**Click Create service account**.\n",
"2. Click **Create service account**.\n",
"\n",
"In the **Service account name** field, enter a name, and click **Create**.\n",
"3. In the **Service account name** field, enter a name, and\n",
" click **Create**.\n",
"\n",
"In the **Grant this service account access to project** section, click the Role drop-down list. Type \"Vertex\" into the filter box, and select **Vertex Administrator**. Type \"Storage Object Admin\" into the filter box, and select **Storage Object Admin**.\n",
"4. In the **Grant this service account access to project** section, click the **Role** drop-down list. Type \"Vertex AI\"\n",
"into the filter box, and select\n",
" **Vertex AI Administrator**. Type \"Storage Object Admin\" into the filter box, and select **Storage Object Admin**.\n",
"\n",
"Click Create. A JSON file that contains your key downloads to your local environment.\n",
"5. Click *Create*. A JSON file that contains your key downloads to your\n",
"local environment.\n",
"\n",
"Enter the path to your service account key as the GOOGLE_APPLICATION_CREDENTIALS variable in the cell below and run the cell."
"6. Enter the path to your service account key as the\n",
"`GOOGLE_APPLICATION_CREDENTIALS` variable in the cell below and run the cell."
]
},
{
@@ -419,8 +399,11 @@
"import os\n",
"import sys\n",
"\n",
"# If on Google Cloud Notebook, then don't execute this code\n",
"if not os.path.exists(\"/opt/deeplearning/metadata/env_version\"):\n",
"# If on Vertex AI Workbench, then don't execute this code\n",
"IS_COLAB = \"google.colab\" in sys.modules\n",
"if not os.path.exists(\"/opt/deeplearning/metadata/env_version\") and not os.getenv(\n",
" \"DL_ANACONDA_HOME\"\n",
"):\n",
" if \"google.colab\" in sys.modules:\n",
" from google.colab import auth as google_auth\n",
"\n",
@@ -456,7 +439,8 @@
},
"outputs": [],
"source": [
"BUCKET_NAME = \"gs://[your-bucket-name]\" # @param {type:\"string\"}"
"BUCKET_NAME = \"[your-bucket-name]\" # @param {type:\"string\"}BUCKET_NAME = \"[your-bucket-name]\" # @param {type:\"string\"}\n",
"BUCKET_URI = f\"gs://{BUCKET_NAME}\""
]
},
{
@@ -467,8 +451,9 @@
},
"outputs": [],
"source": [
"if BUCKET_NAME == \"\" or BUCKET_NAME is None or BUCKET_NAME == \"gs://[your-bucket-name]\":\n",
" BUCKET_NAME = \"gs://\" + PROJECT_ID + \"aip-\" + TIMESTAMP"
"if BUCKET_NAME == \"\" or BUCKET_NAME is None or BUCKET_NAME == \"[your-bucket-name]\":\n",
" BUCKET_NAME = PROJECT_ID + \"aip-\" + TIMESTAMP\n",
" BUCKET_URI = \"gs://\" + BUCKET_NAME"
]
},
{
@@ -488,7 +473,7 @@
},
"outputs": [],
"source": [
"! gsutil mb -l $REGION $BUCKET_NAME"
"! gsutil mb -l $REGION $BUCKET_URI"
]
},
{
@@ -508,7 +493,7 @@
},
"outputs": [],
"source": [
"! gsutil ls -al $BUCKET_NAME"
"! gsutil ls -al $BUCKET_URI"
]
},
{
@@ -540,9 +525,9 @@
"id": "init_aip:mbsdk"
},
"source": [
"## Initialize Vertex SDK for Python\n",
"### Initialize Vertex AI SDK for Python\n",
"\n",
"Initialize the Vertex SDK for Python for your project and corresponding bucket."
"Initialize the Vertex AI SDK for Python for your project and corresponding bucket."
]
},
{
@@ -553,7 +538,7 @@
},
"outputs": [],
"source": [
"aip.init(project=PROJECT_ID, staging_bucket=BUCKET_NAME)"
"aip.init(project=PROJECT_ID, staging_bucket=BUCKET_URI)"
]
},
{
@@ -568,7 +553,7 @@
"\n",
"Set the variables `TRAIN_GPU/TRAIN_NGPU` and `DEPLOY_GPU/DEPLOY_NGPU` to use a container image supporting a GPU and the number of GPUs allocated to the virtual machine (VM) instance. For example, to use a GPU container image with 4 Nvidia Telsa K80 GPUs allocated to each VM, you would specify:\n",
"\n",
" (aip.AcceleratorType.NVIDIA_TESLA_K80, 4)\n",
" (aip.gapic.AcceleratorType.NVIDIA_TESLA_K80, 4)\n",
"\n",
"\n",
"Otherwise specify `(None, None)` to use a container image to run on a CPU.\n",
@@ -631,7 +616,7 @@
"if os.getenv(\"IS_TESTING_TF\"):\n",
" TF = os.getenv(\"IS_TESTING_TF\")\n",
"else:\n",
" TF = \"2-1\"\n",
" TF = \"2-5\"\n",
"\n",
"if TF[0] == \"2\":\n",
" if TRAIN_GPU:\n",
@@ -652,8 +637,13 @@
" else:\n",
" DEPLOY_VERSION = \"tf-cpu.{}\".format(TF)\n",
"\n",
"TRAIN_IMAGE = \"gcr.io/cloud-aiplatform/training/{}:latest\".format(TRAIN_VERSION)\n",
"DEPLOY_IMAGE = \"gcr.io/cloud-aiplatform/prediction/{}:latest\".format(DEPLOY_VERSION)\n",
"\n",
"TRAIN_IMAGE = \"{}-docker.pkg.dev/vertex-ai/training/{}:latest\".format(\n",
" REGION.split(\"-\")[0], TRAIN_VERSION\n",
")\n",
"DEPLOY_IMAGE = \"{}-docker.pkg.dev/vertex-ai/prediction/{}:latest\".format(\n",
" REGION.split(\"-\")[0], DEPLOY_VERSION\n",
")\n",
"\n",
"print(\"Training:\", TRAIN_IMAGE, TRAIN_GPU, TRAIN_NGPU)\n",
"print(\"Deployment:\", DEPLOY_IMAGE, DEPLOY_GPU, DEPLOY_NGPU)"
@@ -1727,17 +1717,7 @@
"To clean up all Google Cloud resources used in this project, you can [delete the Google Cloud\n",
"project](https://cloud.google.com/resource-manager/docs/creating-managing-projects#shutting_down_projects) you used for the tutorial.\n",
"\n",
"Otherwise, you can delete the individual resources you created in this tutorial:\n",
"\n",
"- Dataset\n",
"- Pipeline\n",
"- Model\n",
"- Endpoint\n",
"- AutoML Training Job\n",
"- Batch Job\n",
"- Custom Job\n",
"- Hyperparameter Tuning Job\n",
"- Cloud Storage Bucket"
"Otherwise, you can delete the individual resources you created in this tutorial."
]
},
{
@@ -1748,60 +1728,14 @@
},
"outputs": [],
"source": [
"delete_all = True\n",
"delete_bucket = False\n",
"\n",
"if delete_all:\n",
" # Delete the dataset using the Vertex dataset object\n",
" try:\n",
" if \"dataset\" in globals():\n",
" dataset.delete()\n",
" except Exception as e:\n",
" print(e)\n",
"model.delete()\n",
"batch_predict_job.delete()\n",
"\n",
" # Delete the model using the Vertex model object\n",
" try:\n",
" if \"model\" in globals():\n",
" model.delete()\n",
" except Exception as e:\n",
" print(e)\n",
"\n",
" # Delete the endpoint using the Vertex endpoint object\n",
" try:\n",
" if \"endpoint\" in globals():\n",
" endpoint.delete()\n",
" except Exception as e:\n",
" print(e)\n",
"\n",
" # Delete the AutoML or Pipeline trainig job\n",
" try:\n",
" if \"dag\" in globals():\n",
" dag.delete()\n",
" except Exception as e:\n",
" print(e)\n",
"\n",
" # Delete the custom trainig job\n",
" try:\n",
" if \"job\" in globals():\n",
" job.delete()\n",
" except Exception as e:\n",
" print(e)\n",
"\n",
" # Delete the batch prediction job using the Vertex batch prediction object\n",
" try:\n",
" if \"batch_predict_job\" in globals():\n",
" batch_predict_job.delete()\n",
" except Exception as e:\n",
" print(e)\n",
"\n",
" # Delete the hyperparameter tuning job using the Vertex hyperparameter tuning object\n",
" try:\n",
" if \"hpt_job\" in globals():\n",
" hpt_job.delete()\n",
" except Exception as e:\n",
" print(e)\n",
"\n",
" if \"BUCKET_NAME\" in globals():\n",
" ! gsutil rm -r $BUCKET_NAME"
"if delete_bucket or os.getenv(\"IS_TESTING\"):\n",
" ! gsutil rm -r $BUCKET_NAME"
]
}
],