xiaohei/model_garden
1
0
Fork 0
mirror of https://github.com/GoogleCloudPlatform/vertex-ai-samples.git synced 2026-09-26 14:42:04 +00:00
Code Issues Packages Projects Releases Wiki Activity

fix, refactor, chore(egen): follows new template, adds dataflow enable step, refactor (#3089)

Browse Source 
* Updated template of notebook

* fix, refactor, chore: follows new template, adds dataflow enable step, cleanup steps for files, removes IS_TESTING in cleanup, REGION==>LOCATION

* chore: addresses the review comments

---------

Co-authored-by: nileshspringml <nilesh.mahajan@egen.ai>
Co-authored-by: krishr2d2 <krishna.movva@egen.ai>
This commit is contained in:
Krishna Chaithanya Movva
2024-06-13 20:36:50 +00:00
committed by GitHub
co-authored by nileshspringml krishr2d2
parent 847d38642c
commit 7e8bf8a126
1 changed files with 152 additions and 167 deletions
Download Patch File Download Diff File Expand all files Collapse all files
notebooks/official/model_evaluation/custom_tabular_classification_model_evaluation.ipynb
+152 -167
View File
@@ -32,24 +32,26 @@
"# Vertex AI Pipelines: Evaluating BatchPrediction results from a custom tabular classification model\n",
"\n",
"<table align=\"left\">\n",
"\n",
" <td>\n",
" <td style=\"text-align: center\">\n",
" <a href=\"https://colab.research.google.com/github/GoogleCloudPlatform/vertex-ai-samples/blob/main/notebooks/official/model_evaluation/custom_tabular_classification_model_evaluation.ipynb\">\n",
" <img src=\"https://cloud.google.com/ml-engine/images/colab-logo-32px.png\" alt=\"Colab logo\"> Run in Colab\n",
" <img src=\"https://cloud.google.com/ml-engine/images/colab-logo-32px.png\" alt=\"Google Colaboratory logo\"><br> Open in Colab\n",
" </a>\n",
" </td>\n",
" <td>\n",
" <td style=\"text-align: center\">\n",
" <a href=\"https://console.cloud.google.com/vertex-ai/colab/import/https:%2F%2Fraw.githubusercontent.com%2FGoogleCloudPlatform%2Fvertex-ai-samples%2Fmain%2Fnotebooks%2Fofficial%2Fmodel_evaluation%2Fcustom_tabular_classification_model_evaluation.ipynb\">\n",
" <img width=\"32px\" src=\"https://cloud.google.com/ml-engine/images/colab-enterprise-logo-32px.png\" alt=\"Google Cloud Colab Enterprise logo\"><br> Open in Colab Enterprise\n",
" </a>\n",
" </td> \n",
" <td style=\"text-align: center\">\n",
" <a href=\"https://console.cloud.google.com/vertex-ai/workbench/deploy-notebook?download_url=https://raw.githubusercontent.com/GoogleCloudPlatform/vertex-ai-samples/main/notebooks/official/model_evaluation/custom_tabular_classification_model_evaluation.ipynb\">\n",
" <img src=\"https://lh3.googleusercontent.com/UiNooY4LUgW_oTvpsNhPpQzsstV5W8F7rYgxgGBD85cWJoLmrOzhVs_ksK_vgx40SHs7jCqkTkCk=e14-rj-sc0xffffff-h130-w32\" alt=\"Vertex AI logo\"><br> Open in Workbench\n",
" </a>\n",
" </td>\n",
" <td style=\"text-align: center\">\n",
" <a href=\"https://github.com/GoogleCloudPlatform/vertex-ai-samples/blob/main/notebooks/official/model_evaluation/custom_tabular_classification_model_evaluation.ipynb\">\n",
" <img src=\"https://cloud.google.com/ml-engine/images/github-logo-32px.png\" alt=\"GitHub logo\">\n",
" View on GitHub\n",
" <img src=\"https://cloud.google.com/ml-engine/images/github-logo-32px.png\" alt=\"GitHub logo\"><br> View on GitHub\n",
" </a>\n",
" </td>\n",
" <td>\n",
"<a href=\"https://console.cloud.google.com/vertex-ai/workbench/deploy-notebook?download_url=https://raw.githubusercontent.com/GoogleCloudPlatform/vertex-ai-samples/main/notebooks/official/model_evaluation/custom_tabular_classification_model_evaluation.ipynb\" target='_blank'>\n",
" <img src=\"https://lh3.googleusercontent.com/UiNooY4LUgW_oTvpsNhPpQzsstV5W8F7rYgxgGBD85cWJoLmrOzhVs_ksK_vgx40SHs7jCqkTkCk=e14-rj-sc0xffffff-h130-w32\" alt=\"Vertex AI logo\">\n",
" Open in Vertex AI Workbench\n",
" </a>\n",
" </td> \n",
"</table>"
]
},
@@ -63,7 +65,7 @@
"\n",
"This notebook demonstrates how to use the Vertex AI classification model evaluation component to evaluate a custom-trained tabular classification model saved in Vertex AI Model Registry. Model evaluation helps you determine your model performance based on the evaluation metrics and improve the model if necessary. \n",
"\n",
"Learn more about [Vertex AI Model Evaluation](https://cloud.google.com/vertex-ai/docs/evaluation/introduction) and [Vertex AI Training](https://cloud.google.com/vertex-ai/docs/training/custom-training)."
"Learn more about [Vertex AI custom training](https://cloud.google.com/vertex-ai/docs/training/custom-training) and [Vertex AI model evaluation](https://cloud.google.com/vertex-ai/docs/evaluation/introduction)."
]
},
{
@@ -74,9 +76,9 @@
"source": [
"### Objective\n",
"\n",
"In this tutorial, you train a scikit-learn RandomForest model, save it in Vertex AI Model Registry and learn how to evaluate it through a Vertex AI pipeline job using `google_cloud_pipeline_components`.\n",
"In this tutorial, you train a scikit-learn RandomForest model, save the model in Vertex AI Model Registry and learn how to evaluate the model through a Vertex AI pipeline job using Google Cloud Pipeline Components Python SDK.\n",
"\n",
"This tutorial uses the following Google Cloud ML services and resources:\n",
"This tutorial uses the following Vertex AI services and resources:\n",
"\n",
"- Vertex AI Model Registry\n",
"- Vertex AI Pipelines\n",
@@ -134,19 +136,26 @@
{
"cell_type": "markdown",
"metadata": {
"id": "i7EUnXsZhAGF"
"id": "61RBz8LLbxCR"
},
"source": [
"## Installation\n",
"\n",
"Install the following packages required to execute this notebook. "
"## Get started"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "No17Cw5hgx12"
},
"source": [
"### Install Vertex AI SDK for Python and other required packages\n"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "2b4ef9b72d43"
"id": "tFy3H3aPgx12"
},
"outputs": [],
"source": [
@@ -166,168 +175,91 @@
{
"cell_type": "markdown",
"metadata": {
"id": "hhq5zEbGg0XX"
"id": "R5Xep4W9lq-Z"
},
"source": [
"### Colab only: Uncomment the following cell to restart the kernel."
"### Restart runtime (Colab only)\n",
"\n",
"To use the newly installed packages, you must restart the runtime on Google Colab."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "EzrelQZ22IZj"
"id": "XRvKdaPDTznN"
},
"outputs": [],
"source": [
"# Automatically restart kernel after installs so that your environment can access the new packages\n",
"# import IPython\n",
"import sys\n",
"\n",
"# app = IPython.Application.instance()\n",
"# app.kernel.do_shutdown(True)"
"if \"google.colab\" in sys.modules:\n",
"\n",
" import IPython\n",
"\n",
" app = IPython.Application.instance()\n",
" app.kernel.do_shutdown(True)"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "BF1j6f9HApxa"
"id": "SbmM4z7FOBpM"
},
"source": [
"## Before you begin\n",
"\n",
"### Set up your Google Cloud project\n",
"\n",
"**The following steps are required, regardless of your notebook environment.**\n",
"\n",
"1. [Select or create a Google Cloud project](https://console.cloud.google.com/cloud-resource-manager). When you first create an account, you get a $300 free credit towards your compute/storage costs.\n",
"\n",
"1. [Make sure that billing is enabled for your project](https://cloud.google.com/billing/docs/how-to/modify-project).\n",
"\n",
"1. [Enable the Vertex AI, Compute Engine, Artifact Registry, Cloud Build and Dataflow APIs](https://console.cloud.google.com/flows/enableapi?apiid=aiplatform.googleapis.com,compute.googleapis.com,artifactregistry.googleapis.com,cloudbuild.googleapis.com,dataflow.googleapis.com).\n",
"\n",
"1. If you are running this notebook locally, you need to install the [Cloud SDK](https://cloud.google.com/sdk)."
"<div class=\"alert alert-block alert-warning\">\n",
"<b>⚠️ The kernel is going to restart. Wait until it's finished before continuing to the next step. ⚠️</b>\n",
"</div>\n"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "WReHDGG5g0XY"
"id": "dmWOrTJ3gx13"
},
"source": [
"#### Set your project ID\n",
"### Authenticate your notebook environment (Colab only)\n",
"\n",
"**If you don't know your project ID**, try the following:\n",
"* Run `gcloud config list`.\n",
"* Run `gcloud projects list`.\n",
"* See the support page: [Locate the project ID](https://support.google.com/googleapi/answer/7014113)"
"Authenticate your environment on Google Colab.\n"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "oM1iC_MfAts1"
"id": "NyKGtVQjgx13"
},
"outputs": [],
"source": [
"import sys\n",
"\n",
"if \"google.colab\" in sys.modules:\n",
"\n",
" from google.colab import auth\n",
"\n",
" auth.authenticate_user()"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "DF4l8DTdWgPY"
},
"source": [
"### Set Google Cloud project information\n",
"Learn more about [setting up a project and a development environment](https://cloud.google.com/vertex-ai/docs/start/cloud-environment)."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "Nqwi-5ufWp_B"
},
"outputs": [],
"source": [
"PROJECT_ID = \"[your-project-id]\" # @param {type:\"string\"}\n",
"\n",
"# Set the project id\n",
"! gcloud config set project {PROJECT_ID}"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "region"
},
"source": [
"#### Region\n",
"\n",
"You can also change the `REGION` variable used by Vertex AI. Learn more about [Vertex AI regions](https://cloud.google.com/vertex-ai/docs/general/locations)."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "sduDOFQVF6kv"
},
"outputs": [],
"source": [
"REGION = \"us-central1\" # @param {type: \"string\"}"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "sBCra4QMA2wR"
},
"source": [
"### Authenticate your Google Cloud account\n",
"\n",
"Depending on your Jupyter environment, you may have to manually authenticate. Follow the relevant instructions below."
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "af349043f23b"
},
"source": [
"**1. Vertex AI Workbench**\n",
"* Do nothing as you are already authenticated."
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "ad1138a125ea"
},
"source": [
"**2. Local JupyterLab instance, uncomment and run:**"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "ce6043da7b33"
},
"outputs": [],
"source": [
"# ! gcloud auth login"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "0367eac06a10"
},
"source": [
"**3. Colab, uncomment and run:**"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "21ad4dbb4a61"
},
"outputs": [],
"source": [
"# from google.colab import auth\n",
"# auth.authenticate_user()"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "c13224697bfb"
},
"source": [
"**4. Service account or other**\n",
"* See how to grant Cloud Storage permissions to your service account at https://cloud.google.com/storage/docs/gsutil/commands/iam#ch-examples."
"LOCATION = \"us-central1\" # @param {type:\"string\"}"
]
},
{
@@ -349,7 +281,7 @@
},
"outputs": [],
"source": [
"BUCKET_URI = \"gs://your-bucket-name-unique\" # @param {type:\"string\"}"
"BUCKET_URI = f\"gs://your-bucket-name-{PROJECT_ID}-unique\" # @param {type:\"string\"}"
]
},
{
@@ -358,7 +290,7 @@
"id": "-EcIXiGsCePi"
},
"source": [
"**Only if your bucket doesn't already exist**: Run the following cell to create your Cloud Storage bucket."
"**If your bucket doesn't already exist**: Run the following cell to create your Cloud Storage bucket."
]
},
{
@@ -369,7 +301,32 @@
},
"outputs": [],
"source": [
"! gsutil mb -l $REGION -p $PROJECT_ID $BUCKET_URI"
"! gsutil mb -l $LOCATION -p $PROJECT_ID $BUCKET_URI"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "init_aip:mbsdk,all"
},
"source": [
"### Initialize Vertex AI SDK for Python\n",
"\n",
"To get started using Vertex AI, you must have an existing Google Cloud project and enable the [Vertex AI API](https://console.cloud.google.com/flows/enableapi?apiid=aiplatform.googleapis.com). \n",
"Initialize the Vertex AI SDK for Python with your project and the created bucket."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "ksAefQcCF6ky"
},
"outputs": [],
"source": [
"from google.cloud import aiplatform\n",
"\n",
"aiplatform.init(project=PROJECT_ID, location=LOCATION, staging_bucket=BUCKET_URI)"
]
},
{
@@ -378,9 +335,9 @@
"id": "set_service_account"
},
"source": [
"#### Service Account\n",
"### Service Account\n",
"\n",
"You use a service account to create Vertex AI Pipeline jobs. If you do not want to use your project's Compute Engine service account, set `SERVICE_ACCOUNT` to another service account ID."
"You use a service account to create Vertex AI Pipeline jobs. If you don't want to use your project's Compute Engine service account, set `SERVICE_ACCOUNT` to another service account ID."
]
},
{
@@ -469,7 +426,7 @@
"import joblib\n",
"import kfp\n",
"import matplotlib.pyplot as plt\n",
"from google.cloud import aiplatform, aiplatform_v1, bigquery\n",
"from google.cloud import aiplatform_v1, bigquery\n",
"from kfp.v2 import compiler\n",
"from sklearn.ensemble import RandomForestClassifier\n",
"from sklearn.feature_selection import SelectKBest\n",
@@ -484,9 +441,9 @@
"id": "init_aip:mbsdk,all"
},
"source": [
"### Initialize Vertex AI and BigQuery SDK for Python\n",
"### Initialize BigQuery SDK for Python\n",
"\n",
"Initialize the Vertex AI and BigQuery SDK for Python with your project and the created bucket."
"Initialize the BigQuery SDK for Python with your project and the created bucket."
]
},
{
@@ -497,8 +454,6 @@
},
"outputs": [],
"source": [
"aiplatform.init(project=PROJECT_ID, location=REGION, staging_bucket=BUCKET_URI)\n",
"\n",
"bq_client = bigquery.Client(\n",
" project=PROJECT_ID,\n",
" credentials=aiplatform.initializer.global_config.credentials,\n",
@@ -1098,7 +1053,7 @@
"source": [
"!gcloud artifacts repositories create {REPOSITORY} \\\n",
" --repository-format=docker \\\n",
" --location=$REGION"
" --location=$LOCATION"
]
},
{
@@ -1120,7 +1075,7 @@
"outputs": [],
"source": [
"%cd $SRC_DIR/\n",
"!gcloud builds submit --region={REGION} --tag={REGION}-docker.pkg.dev/{PROJECT_ID}/{REPOSITORY}/{IMAGE} --suppress-logs\n",
"!gcloud builds submit --region={LOCATION} --tag={LOCATION}-docker.pkg.dev/{PROJECT_ID}/{REPOSITORY}/{IMAGE} --suppress-logs\n",
"%cd .."
]
},
@@ -1154,7 +1109,7 @@
"aip_model = aiplatform.Model.upload(\n",
" display_name=MODEL_DISPLAY_NAME,\n",
" artifact_uri=f\"{BUCKET_URI}/{MODEL_ARTIFACT_DIR}\",\n",
" serving_container_image_uri=f\"{REGION}-docker.pkg.dev/{PROJECT_ID}/{REPOSITORY}/{IMAGE}\",\n",
" serving_container_image_uri=f\"{LOCATION}-docker.pkg.dev/{PROJECT_ID}/{REPOSITORY}/{IMAGE}\",\n",
" serving_container_predict_route=\"/predict\",\n",
" serving_container_health_route=\"/health\",\n",
")"
@@ -1163,7 +1118,7 @@
{
"cell_type": "markdown",
"metadata": {
"id": "77f4838ece15"
"id": "7f463f9183ef"
},
"source": [
"## Create and run the evaluation pipeline\n",
@@ -1174,9 +1129,37 @@
"1. Removes the target field from the sampled test data.\n",
"1. Runs the batch prediction job.\n",
"1. Evaluates results from the batch prediction job using the ground-truth/target information.\n",
"1. Imports the generated evaluation metrics to the Vertex AI model.\n",
"\n",
"1. Imports the generated evaluation metrics to the Vertex AI model."
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "9f72ff491712"
},
"source": [
"### Enable the Dataflow API\n",
"\n",
"The model evaluation components from the Google Cloud Pipeline Components create dataflow jobs internally to perform the underlying tasks. It's essential to enable the Dataflow API before running the evaluation pipeline."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "795fdfef5736"
},
"outputs": [],
"source": [
"!gcloud services enable dataflow.googleapis.com"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "77f4838ece15"
},
"source": [
"### Define the pipeline\n",
"To define the Vertex AI Pipeline for evaluating your model, you use the `google-cloud-pipeline-components` Python package. Google Cloud Pipeline Components provides an SDK with a set of pipeline components for users to interact with Google Cloud services such as Vertex AI, Dataflow and BigQuery. \n",
"\n",
@@ -1378,7 +1361,7 @@
"Before running your pipeline, set the following parameters :\n",
"\n",
"- `project`: Project ID of the Google Cloud project.\n",
"- `location`: Region where the pipeline needs to be run. If not set, the pipeline defaults to the region that Vertex AI SDK is configured with.\n",
"- `location`: Location where the pipeline needs to be run. If not set, the pipeline defaults to the location that Vertex AI SDK is configured with.\n",
"- `root_dir`: The Cloud Storage directory for keeping the staged files and artifacts. A random subdirectory is created under the directory to keep the job information for resuming the job in case of a failure.\n",
"- `model_name`: Resource name of the trained custom tabular classification model.\n",
"- `target_field_name`: Name of the column to be used as the ground truth for evaluation.\n",
@@ -1400,7 +1383,7 @@
"source": [
"parameters = {\n",
" \"project\": PROJECT_ID,\n",
" \"location\": REGION,\n",
" \"location\": LOCATION,\n",
" \"root_dir\": PIPELINE_ROOT,\n",
" \"model_name\": aip_model.resource_name,\n",
" \"target_field_name\": TARGET,\n",
@@ -1554,8 +1537,6 @@
},
"outputs": [],
"source": [
"import os\n",
"\n",
"# Delete model resource\n",
"aip_model.delete()\n",
"\n",
@@ -1570,8 +1551,12 @@
"\n",
"delete_bucket = False\n",
"# Delete Cloud Storage objects\n",
"if delete_bucket or os.getenv(\"IS_TESTING\"):\n",
" ! gsutil -m rm -r $BUCKET_URI"
"if delete_bucket:\n",
" ! gsutil -m rm -r $BUCKET_URI\n",
"\n",
"# Delete the locally generated files and folders\n",
"! rm $PIPELINE_PACKAGE_PATH\n",
"! rm -rf $SRC_DIR"
]
}
],