xiaohei/model_garden
1
0
Fork 0
mirror of https://github.com/GoogleCloudPlatform/vertex-ai-samples.git synced 2026-09-26 14:42:04 +00:00
Code Issues Packages Projects Releases Wiki Activity

chore, refactor(egen): follows new template, removes IS_TESTING, spell corrections (#3148)

Browse Source 
* chore, refactor: follows new template, removes IS_TESTING in clean up, contracts words and some steps, spell correct, K80 to T4

* chore: addresses review comments

---------

Co-authored-by: krishr2d2 <krishna.movva@egen.ai>
This commit is contained in:
Krishna Chaithanya Movva
2024-06-26 22:34:35 +00:00
committed by GitHub
co-authored by krishr2d2
parent 01317fda2a
commit d43c5fee3d
1 changed files with 199 additions and 230 deletions
Download Patch File Download Diff File Expand all files Collapse all files
notebooks/official/model_monitoring/get_started_with_model_monitoring_custom.ipynb
+199 -230
View File
@@ -34,21 +34,24 @@
"# Vertex AI Model Monitoring for custom tabular models\n",
"\n",
"<table align=\"left\">\n",
" <td>\n",
" <td style=\"text-align: center\">\n",
" <a href=\"https://colab.research.google.com/github/GoogleCloudPlatform/vertex-ai-samples/blob/main/notebooks/official/model_monitoring/get_started_with_model_monitoring_custom.ipynb\">\n",
" <img src=\"https://cloud.google.com/ml-engine/images/colab-logo-32px.png\" alt=\"Colab logo\"> Run in Colab\n",
" <img src=\"https://cloud.google.com/ml-engine/images/colab-logo-32px.png\" alt=\"Google Colaboratory logo\"><br> Open in Colab\n",
" </a>\n",
" </td>\n",
" <td>\n",
" <a href=\"https://github.com/GoogleCloudPlatform/vertex-ai-samples/blob/main/notebooks/official/model_monitoring/get_started_with_model_monitoring_custom.ipynb\">\n",
" <img src=\"https://cloud.google.com/ml-engine/images/github-logo-32px.png\" alt=\"GitHub logo\">\n",
" View on GitHub\n",
" <td style=\"text-align: center\">\n",
" <a href=\"https://console.cloud.google.com/vertex-ai/colab/import/https:%2F%2Fraw.githubusercontent.com%2FGoogleCloudPlatform%2Fvertex-ai-samples%2Fmain%2Fnotebooks%2Fofficial%2Fmodel_monitoring%2Fget_started_with_model_monitoring_custom.ipynb\">\n",
" <img width=\"32px\" src=\"https://cloud.google.com/ml-engine/images/colab-enterprise-logo-32px.png\" alt=\"Google Cloud Colab Enterprise logo\"><br> Open in Colab Enterprise\n",
" </a>\n",
" </td> \n",
" <td>\n",
" </td> \n",
" <td style=\"text-align: center\">\n",
" <a href=\"https://console.cloud.google.com/vertex-ai/workbench/deploy-notebook?download_url=https://raw.githubusercontent.com/GoogleCloudPlatform/vertex-ai-samples/main/notebooks/official/model_monitoring/get_started_with_model_monitoring_custom.ipynb\">\n",
" <img src=\"https://lh3.googleusercontent.com/UiNooY4LUgW_oTvpsNhPpQzsstV5W8F7rYgxgGBD85cWJoLmrOzhVs_ksK_vgx40SHs7jCqkTkCk=e14-rj-sc0xffffff-h130-w32\" alt=\"Vertex AI logo\">\n",
" Open in Vertex AI Workbench\n",
" <img src=\"https://lh3.googleusercontent.com/UiNooY4LUgW_oTvpsNhPpQzsstV5W8F7rYgxgGBD85cWJoLmrOzhVs_ksK_vgx40SHs7jCqkTkCk=e14-rj-sc0xffffff-h130-w32\" alt=\"Vertex AI logo\"><br> Open in Workbench\n",
" </a>\n",
" </td>\n",
" <td style=\"text-align: center\">\n",
" <a href=\"https://github.com/GoogleCloudPlatform/vertex-ai-samples/blob/main/notebooks/official/model_monitoring/get_started_with_model_monitoring_custom.ipynb\">\n",
" <img src=\"https://cloud.google.com/ml-engine/images/github-logo-32px.png\" alt=\"GitHub logo\"><br> View on GitHub\n",
" </a>\n",
" </td>\n",
"</table>"
@@ -75,27 +78,27 @@
"source": [
"### Objective\n",
"\n",
"In this notebook, you learn to use the `Vertex AI Model Monitoring` service to detect feature skew and drift in the input predict requests, for custom tabular models.\n",
"In this notebook, you learn to use the Vertex AI Model Monitoring service to detect feature skewness and drift in the input predict requests, for custom tabular models.\n",
"\n",
"This tutorial uses the following Google Cloud ML services:\n",
"This tutorial uses the following Vertex AI services:\n",
"\n",
"- `Vertex AI Model Monitoring`\n",
"- `Vertex AI Prediction`\n",
"- `Vertex AI Model` resource\n",
"- `Vertex AI Endpoint` resource\n",
"- Vertex AI Model Monitoring\n",
"- Vertex AI Prediction\n",
"- Vertex AI Model resource\n",
"- Vertex AI Endpoint resource\n",
"\n",
"The steps performed include:\n",
"\n",
"- Download a pre-trained custom tabular model.\n",
"- Upload the pre-trained model as a `Model` resource.\n",
"- Deploy the `Model` resource to the `Endpoint` resource.\n",
"- Configure the `Endpoint` resource for model monitoring.\n",
"- Generate synthetic prediction requests for skew.\n",
"- Wait for email alert notification.\n",
"- Generate synthetic prediction requests for drift.\n",
"- Wait for email alert notification.\n",
"- Upload the pre-trained model to Vertex AI Model Registry.\n",
"- Deploy the model resource to a Vertex AI endpoint resource.\n",
"- Configure the endpoint resource for model monitoring.\n",
"- Generate synthetic prediction requests to simulate skewness.\n",
"- Wait for email alert notifications.\n",
"- Generate synthetic prediction requests to simulate drift.\n",
"- Wait for email alert notifications.\n",
"\n",
"Learn more about [Introduction to Vertex AI Model Monitoring](https://cloud.google.com/vertex-ai/docs/model-monitoring/overview)."
"Learn more about [Vertex AI Model Monitoring](https://cloud.google.com/vertex-ai/docs/model-monitoring/overview)."
]
},
{
@@ -108,12 +111,12 @@
"\n",
"This tutorial uses a pre-trained model, where the model artifacts are stored in a public Cloud Storage bucket. \n",
"\n",
"The model is based on [the blog post](https://cloud.google.com/blog/topics/developers-practitioners/churn-prediction-game-developers-using-google-analytics-4-ga4-and-bigquery-ml). The idea behind this model is that your company has extensive log data describing how your game users have interacted with the site. The raw data contains the following categories of information:\n",
"The model is based on a [blog post about Churn prediction model](https://cloud.google.com/blog/topics/developers-practitioners/churn-prediction-game-developers-using-google-analytics-4-ga4-and-bigquery-ml). This model involves extensive log data describing how game users have interacted with a site. The raw data contains the following categories of information:\n",
"\n",
"- identity - unique player identitity numbers\n",
"- demographic features - information about the player, such as the geographic region in which a player is located\n",
"- behavioral features - counts of the number of times a player has triggered certain game events, such as reaching a new level\n",
"- churn propensity - this is the label or target feature, it provides an estimated probability that this player may churn, i.e. stop being an active player."
"- identity - unique player identitity numbers.\n",
"- demographic features - information about the player, such as the geographic region in which a player is located.\n",
"- behavioral features - counts of the number of times a player has triggered certain game events, such as reaching a new level.\n",
"- churn propensity - this is the label or target feature. It provides an estimated probability that this player may churn, i.e., stop being an active player."
]
},
{
@@ -137,15 +140,22 @@
"to generate a cost estimate based on your projected usage."
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "4b331e2fd155"
},
"source": [
"## Get Started"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "f3848df1e5b0"
},
"source": [
"## Installation\n",
"\n",
"Install the packages required for executing this notebook."
"### Install Vertex AI SDK for Python and other required packages\n"
]
},
{
@@ -167,7 +177,9 @@
"id": "restart"
},
"source": [
"### Colab only: Uncomment the following cell to restart the kernel"
"### Restart runtime (Colab only)\n",
"\n",
"To use the newly installed packages, you must restart the runtime on Google Colab."
]
},
{
@@ -178,11 +190,53 @@
},
"outputs": [],
"source": [
"# Automatically restart kernel after installs so that your environment can access the new packages\n",
"# import IPython\n",
"import sys\n",
"\n",
"# app = IPython.Application.instance()\n",
"# app.kernel.do_shutdown(True)"
"if \"google.colab\" in sys.modules:\n",
"\n",
" import IPython\n",
"\n",
" app = IPython.Application.instance()\n",
" app.kernel.do_shutdown(True)"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "ee775571c2b5"
},
"source": [
"<div class=\"alert alert-block alert-warning\">\n",
"<b>⚠️ The kernel is going to restart. Wait until it's finished before continuing to the next step. ⚠️</b>\n",
"</div>\n"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "92e68cfc3a90"
},
"source": [
"### Authenticate your notebook environment (Colab only)\n",
"\n",
"Authenticate your environment on Google Colab.\n"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "46604f70e831"
},
"outputs": [],
"source": [
"import sys\n",
"\n",
"if \"google.colab\" in sys.modules:\n",
"\n",
" from google.colab import auth\n",
"\n",
" auth.authenticate_user()"
]
},
{
@@ -191,14 +245,9 @@
"id": "before_you_begin:nogpu"
},
"source": [
"## Before you begin\n",
"### Set Google Cloud project information \n",
"\n",
"### Set your project ID\n",
"\n",
"**If you don't know your project ID**, try the following:\n",
"* Run `gcloud config list`.\n",
"* Run `gcloud projects list`.\n",
"* See the support page: [Locate the project ID](https://support.google.com/googleapi/answer/7014113)"
"Learn more about [setting up a project and a development environment](https://cloud.google.com/vertex-ai/docs/start/cloud-environment)."
]
},
{
@@ -210,31 +259,7 @@
"outputs": [],
"source": [
"PROJECT_ID = \"[your-project-id]\" # @param {type:\"string\"}\n",
"\n",
"# Set the project id\n",
"! gcloud config set project {PROJECT_ID}"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "region"
},
"source": [
"#### Region\n",
"\n",
"You can also change the `REGION` variable used by Vertex AI. Learn more about [Vertex AI regions](https://cloud.google.com/vertex-ai/docs/general/locations)."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "2dw8q9fdQEH5"
},
"outputs": [],
"source": [
"REGION = \"us-central1\""
"LOCATION = \"us-central1\" # @param {type:\"string\"}"
]
},
{
@@ -243,7 +268,7 @@
"id": "42c8a7c56abd"
},
"source": [
"#### User Email\n",
"### User email\n",
"\n",
"Set your user email address to receive monitoring alerts."
]
@@ -264,64 +289,6 @@
" USER_EMAIL = \"noreply@google.com\""
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "gcp_authenticate"
},
"source": [
"### Authenticate your Google Cloud account\n",
"\n",
"Depending on your Jupyter environment, you may have to manually authenticate. Follow the relevant instructions below.\n",
"\n",
"**1. Vertex AI Workbench**\n",
"* Do nothing as you are already authenticated.\n",
"\n",
"**2. Local JupyterLab instance, uncomment and run:**"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "ce6043da7b33"
},
"outputs": [],
"source": [
"# ! gcloud auth login"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "0367eac06a10"
},
"source": [
"**3. Colab, uncomment and run:**"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "21ad4dbb4a61"
},
"outputs": [],
"source": [
"# from google.colab import auth\n",
"# auth.authenticate_user()"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "c13224697bfb"
},
"source": [
"**4. Service account or other**\n",
"* See how to grant Cloud Storage permissions to your service account at https://cloud.google.com/storage/docs/gsutil/commands/iam#ch-examples."
]
},
{
"cell_type": "markdown",
"metadata": {
@@ -338,14 +305,24 @@
"|service-PROJECT_NUMBER@gcp-sa-aiplatform.iam.gserviceaccount.com|AI Platform Service Agent|Vertex AI Service Agent|\n",
"\n",
"\n",
"1. Goto https://console.cloud.google.com/iam-admin/iam.\n",
"2. Check the \"Include Google-provided role grants\" checkbox.\n",
"1. Go to [IAM console](https://console.cloud.google.com/iam-admin/iam).\n",
"2. Check the **Include Google-provided role grants** checkbox.\n",
"3. Find the above emails.\n",
"4. Grant the corresponding roles.\n",
"\n",
"4. Grant the corresponding roles."
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "1d8540ae3bb4"
},
"source": [
"### Using data source from a different project\n",
"- For the BQ data source, grant both service accounts the \"BigQuery Data Viewer\" role.\n",
"- For the CSV data source, grant both service accounts the \"Storage Object Viewer\" role."
"\n",
"If you're using data sources from a different project:\n",
"\n",
"- For BigQuery data source, grant the \"BigQuery Data Viewer\" role to both the service accounts.\n",
"- For CSV data source, grant the \"Storage Object Viewer\" role to both the service accounts."
]
},
{
@@ -376,7 +353,7 @@
"id": "autoset_bucket"
},
"source": [
"**Only if your bucket doesn't already exist**: Run the following cell to create your Cloud Storage bucket."
"**If your bucket doesn't already exist**: Run the following cell to create your Cloud Storage bucket."
]
},
{
@@ -387,7 +364,7 @@
},
"outputs": [],
"source": [
"! gsutil mb -l $REGION -p $PROJECT_ID $BUCKET_URI"
"! gsutil mb -l $LOCATION -p $PROJECT_ID $BUCKET_URI"
]
},
{
@@ -415,12 +392,12 @@
{
"cell_type": "markdown",
"metadata": {
"id": "init_aip:mbsdk,all"
"id": "daecf7b4cdc0"
},
"source": [
"### Initialize Vertex AI SDK for Python\n",
"\n",
"Initialize the Vertex AI SDK for Python for your project and corresponding bucket."
"To get started using Vertex AI, you must [enable the Vertex AI API](https://console.cloud.google.com/flows/enableapi?apiid=aiplatform.googleapis.com)."
]
},
{
@@ -431,7 +408,7 @@
},
"outputs": [],
"source": [
"aiplatform.init(project=PROJECT_ID, location=REGION)"
"aiplatform.init(project=PROJECT_ID, location=LOCATION, staging_bucket=BUCKET_URI)"
]
},
{
@@ -464,11 +441,11 @@
"source": [
"#### Set hardware accelerators\n",
"\n",
"You can set hardware accelerators for prediction (e.g., GPUs) or choose not to use any (CPU). Hardware accelertors lower the latency response for a prediction request. When choosing a hardware accelerators, consider the additional cost trade-off over latency.\n",
"You can set hardware accelerators for prediction (e.g., GPUs) or use only CPUs. Hardware accelertors lower the latency response for a prediction request. When choosing a hardware accelerator, consider the additional cost trade-off over latency.\n",
"\n",
"Set the variables `DEPLOY_GPU/DEPLOY_NGPU` to use a container image supporting a GPU and the number of GPUs allocated to the virtual machine (VM) instance. For example, to use a GPU container image with 4 Nvidia Tesla K80 GPUs allocated to each VM, you would specify:\n",
"Set the variables `DEPLOY_GPU/DEPLOY_NGPU` to use a container image supporting a GPU and the number of GPUs allocated to the virtual machine (VM) instance. For example, to use a GPU container image with 4 Nvidia Tesla T4 GPUs allocated to each VM, you would specify:\n",
"\n",
" (aiplatform.gapic.AcceleratorType.NVIDIA_TESLA_K80, 4)\n",
" (aiplatform.gapic.AcceleratorType.NVIDIA_TESLA_T4, 4)\n",
"\n",
"See the [locations where accelerators are available](https://cloud.google.com/vertex-ai/docs/general/locations#accelerators).\n",
"\n",
@@ -485,7 +462,7 @@
"source": [
"GPU = False\n",
"if GPU:\n",
" DEPLOY_GPU, DEPLOY_NGPU = (aiplatform.gapic.AcceleratorType.NVIDIA_TESLA_K80, 1)\n",
" DEPLOY_GPU, DEPLOY_NGPU = (aiplatform.gapic.AcceleratorType.NVIDIA_TESLA_T4, 1)\n",
"else:\n",
" DEPLOY_GPU, DEPLOY_NGPU = (None, None)"
]
@@ -517,7 +494,7 @@
" DEPLOY_VERSION = \"tf2-cpu.2-5\"\n",
"\n",
"DEPLOY_IMAGE = \"{}-docker.pkg.dev/vertex-ai/prediction/{}:latest\".format(\n",
" REGION.split(\"-\")[0], DEPLOY_VERSION\n",
" LOCATION.split(\"-\")[0], DEPLOY_VERSION\n",
")\n",
"\n",
"print(\"Deployment:\", DEPLOY_IMAGE, DEPLOY_GPU, DEPLOY_NGPU)"
@@ -534,13 +511,13 @@
"Next, set the machine types to use for training and prediction.\n",
"\n",
"- Set the variable `DEPLOY_COMPUTE` to configure your compute resources for prediction.\n",
" - `machine type`\n",
" - Set a `machine type`:\n",
" - `n1-standard`: 3.75GB of memory per vCPU\n",
" - `n1-highmem`: 6.5GB of memory per vCPU\n",
" - `n1-highcpu`: 0.9 GB of memory per vCPU\n",
" - `vCPUs`: number of \\[2, 4, 8, 16, 32, 64, 96 \\]\n",
"\n",
"*Note: You may also use n2 and e2 machine types for training and deployment, but they do not support GPUs*."
"**Note**: You may also use n2 and e2 machine types for training and deployment, but they don't support GPUs."
]
},
{
@@ -572,26 +549,26 @@
"source": [
"## Introduction to Vertex AI Model Monitoring\n",
"\n",
"Vertex AI Model Monitoring is supported for AutoML tabular models and custom tabular models. You can monitor for skew and drift detection of the features in the inbound prediction requests or skew and drift detection of the feature attributions (Explainable AI) in the outbound prediction response -- that is, the distribution of the attributions on how they contributed to the output (predictions).\n",
"Vertex AI Model Monitoring is supported for AutoML tabular models and custom tabular models. You can monitor for skewness and drift detection of the features in the inbound prediction requests or the feature attributions (Explainable AI) in the outbound prediction responses. In other words, you monitor the distribution of the attributions that quantify feature contributions to the output (predictions).\n",
"\n",
"The following are the basic steps to enable model monitoring:\n",
"\n",
"1. Deploy a `Vertex AI` AutoML or custom tabular model to an `Vertex AI Endpoint`.\n",
"1. Deploy a Vertex AI AutoML or custom tabular model to a Vertex AI endpoint.\n",
"2. Configure a model monitoring specification.\n",
"3. Upload the model monitoring specification to the `Vertex AI Endpoint`.\n",
"4. Upload or automatic generation of the `input schema` for parsing.\n",
"5. For feature skew detection, upload the training data for automatic generation of the feature distribution.\n",
"6. For feature attributions, upload corresponding `Vertex AI Explainability` specification.\n",
"3. Upload the model monitoring specification to the Vertex AI endpoint.\n",
"4. Upload schema or use automatic generation of the *input schema* for parsing.\n",
"5. For feature skewness detection, upload the training data. This enables automatic generation of the feature distributions.\n",
"6. For feature attributions, upload the corresponding Vertex Explainable AI specification.\n",
"\n",
"Once configured, you can enable/disable monitoring, change alerts and update the model monitoring configuration. \n",
"\n",
"When model monitoring is enabled, the sampled incoming prediction requests are logged into a BigQuery table. The input feature values contained in the logged requests are then analyzed for skew or drift on an specified interval basis. You set a sampling rate to monitor a subset of the production inputs to a model, and the monitoring interval.\n",
"When model monitoring is enabled, the sampled incoming prediction requests are logged into a BigQuery table. The input feature values contained in the logged requests are then analyzed for skewness or drift on a specified interval basis. You set a sampling rate to monitor a subset of the production inputs to the model, and the monitoring interval.\n",
"\n",
"The model monitoring service needs to know how to parse the feature values, which is referred to as the input schema. For AutoML tabular models, the input schema is automatically generated. For custom tabular models, the service attempts to automatically derive the input schema from the first 1000 prediction requests. Alternatively, one can upload the input schema.\n",
"\n",
"For skew detection, the monitoring service requires a baseline for the statistical distribution of values in the training data. For AutoML tabular models this is automatically derived. For custom tabular models, you upload the training data to the service, and have the service automatically derive the distribution.\n",
"For skewness detection, the monitoring service requires a baseline for the statistical distribution of values in the training data. For AutoML tabular models this is automatically derived. For custom tabular models, you upload the training data to the service, and have the service automatically derive the distribution.\n",
"\n",
"For feature attribution skew and drift detection, requires enabling your deployed model for `Vertex AI Explainability` for custom tabular models. For AutoML models, `Vertex AI Explainability` is automatically enabled.\n",
"For skewness and drift detection in feature attributions, you're required to enable Vertex Explainable AI feature for your deployed custom tabular models. For AutoML models, Vertex Explainable AI is automatically enabled.\n",
"\n",
"Learn more about [Introduction to Vertex AI Model Monitoring](https://cloud.google.com/vertex-ai/docs/model-monitoring/overview)."
]
@@ -602,13 +579,13 @@
"id": "9bf06cd476e9"
},
"source": [
"### Upload the model artifacts as a `Vertex AI Model` resource\n",
"### Upload the model artifacts to Vertex AI Model Registry\n",
"\n",
"First, you upload the pre-trained custom tabular model artifacts as a `Vertex AI Model` resource using the `upload()` method, with the following parameters:\n",
"First, upload the pre-trained custom tabular model artifacts as a Vertex AI model resource using the `upload()` method, with the following parameters:\n",
"\n",
"- `display_name`: The human readable name for the `Model` resource.\n",
"- `display_name`: The human readable name for the model resource.\n",
"- `artifact_uri`: The Cloud Storage location of the model artifacts.\n",
"- `serving_container_image`: The serving container image to use when the model is deployed to a `Vertex AI Endpoint` resource.\n",
"- `serving_container_image`: The serving container image to use when the model is deployed to a Vertex AI endpoint resource.\n",
"- `sync`: Whether to wait for the process to complete, or return immediately (async)."
]
},
@@ -638,9 +615,9 @@
"id": "1069c6f0eba8"
},
"source": [
"### Deploy the `Vertex AI Model` resource to a `Vertex AI Endpoint` resource\n",
"### Deploy the model to an endpoint\n",
"\n",
"Next, you deploy your `Vertex AI Model` resource to a `Vertex AI Endpoint` resource using the `deploy()` method, with the following parameters:\n",
"Next, deploy your Vertex AI model resource to a Vertex AI endpoint resource using the `deploy()` method, with the following parameters:\n",
"\n",
"- `deploy_model_display`: The human reable name for the deployed model.\n",
"- `machine_type`: The machine type for each VM node instance.\n",
@@ -689,11 +666,11 @@
"\n",
"Configuring the monitoring job consists of the following specifications:\n",
"\n",
"- `alert_config`: The email address(es) to send monitoring alerts to.\n",
"- `alert_config`: The email address(es) that are supposed to receive the monitoring alerts.\n",
"- `schedule_config`: The time window to analyze predictions.\n",
"- `logging_sampling_strategy`: The rate for sampling prediction requests. \n",
"- `drift_config`: The features and drift thresholds to monitor.\n",
"- `skew_config`: The features and skew thresholds to monitor."
"- `skew_config`: The features and skewness thresholds to monitor."
]
},
{
@@ -704,9 +681,9 @@
"source": [
"### Configure the alerting specification\n",
"\n",
"First, you configure the `alerting_config` specification with the following settings:\n",
"Configure the `alerting_config` specification with the following settings:\n",
"\n",
"- `user_emails`: A list of one or more email to send alerts to.\n",
"- `user_emails`: A list of one or more emails that should receive the alerts.\n",
"- `enable_logging`: Streams detected anomalies to Cloud Logging. Default is False."
]
},
@@ -760,7 +737,7 @@
"source": [
"### Configure the sampling specification\n",
"\n",
"Next, you configure the `logging_sampling_strategy` specification with the following settings:\n",
"Now, you configure the `logging_sampling_strategy` specification with the following settings:\n",
"\n",
"- `sample_rate`: The rate as a percentage (between 0 and 1) to randomly sample prediction requests for monitoring. Selected samples are logged to a BigQuery table."
]
@@ -788,11 +765,11 @@
"source": [
"### Configure the drift detection specification\n",
"\n",
"Next, you configure the `drift_config` specification with the following settings:\n",
"Then, you configure the `drift_config` specification with the following settings:\n",
"\n",
"- `drift_thresholds`: A dictionary of key/value pairs where the keys are the input features for monitor for drift. The value is the detection threshold. When not specified, the default drift threshold for a feature is 0.3 (30%).\n",
"- `drift_thresholds`: A dictionary of key/value pairs where the keys are the input features for monitoring the drift. The value represents the detection threshold. When not specified, the default drift threshold for a feature is 0.3 (30%).\n",
"\n",
"*Note:* Enabling drift detection is optional."
"**Note:** Enabling drift detection is optional."
]
},
{
@@ -824,13 +801,13 @@
"Next, you configure the `skew_config` specification with the following settings:\n",
"\n",
"- `data_source`: The source of the dataset of the original training data. The format of the source defaults to a BigQuery table. Otherwise the setting `data_format` must be set to one of the values below. The location of the data must be a Cloud Storage location.\n",
" - `csv`: \n",
" - `jsonl`:\n",
" - `tf-record`:\n",
"- `skew_thresholds`: A dictionary of key/value pairs where the keys are the input features for monitor for skew. The value is the detection threshold. When not specified, the default skew threshold for a feature is 0.3 (30%).\n",
" - `csv`\n",
" - `jsonl`\n",
" - `tf-record`\n",
"- `skew_thresholds`: A dictionary of key/value pairs where the keys are the input features for monitoring the skewness. The value represents the detection threshold. When not specified, the default skew threshold for a feature is 0.3 (30%).\n",
"- `target_field`: The target label for the training dataset\n",
"\n",
"*Note:* Enabling skew detection is optional."
"**Note:** Enabling skewness detection is optional."
]
},
{
@@ -866,9 +843,9 @@
"source": [
"### Assemble the objective specification\n",
"\n",
"Finally, you assemble the objective specification `objective_config` with the following settings:\n",
"Finally, you assemble the objective specification (`objective_config`) with the following settings:\n",
"\n",
"- `skew_detection_config`: (Optional) The specification for the skew detection configuration.\n",
"- `skew_detection_config`: (Optional) The specification for the skewness detection configuration.\n",
"- `drift_detection_config`: (Optional) The specification for the drift detection configuration.\n",
"- `explanation_config`: (Optional) The specification for explanations when enabling monitoring for feature attributions."
]
@@ -896,15 +873,15 @@
"source": [
"### Create the input schema\n",
"\n",
"The monitoring service needs to know the features and data types for the the feature inputs to the model, which is referred to as the `input schema`. The `input schema` can either be \n",
"The monitoring service needs to know the features and data types for the the feature inputs to the model, which is referred to as the *input schema*. The *input schema* can either be \n",
" - Preloaded to the monitoring service.\n",
" - Automatically generated by the monitoring service after receiving first 1000 prediction instances.\n",
" \n",
"In this tutorial, you preload the `input schema`.\n",
"In this tutorial, you preload the *input schema*.\n",
"\n",
"#### Create the predefined input schema\n",
"\n",
"The predefined `input schema` is specified as a YAML file. In this example, you retrieve the BigQuery schema for the training data, which includes the feature names and data types, to generate the YAML specification. The predefined `input schema` must be loade to a Cloud Storage location.\n",
"The predefined *input schema* is specified as a YAML file. In this example, you retrieve the BigQuery schema for the training data, which includes the feature names and data types, to generate the YAML specification. The predefined *input schema* must be loaded to a Cloud Storage location.\n",
"\n",
"Learn more about [Custom instance schemas for parsing input](https://cloud.google.com/vertex-ai/docs/model-monitoring/overview#custom-input-schemas)."
]
@@ -962,17 +939,17 @@
"source": [
"### Create the monitoring job\n",
"\n",
"You create a monitoring job, with your monitoring specifications, using the `aiplatform.ModelDeploymentMonitoringJob.create()` method, with the following parameters:\n",
"Create a monitoring job, with your monitoring specifications, using Vertex AI's [`ModelDeploymentMonitoringJob.create()`](https://cloud.google.com/python/docs/reference/aiplatform/1.48.0/summary_method#google_cloud_aiplatform_ModelDeploymentMonitoringJob_create_summary) method, with the following parameters:\n",
"\n",
"- `display_name`: The human readable name for the monitoring job.\n",
"- `project`: The project ID.\n",
"- `region`: The region.\n",
"- `endpoint`: The fully qualified resource name of the `Vertex AI Endpoint` to enable monitoring.\n",
"- `endpoint`: The fully qualified resource name of the Vertex AI endpoint to enable monitoring.\n",
"- `logging_sampling_strategy`: The specification for the sampling configuration.\n",
"- `schedule_config`: The specification for the scheduling configuration.\n",
"- `alert_config`: The specification for the alerting configuration.\n",
"- `objective_configs`: The specification for the objectives configuration.\n",
"- `analysis_instance_schema_uri`: The location of the YAML file containing the `input schema`."
"- `analysis_instance_schema_uri`: The location of the YAML file containing the *input schema*."
]
},
{
@@ -986,7 +963,7 @@
"monitoring_job = aiplatform.ModelDeploymentMonitoringJob.create(\n",
" display_name=\"churn\",\n",
" project=PROJECT_ID,\n",
" location=REGION,\n",
" location=LOCATION,\n",
" endpoint=endpoint,\n",
" logging_sampling_strategy=logging_sampling_strategy,\n",
" schedule_config=schedule_config,\n",
@@ -1008,7 +985,7 @@
"\n",
"An email notification is sent to the email address in the alerting configuration, notifying that the model monitoring job is now enabled.\n",
"\n",
"The contents will appear like:\n",
"The contents of the email appear as below:\n",
"\n",
"<blockquote>\n",
"Hello Vertex AI Customer,\n",
@@ -1027,9 +1004,9 @@
"source": [
"#### Monitoring Job State\n",
"\n",
"After you start the `Vertex AI Model Monitoring` job, it will be in a `PENDING` state until `skew distribution baseline` is calculated. The monitoring service will initiate a batch job to generate the distribution baseline from the training data. \n",
"After you start the Vertex AI Model Monitoring job, it stays in a **PENDING** state until `skew distribution baseline` is calculated. The monitoring service initiates a batch job to generate the distribution baseline from the training data. \n",
"\n",
"Once the baseline distribution is generated, then the monitoring job will enter `OFFLINE` state. On the per interval basis -- e.g., once an hour, the monitoring job will enter `RUNNING` state while analyzing the sampled data. Once completed, it will return to an `OFFLINE` state while awaiting the next scheduled analysis."
"Once the baseline distribution is generated, then the monitoring job changes to **OFFLINE** state. On a per interval basis, for example, once an hour, the monitoring job enters **RUNNING** state while analyzing the sampled data. Once completed, it returns to the **OFFLINE** state while awaiting the next scheduled analysis."
]
},
{
@@ -1053,7 +1030,7 @@
"source": [
"### Automatic generation of the baseline distribution\n",
"\n",
"Next, the monitoring service creates a batch job to analyze the training data to generate the baseline distribution. Once completed, the monitoring service will starting monitoring on the specified interval."
"Next, the monitoring service creates a batch job to analyze the training data to generate the baseline distribution. Once completed, the monitoring service starts monitoring on the specified intervals."
]
},
{
@@ -1079,9 +1056,9 @@
"source": [
"### Generate synthetic prediction requests for skew detection\n",
"\n",
"Next, you extract the first 1000 instances from the BigQuery training table to use for prediction requests. You modify the data (synthetic) to trigger the skew detection in the prediction requests from the training distribution versus serving distribution, as follows:\n",
"Now, you extract the first 1000 instances from the BigQuery training table for creating prediction requests. Then, modify the data (synthetic) to trigger the skewness detection in the prediction requests by making the follwing updates:\n",
"\n",
"- `country`: Set all values to Canada"
"- `country`: Set all values to Canada."
]
},
{
@@ -1121,7 +1098,7 @@
"source": [
"### Make the prediction requests\n",
"\n",
"Next, you send the the 1000 prediction requests to your `Vertex AI Endpoint` resource using the `predict()` method."
"Next, you send the 1000 prediction requests to your Vertex AI endpoint resource using the `predict()` method."
]
},
{
@@ -1149,9 +1126,9 @@
"source": [
"### Logging sampled requests\n",
"\n",
"Once the monitoring service has started, the sampled prediction requests will be logged to Cloud Storage. On the next monitoring interval, the sampled predictions are then copied over to the BigQuery logging table. Once the entries are in the BigQuery table, the monitoring service will analyze the sampled data.\n",
"Once the monitoring service has started, the sampled prediction requests are logged to Cloud Storage. On the next monitoring interval, the sampled predictions are copied to the BigQuery logging table. Once the entries are logged, the monitoring service analyzes the sampled data.\n",
"\n",
"Next, you wait for the first logged entres to appear in the BigQuery table used for logging prediction samples. Since you sent 1000 prediction requests, with 50% sampling, you should see around 500 entries."
"Next, you wait for the first logged entries to appear in the BigQuery logging table for prediction samples. Since you sent 1000 prediction requests, with 50% sampling, you should see around 500 entries."
]
},
{
@@ -1182,17 +1159,17 @@
"id": "aeaea3a7a194"
},
"source": [
"### Skew detection during monitoring\n",
"### Skewness detection during monitoring\n",
"\n",
"The feature input skew detection will occur at the next monitoring interval. In this tutorial, you set the monitoring interval to one hour. So, in about an hour your monitoring job will go from `OFFLINE` to `RUNNING`. While running, it will analyze the logged sampled tables from the predictions during this interval and compare them to the baseline distribution.\n",
"The skewness detection for feature inputs occurs at the next monitoring interval. In this tutorial, you set the monitoring interval to one hour. So, in about an hour your monitoring job goes from **OFFLINE** to **RUNNING**. While running, it analyzes the logged sampled tables from the predictions during this interval and compares them to the baseline distribution.\n",
"\n",
"Once the analysis is completed, the monitoring job will send email notifications on the detected skew, in this case `country`, and the monitoring job will go into `OFFLINE` state until the next interval.\n",
"Once the analysis is completed, the monitoring job sends email notifications on the detected skewness, in this case `country`. Further, the monitoring job goes into **OFFLINE** state until the next interval.\n",
"\n",
"#### Wait for monitoring interval\n",
"\n",
"It can take upwards of 40 minutes from when the analyis occurred on the monitoring interval to when you receive an email alert.\n",
"It can take up to 40 minutes or more from the moment the analysis on the monitoring interval is done until you receive an email alert.\n",
"\n",
"The contents will appear like\n",
"The contents of the email appear as below:\n",
"\n",
"<blockquote>\n",
" Hello Vertex AI Customer,\n",
@@ -1242,7 +1219,7 @@
"source": [
"### Generate synthetic prediction requests for drift detection\n",
"\n",
"Next, you extract the same first 1000 instances from the BigQuery training table to use for prediction requests. You modify the data (synthetic) to trigger the drift detection in the prediction requests from the training distribution versus serving distribution, as follows:\n",
"Next, you extract the same first 1000 instances from the BigQuery training table to use for prediction requests. Then, modify the data (synthetic) to trigger the drift detection in the prediction requests by making the following updates:\n",
"\n",
"- `cnt_user_engagement`: increase the value 4x."
]
@@ -1284,7 +1261,7 @@
"source": [
"### Make the prediction requests\n",
"\n",
"Next, you send the the 1000 prediction requests to your `Vertex AI Endpoint` resource using the `predict()` method."
"Next, you send the 1000 prediction requests to your Vertex AI endpoint resource using the `predict()` method."
]
},
{
@@ -1312,9 +1289,9 @@
"source": [
"### Logging sampled requests\n",
"\n",
"On the next monitoring interval, the sampled predictions are then copied over to the BigQuery logging table. Once the entries are in the BigQuery table, the monitoring service will analyze the sampled data.\n",
"On the next monitoring interval, the sampled predictions are copied to the BigQuery logging table. Once the entries are logged, the monitoring service analyzes the sampled data.\n",
"\n",
"Next, you wait for the first logged entres to appear in the BigQuery table used for logging prediction samples. Since you sent 1000 prediction requests, with 50% sampling, you should see around 1000 entries."
"Next, you wait for the first logged entries to appear in the BigQuery logging table for prediction samples. Since you sent 1000 prediction requests, with 50% sampling, you should see around 1000 entries."
]
},
{
@@ -1347,13 +1324,13 @@
"source": [
"### Drift detection during monitoring\n",
"\n",
"The feature input drift detection will occur at the next monitoring interval. In this tutorial, you set the monitoring interval to one hour. So, in about an hour your monitoring job will go from `OFFLINE` to `RUNNING`. While running, it will analyze the logged sampled tables from the predictions during this interval and compare them to the previous monitoring interva distribution.\n",
"The drift detection for feature inputs occurs at the next monitoring interval. In this tutorial, you set the monitoring interval to one hour. So, in about an hour your monitoring job goes from **OFFLINE** to **RUNNING**. While running, it analyzes the logged sampled tables from the predictions during this interval and compares them to the previous monitoring interval distribution.\n",
"\n",
"Once the analysis is completed, the monitoring job will send email notifications on the detected drift, in this case `cnt_user_engagement`, and the monitoring job will go into `OFFLINE` state until the next interval.\n",
"Once the analysis is completed, the monitoring job sends email notifications on the detected drift, in this case `cnt_user_engagement`. Then, the monitoring job goes into **OFFLINE** state until the next interval.\n",
"\n",
"#### Wait for monitoring interval\n",
"\n",
"It can take upwards of 40 minutes from when the analyis occurred on the monitoring interval to when you receive an email alert."
"It can take up to 40 minutes or more from the moment the analyis on the monitoring interval is done until you receive an email alert."
]
},
{
@@ -1376,7 +1353,9 @@
"source": [
"### Delete the monitoring job\n",
"\n",
"You can delete the monitoring job using the `delete()` method. "
"Once you've received the email alerts and verified the content, you can:\n",
"- pause the monitoring job using the `pause()` method.\n",
"- delete the monitoring job using the `delete()` method. "
]
},
{
@@ -1387,33 +1366,12 @@
},
"outputs": [],
"source": [
"# Pause the job\n",
"monitoring_job.pause()\n",
"# Delete the job\n",
"monitoring_job.delete()"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "c6280efab664"
},
"source": [
"#### Undeploy and delete the `Vertex AI Endpoint` resource\n",
"\n",
"Your `Vertex AI Endpoint` resource can be deleted using the `delete()` method. Prior to deleting, any model deployed to your `Vertex AI Endpoint` resource, must first be undeployed."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "ad0d28b762e5"
},
"outputs": [],
"source": [
"endpoint.undeploy_all()\n",
"endpoint.delete()"
]
},
{
"cell_type": "markdown",
"metadata": {
@@ -1436,13 +1394,24 @@
},
"outputs": [],
"source": [
"delete_bucket = False\n",
"# Undeploy the model from endpoint before deletion\n",
"endpoint.undeploy_all()\n",
"\n",
"if delete_bucket or os.getenv(\"IS_TESTING\"):\n",
"# Delete the endpoint\n",
"endpoint.delete()\n",
"\n",
"# Delete the model\n",
"model.delete()\n",
"\n",
"# Delete the Cloud Storage bucket\n",
"delete_bucket = False # Set True for deletion\n",
"if delete_bucket:\n",
" ! gsutil rm -rf {BUCKET_URI}\n",
"\n",
"# Delete the locally generated files\n",
"! rm -f schema.yaml\n",
"\n",
"# Delete the BigQuery table\n",
"! bq rm -f {PROJECT_ID}.model_deployment_monitoring_{ENDPOINT_ID}"
]
}