Merge branch 'main' into model_monitoring_automl

notebooks/community/ml_ops/stage7/model_monitoring_custom.ipynb

@@ -765,7 +765,7 @@
       "source": [
         "## Introduction to Vertex AI Model Monitoring\n",
         "\n",
-        "Vertex AI Model Monitoring is supported for AutoML tabular models and custom tabular model. You can monitor for skew and drift detection of the features in the inbound prediction requests or skew and drift detection of the feature attributions (Explainable AI) in the outbound prediction response -- that is, the distribution of the attributions on how they contributed to the output (predictions).\n",
+        "Vertex AI Model Monitoring is supported for AutoML tabular models and custom tabular models. You can monitor for skew and drift detection of the features in the inbound prediction requests or skew and drift detection of the feature attributions (Explainable AI) in the outbound prediction response -- that is, the distribution of the attributions on how they contributed to the output (predictions).\n",
         "\n",
         "The following are the basic steps to enable model monitoring:\n",
         "\n",
@@ -780,11 +780,11 @@
         "\n",
         "When model monitoring is enabled, the sampled incoming prediction requests are logged into a BigQuery table. The input feature values contained in the logged requests are then analyzed for skew or drift on an specified interval basis. You set a sampling rate to monitor a subset of the production inputs to a model, and the monitoring interval.\n",
         "\n",
-        "The model monitoring service needs to know how to parse the feature values, which is referred to as the input schema. For AutoML tabular models, the input schema is automatically provided. For custom tabular models, the service will attempt to automatically derive the input schema from the first 1000 prediction requests. Alternatively, one can upload the input schema.\n",
+        "The model monitoring service needs to know how to parse the feature values, which is referred to as the input schema. For AutoML tabular models, the input schema is automatically generated. For custom tabular models, the service will attempt to automatically derive the input schema from the first 1000 prediction requests. Alternatively, one can upload the input schema.\n",
         "\n",
         "For skew detection, the monitoring service requires a baseline for the statistical distribution of values in the training data. For AutoML tabular models this is automatically derived. For custom tabular models, you upload the training data to the service, and have the service automatically derive the distribution.\n",
         "\n",
-        "For feature attribution skew and drift detection, requires enabling your deployed model for `Vertex AI Explainability`\n",
+        "For feature attribution skew and drift detection, requires enabling your deployed model for `Vertex AI Explainability` for custom tabular models. For AutoML models, `Vertex AI Explainability` is automatically enabled.\n",
         "\n",
         "Learn more about [Introduction to Vertex AI Model Monitoring](https://cloud.google.com/vertex-ai/docs/model-monitoring/overview)."
       ]
@@ -1344,7 +1344,7 @@
         "\n",
         "Once the monitoring service has started, the sampled prediction requests will be logged to Cloud Storage. On the next monitoring interval, the sampled predictions are then copied over to the BigQuery logging table. Once the entries are in the BigQuery table, the monitoring service will analyze the sampled data.\n",
         "\n",
-        "Next, you wait for the first logged entres to appear in the BigQuery table used for logging prediction samples. Since you sent a 1000 prediction requests, with 50% sampling, you should see around 500 entries."
+        "Next, you wait for the first logged entres to appear in the BigQuery table used for logging prediction samples. Since you sent 1000 prediction requests, with 50% sampling, you should see around 500 entries."
       ]
     },
     {
@@ -1404,11 +1404,11 @@
         "\n",
         "Training Prediction Skew Anomalies (Raw Feature):\n",
         "\n",
-        "Anomalies Report Path(Google Cloud Storage): gs://cloud-ai-platform-773884b1-2a32-48d6-8b83-c03cde416b68/model_monitoring/job-8672170640054157312/serving/2022-08-25T00:00/stats_and_anomalies/5653675595884658688/anomalies/training_prediction_skew_anomalies\n",
+        "Anomalies Report Path(Google Cloud Storage): gs://cloud-ai-platform-773884b1-2a32-48d6-8b83-c03cde416b68/model_monitoring/job-8672170640054157312/serving/2022-08-25T00:00/stats_and_anomalies/<deployed-model-id>/anomalies/training_prediction_skew_anomalies\n",
         "\n",
         "For more information about the alert, please visit the model monitoring alert page.\n",
         "\n",
-        "Deployed model id: 5653675595884658688\n",
+        "Deployed model id: <deployed-model-id>\n",
         "\n",
         "Feature name\tAnomaly short description\tAnomaly long description\n",
         "country\tHigh Linfty distance between training and serving\tThe Linfty distance between training and serving is 0.947563 (up to six significant digits), above the threshold 0.5. The feature value with maximum difference is: Canada\n",
@@ -1505,9 +1505,9 @@
       "source": [
         "### Logging sampled requests\n",
         "\n",
-        "Once the monitoring service has started, the sampled prediction requests will be logged to Cloud Storage. On the next monitoring interval, the sampled predictions are then copied over to the BigQuery logging table. Once the entries are in the BigQuery table, the monitoring service will analyze the sampled data.\n",
+        "On the next monitoring interval, the sampled predictions are then copied over to the BigQuery logging table. Once the entries are in the BigQuery table, the monitoring service will analyze the sampled data.\n",
         "\n",
-        "Next, you wait for the first logged entres to appear in the BigQuery table used for logging prediction samples. Since you sent a 1000 prediction requests, with 50% sampling, you should see around 1000 entries."
+        "Next, you wait for the first logged entres to appear in the BigQuery table used for logging prediction samples. Since you sent 1000 prediction requests, with 50% sampling, you should see around 1000 entries."
       ]
     },
     {
@@ -1546,13 +1546,7 @@
         "\n",
         "#### Wait for monitoring interval\n",
         "\n",
-        "It can take upwards of 40 minutes from when the analyis occurred on the monitoring interval to when you receive an email alert.\n",
-        "\n",
-        "The contents will appear like\n",
-        "\n",
-        "<blockquote>\n",
-        "   \n",
-        "<blockquote>"
+        "It can take upwards of 40 minutes from when the analyis occurred on the monitoring interval to when you receive an email alert."
       ]
     },
     {
@@ -1635,12 +1629,14 @@
       },
       "outputs": [],
       "source": [
-        "delete_bucket = True\n",
+        "delete_bucket = False\n",
         "\n",
         "if delete_bucket or os.getenv(\"IS_TESTING\"):\n",
         "    ! gsutil rm -rf {BUCKET_URI}\n",
         "\n",
-        "! rm -f schema.yaml"
+        "! rm -f schema.yaml\n",
+        "\n",
+        "! bq rm -f {PROJECT_ID}.model_deployment_monitoring_{ENDPOINT_ID}"
       ]
     }
   ],

notebooks/community/ml_ops/stage7/model_monitoring_setup.ipynb

@@ -766,7 +766,7 @@
       "source": [
         "## Introduction to Vertex AI Model Monitoring\n",
         "\n",
-        "Vertex AI Model Monitoring is supported for AutoML tabular models and custom tabular model. You can monitor for skew and drift detection of the features in the inbound prediction requests or skew and drift detection of the feature attributions (Explainable AI) in the outbound prediction response -- that is, the distribution of the attributions on how they contributed to the output (predictions).\n",
+        "Vertex AI Model Monitoring is supported for AutoML tabular models and custom tabular models. You can monitor for skew and drift detection of the features in the inbound prediction requests or skew and drift detection of the feature attributions (Explainable AI) in the outbound prediction response -- that is, the distribution of the attributions on how they contributed to the output (predictions).\n",
         "\n",
         "The following are the basic steps to enable model monitoring:\n",
         "\n",
@@ -781,7 +781,7 @@
         "\n",
         "When model monitoring is enabled, the sampled incoming prediction requests are logged into a BigQuery table. The input feature values contained in the logged requests are then analyzed for skew or drift on an specified interval basis. You set a sampling rate to monitor a subset of the production inputs to a model, and the monitoring interval.\n",
         "\n",
-        "The model monitoring service needs to know how to parse the feature values, which is referred to as the input schema. For AutoML tabular models, the input schema is automatically provided. For custom tabular models, the service will attempt to automatically derive the input schema from the first 1000 prediction requests. Alternatively, one can upload the input schema.\n",
+        "The model monitoring service needs to know how to parse the feature values, which is referred to as the input schema. For AutoML tabular models, the input schema is automatically generated. For custom tabular models, the service will attempt to automatically derive the input schema from the first 1000 prediction requests. Alternatively, one can upload the input schema.\n",
         "\n",
         "For skew detection, the monitoring service requires a baseline for the statistical distribution of values in the training data. For AutoML tabular models this is automatically derived. For custom tabular models, you upload the training data to the service, and have the service automatically derive the distribution.\n",
         "\n",