fix: possible confusion on when to wait for the email notification

notebooks/official/model_monitoring/model_monitoring.ipynb

@@ -59,15 +59,8 @@
         "id": "lA32H1oKGgpf"
       },
       "source": [
-        "## Overview"
-      ]
-    },
-    {
-      "cell_type": "markdown",
-      "metadata": {
-        "id": "t6Cd51FkG09E"
-      },
-      "source": [
+        "## Overview\n",
+        "\n",
         "### What is Vertex AI Model Monitoring?\n",
         "\n",
         "Modern applications rely on a well established set of capabilities to monitor the health of their services. Examples include:\n",
@@ -88,7 +81,15 @@
         "\n",
         "[Vertex Explainable AI](https://cloud.google.com/vertex-ai/docs/explainable-ai/overview) adds another facet to model monitoring, which we call feature attribution monitoring. Explainable AI enables you to understand the relative contribution of each feature to a resulting prediction. In essence, it assesses the magnitude of each feature's influence.\n",
         "\n",
-        "If production traffic differs from  training data, or varies substantially over time, **either in terms of model predictions or feature attributions**, that's likely to impact the quality of the answers your model produces. When that happens, you'd like to be alerted automatically and responsively, so that **you can anticipate problems before they affect your customer experiences or your revenue streams**.\n",
+        "If production traffic differs from  training data, or varies substantially over time, **either in terms of model predictions or feature attributions**, that's likely to impact the quality of the answers your model produces. When that happens, you'd like to be alerted automatically and responsively, so that **you can anticipate problems before they affect your customer experiences or your revenue streams**."
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "t6Cd51FkG09E"
+      },
+      "source": [
         "### Objective\n",
         "\n",
         "In this notebook, you learn to use the `Vertex AI Model Monitoring` service to detect drift and anomalies in prediction requests from a deployed `Vertex AI Model` resource. \n",
@@ -107,18 +108,38 @@
         "- Deploy the `Model` resource to the `Endpoint` resource.\n",
         "- Configure the `Endpoint` resource for model monitoring.\n",
         "- Generate synthetic prediction requests.\n",
-        "- Understand how to interpret the statistics, visualizations, other data reported by the model monitoring feature.\n",
+        "- Understand how to interpret the statistics, visualizations, other data reported by the model monitoring feature."
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "edba71dc9840"
+      },
+      "source": [
+        "### Model\n",
         "\n",
+        "This tutorial uses a pre-trained model, where the model artifacts are stored in a public Cloud Storage bucket. The model predicts for an online gaming site, the probability that a player will churn, i.e. stop being an active player."
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "5abcd585354f"
+      },
+      "source": [
         "### Costs \n",
         "\n",
         "This tutorial uses billable components of Google Cloud:\n",
         "\n",
-        "* Vertext AI\n",
+        "* Vertex AI\n",
         "* BigQuery\n",
+        "* Cloud Storage\n",
         "\n",
-        "Learn about [Vertext AI\n",
-        "pricing](https://cloud.google.com/vertex-ai/pricing) and [Cloud Storage\n",
-        "pricing](https://cloud.google.com/storage/pricing), and use the [Pricing\n",
+        "Learn about [Vertext AI pricing](https://cloud.google.com/vertex-ai/pricing), \n",
+        "[Cloud Storage pricing](https://cloud.google.com/storage/pricing), \n",
+        "and [BigQuery pricing](https://cloud.google.com/bigquery/pricing)\n",
+        "and use the [Pricing\n",
         "Calculator](https://cloud.google.com/products/calculator/)\n",
         "to generate a cost estimate based on your projected usage."
       ]
@@ -129,6 +150,8 @@
         "id": "8yVpQt-JHKPF"
       },
       "source": [
+        "## Before you begin\n",
+        "\n",
         "### Set up your local development environment\n",
         "\n",
         "**If you are using Colab or Vertex AI Workbench notebooks**, your environment already meets\n",
@@ -165,15 +188,6 @@
         "1. Open this notebook in the Jupyter Notebook dashboard."
       ]
     },
-    {
-      "cell_type": "markdown",
-      "metadata": {
-        "id": "ze4-nDLfK4pw"
-      },
-      "source": [
-        "## Before you begin"
-      ]
-    },
     {
       "cell_type": "markdown",
       "metadata": {
@@ -301,8 +315,7 @@
       },
       "outputs": [],
       "source": [
-        "if PROJECT_ID == \"\" or PROJECT_ID is None:\n",
-        "    PROJECT_ID = \"[your-project-id]\"  # @param {type:\"string\"}\n",
+        "if PROJECT_ID == \"\" or PROJECT_ID is None or PROJECT_ID == \"[your-project-id]\":\n",
         "    # Get your GCP project id from gcloud\n",
         "    shell_output = ! gcloud config list --format 'value(core.project)' 2>/dev/null\n",
         "    PROJECT_ID = shell_output[0]\n",
@@ -349,7 +362,41 @@
       },
       "outputs": [],
       "source": [
-        "REGION = \"us-central1\"  # @param {type: \"string\"}"
+        "REGION = \"[your-region]\"  # @param {type: \"string\"}\n",
+        "\n",
+        "if REGION == \"[your-region]\":\n",
+        "    REGION = \"us-central1\""
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "06571eb4063b"
+      },
+      "source": [
+        "#### UUID\n",
+        "\n",
+        "If you are in a live tutorial session, you might be using a shared test account or project. To avoid name collisions between users on resources created, you create a uuid for each instance session, and append it onto the name of resources you create in this tutorial."
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "4e166d927e36"
+      },
+      "outputs": [],
+      "source": [
+        "import random\n",
+        "import string\n",
+        "\n",
+        "\n",
+        "# Generate a uuid of a specifed length(default=8)\n",
+        "def generate_uuid(length: int = 8) -> str:\n",
+        "    return \"\".join(random.choices(string.ascii_lowercase + string.digits, k=length))\n",
+        "\n",
+        "\n",
+        "UUID = generate_uuid()"
       ]
     },
     {
@@ -492,16 +539,36 @@
       },
       "outputs": [],
       "source": [
-        "SUFFIX = \"aiplatform.googleapis.com\"\n",
-        "API_ENDPOINT = f\"{REGION}-{SUFFIX}\"\n",
-        "PREDICT_API_ENDPOINT = f\"{REGION}-prediction-{SUFFIX}\"\n",
         "if os.getenv(\"IS_TESTING\"):\n",
-        "    !gcloud --quiet components install beta\n",
-        "    !gcloud --quiet components update\n",
-        "!gcloud config set ai/region $REGION\n",
+        "    ! gcloud --quiet components install beta\n",
+        "    ! gcloud --quiet components update\n",
+        "\n",
+        "! gcloud config set ai/region $REGION\n",
         "os.environ[\"GOOGLE_CLOUD_PROJECT\"] = PROJECT_ID"
       ]
     },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "init_aip:mbsdk,region"
+      },
+      "source": [
+        "### Initialize Vertex AI SDK for Python\n",
+        "\n",
+        "Initialize the Vertex AI SDK for Python for your project and corresponding bucket."
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "init_aip:mbsdk,region"
+      },
+      "outputs": [],
+      "source": [
+        "aiplatform.init(project=PROJECT_ID, location=REGION)"
+      ]
+    },
     {
       "cell_type": "markdown",
       "metadata": {
@@ -510,14 +577,14 @@
       "source": [
         "### The example model\n",
         "\n",
-        "The model you'll use in this notebook is based on [this blog post](https://cloud.google.com/blog/topics/developers-practitioners/churn-prediction-game-developers-using-google-analytics-4-ga4-and-bigquery-ml). The idea behind this model is that your company has extensive log data describing how your game users have interacted with the site. The raw data contains the following categories of information:\n",
+        "The model you use in this notebook is based on [this blog post](https://cloud.google.com/blog/topics/developers-practitioners/churn-prediction-game-developers-using-google-analytics-4-ga4-and-bigquery-ml). The idea behind this model is that your company has extensive log data describing how your game users have interacted with the site. The raw data contains the following categories of information:\n",
         "\n",
         "- identity - unique player identitity numbers\n",
         "- demographic features - information about the player, such as the geographic region in which a player is located\n",
         "- behavioral features - counts of the number of times a  player has triggered certain game events, such as reaching a new level\n",
         "- churn propensity - this is the label or target feature, it provides an estimated probability that this player will churn, i.e. stop being an active player.\n",
         "\n",
-        "The blog article referenced above explains how to use BigQuery to store the raw data, pre-process it for use in machine learning, and train a model. Because this notebook focuses on model monitoring, rather than training models, you're going to reuse a pre-trained version of this model, which has been exported to Cloud Storage. In the next section, you will setup your environment and import this model into your own project."
+        "The blog article referenced above explains how to use BigQuery to store the raw data, pre-process the data for machine learning, and train the corresponding model. Because this notebook focuses on model monitoring, rather than training models, you're going to reuse a pre-trained version of this model, which has been exported to Cloud Storage. In the next section, you will setup your environment and import this model into your own project."
       ]
     },
     {
@@ -544,6 +611,7 @@
         "\n",
         "# Sampling distributions for categorical features...\n",
         "DAYOFWEEK = {1: 1040, 2: 1223, 3: 1352, 4: 1217, 5: 1078, 6: 1011, 7: 1110}\n",
+        "\n",
         "LANGUAGE = {\n",
         "    \"en-us\": 4807,\n",
         "    \"en-gb\": 678,\n",
@@ -566,8 +634,11 @@
         "    \"en-nz\": 29,\n",
         "    \"es-es\": 25,\n",
         "}\n",
+        "\n",
         "OS = {\"IOS\": 3980, \"ANDROID\": 3798, \"null\": 253}\n",
+        "\n",
         "MONTH = {6: 3125, 7: 1838, 8: 1276, 9: 1718, 10: 74}\n",
+        "\n",
         "COUNTRY = {\n",
         "    \"United States\": 4395,\n",
         "    \"India\": 486,\n",
@@ -636,7 +707,7 @@
       "source": [
         "### Generate model metadata for Vertex Explainable AI\n",
         "\n",
-        "Run the following cell to extract metadata from the exported model, which is needed for generating the prediction explanations."
+        "Run the following cell to extract metadata from the exported model, which is needed for generating the explanations for a prediction request."
       ]
     },
     {
@@ -648,10 +719,12 @@
       "outputs": [],
       "source": [
         "MODEL_PATH = \"gs://mco-mm/churn\"\n",
+        "\n",
         "params = {\"sampled_shapley_attribution\": {\"path_count\": 10}}\n",
         "EXPLAIN_PARAMS = aiplatform.explain.ExplanationParameters(params)\n",
+        "\n",
         "builder = saved_model_metadata_builder.SavedModelMetadataBuilder(\n",
-        "    MODEL_PATH, outputs_to_explain=[\"churned_probs\"]\n",
+        "    model_path=MODEL_PATH, outputs_to_explain=[\"churned_probs\"]\n",
         ")\n",
         "EXPLAIN_META = builder.get_metadata_protobuf()"
       ]
@@ -680,15 +753,15 @@
         "MODEL_NAME = \"churn\"\n",
         "IMAGE = \"us-docker.pkg.dev/cloud-aiplatform/prediction/tf2-cpu.2-5:latest\"\n",
         "\n",
-        "aiplatform.init(project=PROJECT_ID, location=REGION)\n",
         "model = aiplatform.Model.upload(\n",
         "    display_name=MODEL_NAME,\n",
         "    artifact_uri=MODEL_PATH,\n",
         "    serving_container_image_uri=IMAGE,\n",
         "    explanation_parameters=EXPLAIN_PARAMS,\n",
         "    explanation_metadata=EXPLAIN_META,\n",
+        "    sync=True,\n",
         ")\n",
-        "model.wait()\n",
+        "\n",
         "MODEL_ID = model.resource_name.split(\"/\")[-1]"
       ]
     },
@@ -709,7 +782,7 @@
       "source": [
         "## Deploy your Model resource to an Endpoint resource\n",
         "\n",
-        "Now that you've imported your model into your project, you need to create an endpoint to serve your model. An endpoint can be thought of as a channel through which your model provides prediction services. Once established, you'll be able to make prediction requests on your model via the public internet.  Your endpoint is also serverless, in the sense that Google Cloud ensures high availability by reducing single points of failure, and scalability by dynamically allocating resources to meet the demand for your service. In this way, you are able to focus on your model quality, and freed from adminstrative and infrastructure concerns.\n",
+        "Now that you've imported your model into your project, you need to create an endpoint to serve your model. An endpoint can be thought of as a channel through which your model provides prediction services. Once established, you can make online prediction requests on your model via the public internet.  Your endpoint is also serverless, in the sense that Google Cloud ensures high availability by reducing single points of failure, and scalability by dynamically allocating resources to meet the demand for your service. In this way, you are able to focus on your model quality, and freed from adminstrative and infrastructure concerns.\n",
         "\n",
         "Run the next cell to deploy your model to an endpoint. **This will take about ten minutes to complete.**"
       ]
@@ -780,7 +853,8 @@
       },
       "source": [
         "### Test results\n",
-        "Taking a closer look at the results, we see the following elements:\n",
+        "\n",
+        "Taking a look at the results, you see the following elements:\n",
         "\n",
         "- **churned_values** - a set of possible values (0 and 1) for the target field\n",
         "- **churned_probs** - a corresponding set of probabilities for each possible target field value (5x10^-40 and 1.0, respectively)\n",
@@ -797,7 +871,7 @@
       "source": [
         "## Run an explanation test\n",
         "\n",
-        "We can also run a test of Explainable AI on this endpoint. Run the next cell to send a test explanation request. If everything works as expected, you should receive a response encoding the feature importance of this prediction in a text representation called JSON, along with a bar chart summarizing the results.\n",
+        "You can run a test of Explainable AI on this endpoint. Run the next cell to send a test explanation request. The response you receive encodes the feature importance of this prediction in a text representation called JSON, along with a bar chart summarizing the results.\n",
         "\n",
         "**Try this now by running the next cell.**"
       ]
@@ -839,7 +913,7 @@
         "\n",
         "Now that you've created an endpoint to serve prediction requests on your model, you're ready to start a monitoring job to keep an eye on model quality and to alert you if and when input begins to deviate in way that may impact your model's prediction quality.\n",
         "\n",
-        "In this section, you will configure and create a model monitoring job based on the churn propensity model you imported from BigQuery ML."
+        "In this section, you configure and create a model monitoring job based on the churn propensity model you imported from BigQuery ML."
       ]
     },
     {
@@ -909,7 +983,7 @@
         "id": "e10f3d0fa538"
       },
       "source": [
-        "You can modefy the threshold values and the config keys, so that you can monitor other features in the model as well."
+        "You can change the threshold values and the configuration settings, so that you can monitor other features in the model as well."
       ]
     },
     {
@@ -937,14 +1011,17 @@
         "    attribute_skew_thresholds=ATTRIB_SKEW_THRESHOLDS,\n",
         "    target_field=TARGET,\n",
         ")\n",
+        "\n",
         "drift_config = model_monitoring.DriftDetectionConfig(\n",
         "    drift_thresholds=DRIFT_THRESHOLDS,\n",
         "    attribute_drift_thresholds=ATTRIB_DRIFT_THRESHOLDS,\n",
         ")\n",
+        "\n",
         "explanation_config = model_monitoring.ExplanationConfig()\n",
         "objective_config = model_monitoring.ObjectiveConfig(\n",
         "    skew_config, drift_config, explanation_config\n",
         ")\n",
+        "\n",
         "# Create sampling configuration\n",
         "random_sampling = model_monitoring.RandomSampleConfig(sample_rate=LOG_SAMPLE_RATE)\n",
         "\n",
@@ -970,22 +1047,6 @@
         ")"
       ]
     },
-    {
-      "cell_type": "code",
-      "execution_count": null,
-      "metadata": {
-        "id": "OiwOVR4D_xhl"
-      },
-      "outputs": [],
-      "source": [
-        "# Run a prediction request to generate schema, if necessary.\n",
-        "try:\n",
-        "    _ = endpoint.predict([DEFAULT_INPUT])\n",
-        "    print(\"prediction succeeded\")\n",
-        "except Exception:\n",
-        "    print(\"prediction failed\")"
-      ]
-    },
     {
       "cell_type": "markdown",
       "metadata": {
@@ -1001,6 +1062,35 @@
         "As your monitoring job collects data, measurements are stored in Cloud Storage and you are free to examine your data at any time. The \"Statistics and Anomalies Root Path\" specifies the location of your measurements in Cloud Storage. Run the following cell to see an example of the layout of these measurements in Cloud Storage. If you substitute the Cloud Storage URL in your job creation email, you can view the structure and content of the data files for your own monitoring job."
       ]
     },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "6f38e8423bce"
+      },
+      "source": [
+        "### Create the sampling distribution\n",
+        "\n",
+        "Next, you send a first test prediction request. The model monitoring service will analyze the distribution of features and automatically create a baseline to monitor deviations from the baseline.\n",
+        "\n",
+        "*Note:* You need to wait for the email notification before making the first prediction request."
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "OiwOVR4D_xhl"
+      },
+      "outputs": [],
+      "source": [
+        "# Run a prediction request to generate schema, if necessary.\n",
+        "try:\n",
+        "    _ = endpoint.predict([DEFAULT_INPUT])\n",
+        "    print(\"prediction succeeded\")\n",
+        "except Exception as e:\n",
+        "    print(f\"prediction failed: {e}\")"
+      ]
+    },
     {
       "cell_type": "code",
       "execution_count": null,
@@ -1038,7 +1128,7 @@
       "source": [
         "### You can create monitoring jobs with other user interfaces\n",
         "\n",
-        "In the previous cells, you created a monitoring job using the Python client library. You can also use the *gcloud* command line tool or the Cloud Console to create a model monitoring job. \n"
+        "In the previous cells, you created a monitoring job using the Python client library. Alternatively, you can use the *gcloud* command line tool or the Cloud Console to create a model monitoring job. \n"
       ]
     },
     {
@@ -1049,9 +1139,9 @@
       "source": [
         "## Interpret your results\n",
         "\n",
-        "Model Monitoring detects an anomaly when the threshold set for a feature is exceeded. The following cells give you a sense of the alerting and reporting experience after model monitoring anomalies have been detected.\n",
+        "Vertex AI Model Monitoring detects an anomaly when the threshold set for a feature is exceeded. The following cells give you a sense of the alerting and reporting experience after model monitoring anomalies have been detected.\n",
         "\n",
-        "Model Monitoring automatically notifies you of detected anomalies through email, but you can also [set up alerts through Cloud Logging](https://cloud.google.com/vertex-ai/docs/model-monitoring/using-model-monitoring#monitor-job)."
+        "Vertex AI Model Monitoring automatically notifies you of detected anomalies through email, but you can also [set up alerts through Cloud Logging](https://cloud.google.com/vertex-ai/docs/model-monitoring/using-model-monitoring#monitor-job)."
       ]
     },
     {
@@ -1132,20 +1222,15 @@
       "cell_type": "code",
       "execution_count": null,
       "metadata": {
-        "id": "TPP_ImwJDFJf"
+        "id": "d6cc924aa1fb"
       },
       "outputs": [],
       "source": [
-        "# Delete endpoint resource\n",
-        "tmp = ! gcloud ai endpoints describe --region=$REGION $ENDPOINT_ID --quiet \\\n",
-        "        | grep \"  id:\"\n",
-        "DEPLOYED_MODEL_ID = tmp[1][7:-1]\n",
-        "! gcloud ai endpoints undeploy-model --deployed-model-id=$DEPLOYED_MODEL_ID \\\n",
-        "    $ENDPOINT_ID --quiet\n",
-        "! gcloud ai endpoints delete $ENDPOINT --quiet\n",
+        "# Undeploy the model and delete the endpoint\n",
+        "endpoint.undeploy_all()\n",
+        "endpoint.delete()\n",
         "\n",
-        "# Delete model resource\n",
-        "! gcloud ai models delete $MODEL_ID --quiet\n",
+        "model.delete()\n",
         "\n",
         "# Delete BQ table and dataset\n",
         "rmtable = f\"bq rm -f model_deployment_monitoring_{ENDPOINT_ID}.serving_predict\"\n",