Merge branch 'main' into model_monitoring_setup

notebooks/community/ml_ops/stage7/model_monitoring_setup.ipynb

@@ -502,6 +502,86 @@
         "- For the CSV data source, grant both service accounts the \"Storage Object Viewer\" role."
       ]
     },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "bucket:custom"
+      },
+      "source": [
+        "### Create a Cloud Storage bucket\n",
+        "\n",
+        "**The following steps are required, regardless of your notebook environment.**\n",
+        "\n",
+        "Set the name of your Cloud Storage bucket below, which you use in this tutorial to upload the `input schema` for the monitoring service.\n",
+        "\n",
+        "Bucket names must be globally unique across all Google Cloud projects, including those outside of your organization."
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "bucket"
+      },
+      "outputs": [],
+      "source": [
+        "BUCKET_NAME = \"[your-bucket-name]\"  # @param {type:\"string\"}\n",
+        "BUCKET_URI = f\"gs://{BUCKET_NAME}\""
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "autoset_bucket"
+      },
+      "outputs": [],
+      "source": [
+        "if BUCKET_NAME == \"\" or BUCKET_NAME is None or BUCKET_NAME == \"[your-bucket-name]\":\n",
+        "    BUCKET_NAME = PROJECT_ID + \"aip-\" + UUID\n",
+        "    BUCKET_URI = \"gs://\" + BUCKET_NAME"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "create_bucket"
+      },
+      "source": [
+        "**Only if your bucket doesn't already exist**: Run the following cell to create your Cloud Storage bucket."
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "create_bucket"
+      },
+      "outputs": [],
+      "source": [
+        "! gsutil mb -l $REGION $BUCKET_URI"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "validate_bucket"
+      },
+      "source": [
+        "Finally, validate access to your Cloud Storage bucket by examining its contents:"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "validate_bucket"
+      },
+      "outputs": [],
+      "source": [
+        "! gsutil ls -al $BUCKET_URI"
+      ]
+    },
     {
       "cell_type": "markdown",
       "metadata": {
@@ -694,16 +774,16 @@
         "2. Configure a model monitoring specification.\n",
         "3. Upload the model monitoring specification to the `Vertex AI Endpoint`.\n",
         "4. Upload or automatic generation of the `input schema` for parsing.\n",
-        "5. For feature skew detection, upload the training data to automatic generation of the feature distribution.\n",
+        "5. For feature skew detection, upload the training data for automatic generation of the feature distribution.\n",
         "6. For feature attributions, upload corresponding `Vertex AI Explainability` specification.\n",
         "\n",
         "Once configured, you can enable/disable monitoring, change alerts and update the model monitoring configuration. \n",
         "\n",
-        "When model monitoring is enabled, incoming prediction requests are logged in a BigQuery table. The input feature values contained in the logged requests are then analyzed for skew or drift. You can set a sampling rate to monitor a subset of the production inputs to a model.\n",
+        "When model monitoring is enabled, the sampled incoming prediction requests are logged into a BigQuery table. The input feature values contained in the logged requests are then analyzed for skew or drift on an specified interval basis. You set a sampling rate to monitor a subset of the production inputs to a model, and the monitoring interval.\n",
         "\n",
         "The model monitoring service needs to know how to parse the feature values, which is referred to as the input schema. For AutoML tabular models, the input schema is automatically provided. For custom tabular models, the service will attempt to automatically derive the input schema from the first 1000 prediction requests. Alternatively, one can upload the input schema.\n",
         "\n",
-        "For skew detection, requires a baseline for the statistical distribution of values in the training data. For AutoML tabular models this is automatically derived. For custom tabular models, you upload the training data to the service, and have the service automatically derived the distribution.\n",
+        "For skew detection, the monitoring service requires a baseline for the statistical distribution of values in the training data. For AutoML tabular models this is automatically derived. For custom tabular models, you upload the training data to the service, and have the service automatically derive the distribution.\n",
         "\n",
         "For feature attribution skew and drift detection, requires enabling your deployed model for `Vertex AI Explainability`\n",
         "\n",
@@ -1144,7 +1224,11 @@
       "source": [
         "#### Monitoring Job State\n",
         "\n",
-        "After you start the `Vertex AI Model Monitoring` job, it will be in a `PENDING` state until the `input schema` and `skew distribution baselines` are calculated. The process happens sequentially. In this example where we use automatic generation of the `input schema` will stay in a `PENDING` state until the 1000 prediction request (discussed subsequently) is sent. Once the `input schema` has been generated, then a batch job will be initiated to generate the distribution baseline from the training data. Once the baseline distribution is generated, then the monitoring job will enter `RUNNING` state."
+        "After you start the `Vertex AI Model Monitoring` job, it will be in a `PENDING` state until the `input schema` and `skew distribution baselines` are calculated. The process happens sequentially. In this example where you use automatic generation of the `input schema`, the service stays in a `PENDING` state until the 1000 prediction request (discussed subsequently) is sent. \n",
+        "\n",
+        "Once the `input schema` has been generated, then a batch job will be initiated to generate the distribution baseline from the training data. Again, the service stays in a `PENDING` state until the baseline distribution is calculated.\n",
+        "\n",
+        "Once the baseline distribution is generated, then the monitoring job will enter `OFFLINE` state. On the per interval basis -- e.g., once an hour, the monitoring job will enter `RUNNING` state while analyzing the sampled data. Once completed, it will return to an `OFFLINE` state while awaiting the next scheduled analysis."
       ]
     },
     {
@@ -1243,7 +1327,7 @@
         "\n",
         "### Automatic generation of the baseline distribution\n",
         "\n",
-        "After the `input schema` is generated, the monitoring service creates a batch job to analyze for first 1000 predictions to generate the baseline distribution. Once completed, the monitoring service will be in `RUNNING` state."
+        "After the `input schema` is generated, the monitoring service creates a batch job to analyze the training data to determine the baseline distribution. "
       ]
     },
     {
@@ -1254,16 +1338,9 @@
       },
       "outputs": [],
       "source": [
-        "import time\n",
-        "\n",
-        "while True:\n",
-        "    time.sleep(60)\n",
-        "    jobs = monitoring_job.list(filter=f\"display_name=churn_{UUID}\")\n",
-        "    job = jobs[0]\n",
-        "    print(job.state)\n",
-        "    if job.state == aiplatform.gapic.JobState.JOB_STATE_PENDING:\n",
-        "        continue\n",
-        "    break"
+        "# Pause a bit for the baseline distribution to be calculated\n",
+        "if os.getenv(\"IS_TESTING\"):\n",
+        "    time.sleep(120)"
       ]
     },
     {
@@ -1415,7 +1492,7 @@
       "source": [
         "# Delete the monitoring logged data BigQuery dataset\n",
         "\n",
-        "! bq rm -r -f {PROJECT_ID}:model_deployment_monitoring_{ENDPOINT_ID}"
+        "! bq rm -r -f {PROJECT_ID}.model_deployment_monitoring_{ENDPOINT_ID}"
       ]
     },
     {
@@ -1430,7 +1507,7 @@
         "\n",
         "#### Create the predefined input schema\n",
         "\n",
-        "The predefined `input schema` is specified as a YAML file. In this example, you retrieve the BigQuery schema for the training data, which includes the feature names and data types, to generate the YAML specification.\n",
+        "The predefined `input schema` is specified as a YAML file. In this example, you retrieve the BigQuery schema for the training data, which includes the feature names and data types, to generate the YAML specification. The predefined `input schema` must be loaded to a Cloud Storage location.\n",
         "\n",
         "Learn more about [Custom instance schemas for parsing input](https://cloud.google.com/vertex-ai/docs/model-monitoring/overview#custom-input-schemas)."
       ]
@@ -1475,7 +1552,9 @@
         "print(yaml)\n",
         "\n",
         "with open(\"schema.yaml\", \"w\") as f:\n",
-        "    f.write(yaml)"
+        "    f.write(yaml)\n",
+        "\n",
+        "! gsutil cp schema.yaml {BUCKET_URI}/schema.yaml"
       ]
     },
     {
@@ -1508,7 +1587,7 @@
         "    schedule_config=schedule_config,\n",
         "    alert_config=alerting_config,\n",
         "    objective_configs=objective_config,\n",
-        "    analysis_instance_schema_uri=\"schema.yaml\",\n",
+        "    analysis_instance_schema_uri=f\"{BUCKET_URI}/schema.yaml\",\n",
         ")\n",
         "\n",
         "print(monitoring_job)"
@@ -1588,7 +1667,12 @@
         "id": "18889460bd33"
       },
       "source": [
-        "### Cleanup"
+        "## Cleaning up\n",
+        "\n",
+        "To clean up all Google Cloud resources used in this project, you can [delete the Google Cloud\n",
+        "project](https://cloud.google.com/resource-manager/docs/creating-managing-projects#shutting_down_projects) you used for the tutorial.\n",
+        "\n",
+        "Otherwise, you can delete the individual resources you created in this tutorial."
       ]
     },
     {
@@ -1599,7 +1683,14 @@
       },
       "outputs": [],
       "source": [
-        "! rm -f schema.yaml"
+        "delete_bucket = False\n",
+        "\n",
+        "if delete_bucket or os.getenv(\"IS_TESTING\"):\n",
+        "    ! gsutil rm -rf {BUCKET_URI}\n",
+        "\n",
+        "! rm -f schema.yaml\n",
+        "\n",
+        "! bq rm -r -f {PROJECT_ID}.model_deployment_monitoring_{ENDPOINT_ID}"
       ]
     }
   ],