Merge branch 'main' into model_monitor_official

notebooks/official/model_monitoring/model_monitoring.ipynb

@@ -35,11 +35,6 @@
         "\n",
         "<table align=\"left\">\n",
         "  <td>\n",
-        "    <a href=\"https://console.cloud.google.com/ai-platform/notebooks/deploy-notebook?name=Model%20Monitoring&download_url=https%3A%2F%2Fraw.githubusercontent.com%2FGoogleCloudPlatform%2Fvertex-ai-samples%2Fmain%2Fnotebooks%2Fofficial%2Fmodel_monitoring%2Fmodel_monitoring.ipynb\">\n",
-        "       <img src=\"https://www.gstatic.com/cloud/images/navigation/vertex-ai.svg\" alt=\"Google Cloud Notebooks\">Open in Workbench AI Notebook\n",
-        "    </a>\n",
-        "  </td> \n",
-        "  <td>\n",
         "    <a href=\"https://colab.research.google.com/github/GoogleCloudPlatform/vertex-ai-samples/blob/main/notebooks/official/model_monitoring/model_monitoring.ipynb\">\n",
         "      <img src=\"https://cloud.google.com/ml-engine/images/colab-logo-32px.png\" alt=\"Colab logo\"> Open in Colab\n",
         "    </a>\n",
@@ -50,6 +45,10 @@
         "      View on GitHub\n",
         "    </a>\n",
         "  </td>\n",
+        "    <td>\n",
+        "    <a href=\"https://console.cloud.google.com/ai-platform/notebooks/deploy-notebook?name=Model%20Monitoring&download_url=https%3A%2F%2Fraw.githubusercontent.com%2FGoogleCloudPlatform%2Fvertex-ai-samples%2Fmain%2Fnotebooks%2Fofficial%2Fmodel_monitoring%2Fmodel_monitoring.ipynb\">\n",
+        "       <img src=\"https://www.gstatic.com/cloud/images/navigation/vertex-ai.svg\" alt=\"Vertex AI logo\">Open in Vertex AI Workbench\n",
+        "    </a>\n",
         "</table>"
       ]
     },
@@ -107,6 +106,7 @@
         "- Create an `Vertex AI Endpoint` resource.\n",
         "- Deploy the `Model` resource to the `Endpoint` resource.\n",
         "- Configure the `Endpoint` resource for model monitoring.\n",
+        "- Initialize the baseline distribution for model monitoring.\n",
         "- Generate synthetic prediction requests.\n",
         "- Understand how to interpret the statistics, visualizations, other data reported by the model monitoring feature."
       ]
@@ -119,7 +119,7 @@
       "source": [
         "### Model\n",
         "\n",
-        "This tutorial uses a pre-trained model, where the model artifacts are stored in a public Cloud Storage bucket. The model predicts for an online gaming site, the probability that a player will churn, i.e. stop being an active player."
+        "This tutorial uses a pre-trained model, where the model artifacts are stored in a public Cloud Storage bucket. The model predicts for an online gaming site, the probability that a player may churn, i.e. stop being an active player."
       ]
     },
     {
@@ -224,22 +224,16 @@
         "    USER_FLAG = \"--user\"\n",
         "\n",
         "# Don't bother installing tensorflow or explainable_ai_sdk on Colab\n",
-        "extra_pkgs = \"tensorflow explainable_ai_sdk\"\n",
+        "extra_pkgs = \"tensorflow==2.7 explainable_ai_sdk\"\n",
         "if \"google.colab\" in sys.modules:\n",
         "  extra_pkgs = \"\"\n",
         "\n",
         "# Install required packages.\n",
-        "! pip3 install {USER_FLAG} \\\n",
+        "! pip3 install --upgrade -q {USER_FLAG} \\\n",
         "    google-cloud-aiplatform \\\n",
+        "    google-cloud-bigquery \\\n",
         "    explainable_ai_sdk \\\n",
-        "    $extra_pkgs \\\n",
-        "    google-api-python-client \\\n",
-        "    google-auth-oauthlib \\\n",
-        "    google-auth-httplib2 \\\n",
-        "    oauth2client \\\n",
-        "    requests \\\n",
-        "    protobuf==3.20.* \\\n",
-        "    google-cloud-storage==1.32.0 "
+        "    $extra_pkgs"
       ]
     },
     {
@@ -537,6 +531,7 @@
         "\n",
         "import google.cloud.aiplatform as aiplatform\n",
         "import matplotlib.pyplot as plt\n",
+        "from google.cloud import bigquery\n",
         "from google.cloud.aiplatform import model_monitoring\n",
         "from google.cloud.aiplatform.explain.metadata.tf.v2 import \\\n",
         "    saved_model_metadata_builder"
@@ -580,6 +575,28 @@
         "aiplatform.init(project=PROJECT_ID, location=REGION)"
       ]
     },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "init_bq"
+      },
+      "source": [
+        "### Create BigQuery client\n",
+        "\n",
+        "In this tutorial, you use data from the same public BigQuery table that was used to train the pre-trained model. You create a client interface, which you subsequently use to access the data."
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "init_bq"
+      },
+      "outputs": [],
+      "source": [
+        "bqclient = bigquery.Client(project=PROJECT_ID)"
+      ]
+    },
     {
       "cell_type": "markdown",
       "metadata": {
@@ -936,7 +953,7 @@
         "### Configure the following fields:\n",
         "\n",
         "1. Log sample rate - Your prediction requests and responses are logged to BigQuery tables, which are automatically created when you create a monitoring job. This parameter specifies the desired logging frequency for those tables.\n",
-        "1. Monitor interval - time window over which to analyze your data and report anomalies. The minimum window is one hour (3600 seconds)\n",
+        "1. Monitor interval - time window over which to analyze your data and report anomalies. The minimum window is one hour (1)\n",
         "1. Target field - prediction target column name in training dataset\n",
         "1. Skew detection threshold - skew threshold for each feature you want to monitor\n",
         "1. Prediction drift threshold - drift threshold for each feature you want to monitor\n",
@@ -958,8 +975,8 @@
         "# Sampling rate (optional, default=.8)\n",
         "LOG_SAMPLE_RATE = 0.8  # @param {type:\"number\"}\n",
         "\n",
-        "# Monitoring Interval in seconds (optional, default=3600).\n",
-        "MONITOR_INTERVAL = 3600  # @param {type:\"number\"}\n",
+        "# Monitoring Interval in seconds (optional, default=1).\n",
+        "MONITOR_INTERVAL = 1  # @param {type:\"number\"}\n",
         "\n",
         "# URI to training dataset.\n",
         "DATASET_BQ_URI = \"bq://mco-mm.bqmlga4.train\"  # @param {type:\"string\"}\n",
@@ -1086,20 +1103,114 @@
         "*Note:* You need to wait for the email notification before making the first prediction request."
       ]
     },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "3960076190ab"
+      },
+      "source": [
+        "## Initialize the parsing for automatically generating the input schema\n",
+        "\n",
+        "After your `Endpoint` receives a 1000 prediction requests, the modeling service will automatically parse and create the `input schema`.\n",
+        "\n",
+        "### Create the 1000 instance data\n",
+        "\n",
+        "In this example, the first 1000 entries in the BigQuery training data are used as the first 1000 prediction requests. \n",
+        "\n",
+        "*Note:* In this context, each instance is a prediction request. In otherwords, sending 1000 prediction requests of a single instance is the same as sending a single prediction request with 1000 instances."
+      ]
+    },
     {
       "cell_type": "code",
       "execution_count": null,
       "metadata": {
-        "id": "OiwOVR4D_xhl"
+        "id": "cb26c3dea306"
       },
       "outputs": [],
       "source": [
-        "# Run a prediction request to generate schema, if necessary.\n",
-        "try:\n",
-        "    _ = endpoint.predict([DEFAULT_INPUT])\n",
-        "    print(\"prediction succeeded\")\n",
-        "except Exception as e:\n",
-        "    print(f\"prediction failed: {e}\")"
+        "# Download the table.\n",
+        "table = bigquery.TableReference.from_string(DATASET_BQ_URI[5:])\n",
+        "\n",
+        "rows = bqclient.list_rows(table, max_results=1000)\n",
+        "\n",
+        "instances = []\n",
+        "for row in rows:\n",
+        "    instance = {}\n",
+        "    for key, value in row.items():\n",
+        "        if key == TARGET:\n",
+        "            continue\n",
+        "        if value is None:\n",
+        "            value = \"\"\n",
+        "        instance[key] = value\n",
+        "    instances.append(instance)\n",
+        "\n",
+        "print(len(instances))"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "6d002569dadc"
+      },
+      "source": [
+        "### Make the initial prediction request\n",
+        "\n",
+        "Next, you send the the 1000 prediction request to your `Vertex AI Endpoint` resource using the `predict()` method."
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "b2d69d89cd66"
+      },
+      "outputs": [],
+      "source": [
+        "response = endpoint.predict(instances=instances)\n",
+        "\n",
+        "prediction = response[0]\n",
+        "\n",
+        "# print the prediction for the first instance\n",
+        "print(prediction[0])"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "e990a8821178"
+      },
+      "source": [
+        "### Automatic generation of the input schema\n",
+        "\n",
+        "After the model monitoring service receives 1000 instances of prediction requests, the monitoring will start analyzing the prediction requests to automatically generate an `input schema` for the feature inputs.\n",
+        "\n",
+        "### Automatic generation of the baseline distribution\n",
+        "\n",
+        "After the `input schema` is generated, the monitoring service creates a batch job to analyze the training data to determine the baseline distribution. "
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "670b5bc98c2a"
+      },
+      "outputs": [],
+      "source": [
+        "# Pause a bit for the baseline distribution to be calculated\n",
+        "if os.getenv(\"IS_TESTING\"):\n",
+        "    import time\n",
+        "\n",
+        "    time.sleep(120)"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "8f9455fc894d"
+      },
+      "source": [
+        "### Example of monitoring data stored in Cloud Storage"
       ]
     },
     {